When a stack fails: rollbacks, UPDATE_ROLLBACK_FAILED and drift
CloudFormation's best feature is also the one that confuses people at 2 am: when a step fails, it rolls back on its own, and the event list fills with dozens of lines that look like errors but are only the cleanup. This lesson reads failed stacks the way an on-call engineer has to - find the one line that is the cause, know which states a stack can be updated from and which ones need a different command - and then looks at the quieter failure: drift, when the resources no longer match the template because someone changed them by hand.
Need to know: a failed create ends in ROLLBACK_COMPLETE (everything it made is deleted; the stack can only be deleted). A failed update ends in UPDATE_ROLLBACK_COMPLETE (back to the previous configuration; healthy). When the rollback itself fails - usually because a resource was changed or deleted outside CloudFormation - the stack is stuck in UPDATE_ROLLBACK_FAILED: no updates, no deploys, until continue-update-rollback finishes it, if necessary with --resources-to-skip. Drift detection (detect-stack-drift) compares every resource with the template and says IN_SYNC, MODIFIED, DELETED or NOT_CHECKED.
A create that failed: find the first failure
The lab has three stacks in three different states:
$ aws cloudformation list-stacks --query 'StackSummaries[?starts_with(StackName,`try-`)].[StackName,StackStatus]' --output text
try-web CREATE_COMPLETE
try-checkout UPDATE_ROLLBACK_FAILED
try-broken ROLLBACK_COMPLETE
try-broken tried to create a bucket, a log group and a parameter forty minutes ago:
$ aws cloudformation describe-stack-events --stack-name try-broken --query 'reverse(StackEvents)[].[LogicalResourceId,ResourceStatus]' --output text
try-broken CREATE_IN_PROGRESS
Assets CREATE_IN_PROGRESS
AssetLogs CREATE_IN_PROGRESS
AssetLogs CREATE_IN_PROGRESS
Assets CREATE_FAILED
AssetLogs CREATE_FAILED
try-broken ROLLBACK_IN_PROGRESS
AssetLogs DELETE_IN_PROGRESS
AssetLogs DELETE_COMPLETE
try-broken ROLLBACK_COMPLETE
Three failures in that list, and only one of them is the cause. Resource creation cancelled is CloudFormation stopping the work that was running in parallel; the ROLLBACK_IN_PROGRESS line just repeats which resource failed. The query every on-call engineer should have in their shell history - the oldest event whose status ends in _FAILED:
$ aws cloudformation describe-stack-events --stack-name try-broken --query 'reverse(StackEvents)[?ends_with(ResourceStatus,`_FAILED`)] | [0].[LogicalResourceId,ResourceStatusReason]' --output text
Assets Resource handler returned message: "Your previous request to create the named bucket succeeded and you already own it. (Service: S3, Status Code: 409, Request ID: fef6519f-e156-499d-8e9e-82abf38477e8)" (RequestToken: 23b58b73-deb4-4aa7-b28c-f6845992b026, HandlerErrorCode: AlreadyExists)
Resource handler returned message: "..." wraps the service's own error (here S3's: the bucket name already exists - in this account); the part in brackets is the service and the request ID to give AWS Support. The bucket was made by hand months ago, outside any stack.
A stack in ROLLBACK_COMPLETE has nothing in it, but its name is taken and it cannot be updated:
$ aws cloudformation update-stack --stack-name try-broken --use-previous-template
aws: [ERROR]: An error occurred (ValidationError) when calling the UpdateStack operation: Stack:arn:aws:cloudformation:eu-central-1:111122223333:stack/try-broken/42131d1f-3a86-45c1-8b7b-85a24927642c is in ROLLBACK_COMPLETE state and can not be updated.
Delete it, fix the template (or import the existing bucket, or pick another name), create again. For debugging a create, --on-failure DO_NOTHING (or --disable-rollback) keeps the half-built stack so you can look at the resources that did get created; never in a pipeline.
An update that failed - and a rollback that failed too
try-checkout is worse. An update fifty minutes ago added a port to the security group, bumped the version parameter and added a subnet:
$ aws cloudformation describe-stacks --stack-name try-checkout --query 'Stacks[0].[StackStatus,StackStatusReason]' --output text
UPDATE_ROLLBACK_FAILED The following resource(s) failed to update: [AppSg].
$ aws cloudformation describe-stack-events --stack-name try-checkout --max-items 11 --query 'StackEvents[].[LogicalResourceId,ResourceStatus,ResourceStatusReason]' --output text
try-checkout UPDATE_ROLLBACK_FAILED The following resource(s) failed to update: [AppSg].
Version UPDATE_COMPLETE None
AppSg UPDATE_FAILED Resource handler returned message: "The security group 'sg-0e3e495101ef96a9e' does not exist (Service: Ec2, Status Code: 400, Request ID: d3d64c27-232b-4854-9d4c-9193ff464773)" (RequestToken: d13640f4-06b6-474f-9d58-b8a4776fa1b5, HandlerErrorCode: NotFound)
Version UPDATE_IN_PROGRESS None
AppSg UPDATE_IN_PROGRESS None
try-checkout UPDATE_ROLLBACK_IN_PROGRESS The following resource(s) failed to create: [Cache].
Cache CREATE_FAILED Resource handler returned message: "The CIDR '10.250.0.0/24' is invalid. (Service: Ec2, Status Code: 400, Request ID: b8838555-55f0-4721-a43a-761547f71976)" (RequestToken: 9f351665-a34d-4426-a370-66de5fa53768, HandlerErrorCode: InvalidRequest)
Cache CREATE_IN_PROGRESS None
AppSg UPDATE_COMPLETE None
Version UPDATE_COMPLETE None
Version UPDATE_IN_PROGRESS None
Read it bottom up: AppSg and Version updated fine; Cache failed (a CIDR outside the VPC); CloudFormation started rolling back - and to roll AppSg back it had to update the security group again, which was gone: someone deleted it while the update ran. A rollback that cannot finish leaves UPDATE_ROLLBACK_FAILED, and in that state the stack refuses everything a pipeline does:
$ cd ~/oncall-lab/labs/aws/try4
$ aws cloudformation update-stack --stack-name try-checkout --template-body file://checkout-v2.yaml --parameters ParameterKey=VpcId,UsePreviousValue=true
aws: [ERROR]: An error occurred (ValidationError) when calling the UpdateStack operation: Stack:arn:aws:cloudformation:eu-central-1:111122223333:stack/try-checkout/5904a00c-638f-4793-a471-95f21d1db67f is in UPDATE_ROLLBACK_FAILED state and can not be updated.
The way out is continue-update-rollback. If the cause is gone (a permission restored, a dependency back), plain continue-update-rollback retries. Here it would fail again - the group cannot come back with the same ID - so the resource is skipped: CloudFormation marks it as rolled back without touching it.
$ aws cloudformation continue-update-rollback --stack-name try-checkout --resources-to-skip AppSg
$ aws cloudformation wait stack-rollback-complete --stack-name try-checkout && echo rolled back
rolled back
$ aws cloudformation describe-stack-events --stack-name try-checkout --max-items 4 --query 'StackEvents[].[LogicalResourceId,ResourceStatus,ResourceStatusReason]' --output text
try-checkout UPDATE_ROLLBACK_COMPLETE None
try-checkout UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS None
AppSg UPDATE_COMPLETE Resource skipped during UpdateRollback
try-checkout UPDATE_ROLLBACK_IN_PROGRESS User Initiated
Only resources in UPDATE_FAILED can be skipped, and skipping is a promise you now have to keep: CloudFormation believes AppSg is the old group with the old ID, and it does not exist. Drift detection shows exactly that:
$ aws cloudformation detect-stack-drift --stack-name try-checkout --query StackDriftDetectionId --output text
6b454785-a13b-4f83-856f-a18efb221508
$ sleep 10; aws cloudformation describe-stack-resource-drifts --stack-name try-checkout --query 'StackResourceDrifts[].[LogicalResourceId,StackResourceDriftStatus]' --output text
AppSg DELETED
Version IN_SYNC
The clean repair is an update that gives the group a new logical ID (AppSgV2): CloudFormation creates a new group and "deletes" the old one, which succeeds because a delete of something that is already gone counts as done. Then the stack is consistent again.
cancel-update-stack is the other emergency command: it stops an update that is still running (UPDATE_IN_PROGRESS) and rolls it back - useful when you see a bad change start, useless after the fact.
Drift: when someone fixed it in the console
try-web is a healthy stack - but healthy only means the last operation worked. Drift detection checks the live resources against the template:
$ aws cloudformation detect-stack-drift --stack-name try-web --query StackDriftDetectionId --output text
2b94665d-715b-4c0d-ae63-592218303513
$ sleep 10; aws cloudformation describe-stacks --stack-name try-web --query 'Stacks[0].DriftInformation'
{
"StackDriftStatus": "DRIFTED",
"LastCheckTimestamp": "2026-09-22T20:00:51.100000+00:00"
}
$ aws cloudformation describe-stack-resource-drifts --stack-name try-web --stack-resource-drift-status-filters MODIFIED DELETED --query 'StackResourceDrifts[].[LogicalResourceId,StackResourceDriftStatus]' --output text
WebSg MODIFIED
Workers MODIFIED
$ aws cloudformation describe-stack-resource-drifts --stack-name try-web --stack-resource-drift-status-filters MODIFIED --query 'StackResourceDrifts[].PropertyDifferences[]'
[
{
"PropertyPath": "/SecurityGroupIngress/0",
"ExpectedValue": "null",
"ActualValue": "{\"IpProtocol\":\"tcp\",\"FromPort\":22,\"ToPort\":22,\"CidrIp\":\"0.0.0.0/0\",\"Description\":\"debug - remove later\"}",
"DifferenceType": "ADD"
},
{
"PropertyPath": "/Value",
"ExpectedValue": "8",
"ActualValue": "16",
"DifferenceType": "NOT_EQUAL"
}
]
Someone opened SSH to the world ("debug - remove later") and doubled the worker count. Each PropertyDifference gives the path, the expected value (the template) and the actual one, with DifferenceType ADD, REMOVE or NOT_EQUAL.
What drift detection is not:
- not continuous - it runs when you ask (or on a schedule you build: AWS Config has a managed rule,
cloudformation-stack-drift-detection-check, that does it for you); - not a fix -
update-stackwith the same template says "No updates are to be performed", because CloudFormation compares templates, not reality. You fix drift by changing the resource back by hand, or by changing the template to match reality and updating; - not complete - only properties you set in the template are compared, and some resource types are not supported (
NOT_CHECKED).
The decision is a team one: if the console change was right, put it in the template; if it was a shortcut, undo it. What must not happen is leaving it - the next update that touches the security group silently removes the SSH rule, or fails because of it.
On the exam (SOA-C03 3.1): "identify and remediate CloudFormation errors": the first
CREATE_FAILEDevent is the cause;ROLLBACK_COMPLETEstacks must be deleted and recreated;UPDATE_ROLLBACK_FAILEDis fixed withContinueUpdateRollback(fix the cause, or skip the resource); a stack's resources changed outside CloudFormation are found with drift detection. Insufficient IAM permissions of the person or of the stack's service role are a classic cause of both failures.
In an interview: "A CloudFormation stack is stuck in UPDATE_ROLLBACK_FAILED. What do you do?" - "Find why the rollback failed: describe-stack-events, the UPDATE_FAILED events after UPDATE_ROLLBACK_IN_PROGRESS - usually a resource deleted or changed by hand, or a permission the role lost. Fix that cause if I can and run continue-update-rollback; if the resource cannot come back, continue-update-rollback with --resources-to-skip for it. Then the stack is UPDATE_ROLLBACK_COMPLETE, and I run drift detection and an update that brings the skipped resource back in line."
You can now: find the first failure in a stack's events, explain ROLLBACK_COMPLETE, UPDATE_ROLLBACK_COMPLETE and UPDATE_ROLLBACK_FAILED, get a stuck stack out with continue-update-rollback and --resources-to-skip, know when cancel-update-stack helps, and detect and read drift.