OnCallReadyStart learning

AWS Lesson 37.7 · AWS IV: Operations Automation · 21 min read

When a stack fails: rollbacks, UPDATE_ROLLBACK_FAILED and drift

In plain words

When a stack update fails halfway, CloudFormation walks back through what it did and undoes it, like retracing your steps to put furniture back. Usually that works.

But if someone moved a piece of furniture out of the house while you were busy, you cannot put it back where it was, and you get stuck in the doorway. That is UPDATE_ROLLBACK_FAILED. Drift is the gentler cousin: someone changed things by hand, the stack does not know, and you only find out when you ask it to compare.

When a stack fails: rollbacks, UPDATE_ROLLBACK_FAILED and drift

CloudFormation's best feature is also the one that confuses people at 2 am: when a step fails, it rolls back on its own, and the event list fills with dozens of lines that look like errors but are only the cleanup. This lesson reads failed stacks the way an on-call engineer has to - find the one line that is the cause, know which states a stack can be updated from and which ones need a different command - and then looks at the quieter failure: drift, when the resources no longer match the template because someone changed them by hand.

Need to know: a failed create ends in ROLLBACK_COMPLETE (everything it made is deleted; the stack can only be deleted). A failed update ends in UPDATE_ROLLBACK_COMPLETE (back to the previous configuration; healthy). When the rollback itself fails - usually because a resource was changed or deleted outside CloudFormation - the stack is stuck in UPDATE_ROLLBACK_FAILED: no updates, no deploys, until continue-update-rollback finishes it, if necessary with --resources-to-skip. Drift detection (detect-stack-drift) compares every resource with the template and says IN_SYNC, MODIFIED, DELETED or NOT_CHECKED.

A create that failed: find the first failure

The lab has three stacks in three different states:

$ aws cloudformation list-stacks --query 'StackSummaries[?starts_with(StackName,`try-`)].[StackName,StackStatus]' --output text
try-web	CREATE_COMPLETE
try-checkout	UPDATE_ROLLBACK_FAILED
try-broken	ROLLBACK_COMPLETE

try-broken tried to create a bucket, a log group and a parameter forty minutes ago:

$ aws cloudformation describe-stack-events --stack-name try-broken --query 'reverse(StackEvents)[].[LogicalResourceId,ResourceStatus]' --output text
try-broken	CREATE_IN_PROGRESS
Assets	CREATE_IN_PROGRESS
AssetLogs	CREATE_IN_PROGRESS
AssetLogs	CREATE_IN_PROGRESS
Assets	CREATE_FAILED
AssetLogs	CREATE_FAILED
try-broken	ROLLBACK_IN_PROGRESS
AssetLogs	DELETE_IN_PROGRESS
AssetLogs	DELETE_COMPLETE
try-broken	ROLLBACK_COMPLETE

Three failures in that list, and only one of them is the cause. Resource creation cancelled is CloudFormation stopping the work that was running in parallel; the ROLLBACK_IN_PROGRESS line just repeats which resource failed. The query every on-call engineer should have in their shell history - the oldest event whose status ends in _FAILED:

$ aws cloudformation describe-stack-events --stack-name try-broken --query 'reverse(StackEvents)[?ends_with(ResourceStatus,`_FAILED`)] | [0].[LogicalResourceId,ResourceStatusReason]' --output text
Assets	Resource handler returned message: "Your previous request to create the named bucket succeeded and you already own it. (Service: S3, Status Code: 409, Request ID: fef6519f-e156-499d-8e9e-82abf38477e8)" (RequestToken: 23b58b73-deb4-4aa7-b28c-f6845992b026, HandlerErrorCode: AlreadyExists)

Resource handler returned message: "..." wraps the service's own error (here S3's: the bucket name already exists - in this account); the part in brackets is the service and the request ID to give AWS Support. The bucket was made by hand months ago, outside any stack.

A stack in ROLLBACK_COMPLETE has nothing in it, but its name is taken and it cannot be updated:

$ aws cloudformation update-stack --stack-name try-broken --use-previous-template
aws: [ERROR]: An error occurred (ValidationError) when calling the UpdateStack operation: Stack:arn:aws:cloudformation:eu-central-1:111122223333:stack/try-broken/42131d1f-3a86-45c1-8b7b-85a24927642c is in ROLLBACK_COMPLETE state and can not be updated.

Delete it, fix the template (or import the existing bucket, or pick another name), create again. For debugging a create, --on-failure DO_NOTHING (or --disable-rollback) keeps the half-built stack so you can look at the resources that did get created; never in a pipeline.

An update that failed - and a rollback that failed too

try-checkout is worse. An update fifty minutes ago added a port to the security group, bumped the version parameter and added a subnet:

$ aws cloudformation describe-stacks --stack-name try-checkout --query 'Stacks[0].[StackStatus,StackStatusReason]' --output text
UPDATE_ROLLBACK_FAILED	The following resource(s) failed to update: [AppSg].
$ aws cloudformation describe-stack-events --stack-name try-checkout --max-items 11 --query 'StackEvents[].[LogicalResourceId,ResourceStatus,ResourceStatusReason]' --output text
try-checkout	UPDATE_ROLLBACK_FAILED	The following resource(s) failed to update: [AppSg].
Version	UPDATE_COMPLETE	None
AppSg	UPDATE_FAILED	Resource handler returned message: "The security group 'sg-0e3e495101ef96a9e' does not exist (Service: Ec2, Status Code: 400, Request ID: d3d64c27-232b-4854-9d4c-9193ff464773)" (RequestToken: d13640f4-06b6-474f-9d58-b8a4776fa1b5, HandlerErrorCode: NotFound)
Version	UPDATE_IN_PROGRESS	None
AppSg	UPDATE_IN_PROGRESS	None
try-checkout	UPDATE_ROLLBACK_IN_PROGRESS	The following resource(s) failed to create: [Cache].
Cache	CREATE_FAILED	Resource handler returned message: "The CIDR '10.250.0.0/24' is invalid. (Service: Ec2, Status Code: 400, Request ID: b8838555-55f0-4721-a43a-761547f71976)" (RequestToken: 9f351665-a34d-4426-a370-66de5fa53768, HandlerErrorCode: InvalidRequest)
Cache	CREATE_IN_PROGRESS	None
AppSg	UPDATE_COMPLETE	None
Version	UPDATE_COMPLETE	None
Version	UPDATE_IN_PROGRESS	None

Read it bottom up: AppSg and Version updated fine; Cache failed (a CIDR outside the VPC); CloudFormation started rolling back - and to roll AppSg back it had to update the security group again, which was gone: someone deleted it while the update ran. A rollback that cannot finish leaves UPDATE_ROLLBACK_FAILED, and in that state the stack refuses everything a pipeline does:

$ cd ~/oncall-lab/labs/aws/try4
$ aws cloudformation update-stack --stack-name try-checkout --template-body file://checkout-v2.yaml --parameters ParameterKey=VpcId,UsePreviousValue=true
aws: [ERROR]: An error occurred (ValidationError) when calling the UpdateStack operation: Stack:arn:aws:cloudformation:eu-central-1:111122223333:stack/try-checkout/5904a00c-638f-4793-a471-95f21d1db67f is in UPDATE_ROLLBACK_FAILED state and can not be updated.

The way out is continue-update-rollback. If the cause is gone (a permission restored, a dependency back), plain continue-update-rollback retries. Here it would fail again - the group cannot come back with the same ID - so the resource is skipped: CloudFormation marks it as rolled back without touching it.

$ aws cloudformation continue-update-rollback --stack-name try-checkout --resources-to-skip AppSg
$ aws cloudformation wait stack-rollback-complete --stack-name try-checkout && echo rolled back
rolled back
$ aws cloudformation describe-stack-events --stack-name try-checkout --max-items 4 --query 'StackEvents[].[LogicalResourceId,ResourceStatus,ResourceStatusReason]' --output text
try-checkout	UPDATE_ROLLBACK_COMPLETE	None
try-checkout	UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS	None
AppSg	UPDATE_COMPLETE	Resource skipped during UpdateRollback
try-checkout	UPDATE_ROLLBACK_IN_PROGRESS	User Initiated

Only resources in UPDATE_FAILED can be skipped, and skipping is a promise you now have to keep: CloudFormation believes AppSg is the old group with the old ID, and it does not exist. Drift detection shows exactly that:

$ aws cloudformation detect-stack-drift --stack-name try-checkout --query StackDriftDetectionId --output text
6b454785-a13b-4f83-856f-a18efb221508
$ sleep 10; aws cloudformation describe-stack-resource-drifts --stack-name try-checkout --query 'StackResourceDrifts[].[LogicalResourceId,StackResourceDriftStatus]' --output text
AppSg	DELETED
Version	IN_SYNC

The clean repair is an update that gives the group a new logical ID (AppSgV2): CloudFormation creates a new group and "deletes" the old one, which succeeds because a delete of something that is already gone counts as done. Then the stack is consistent again.

cancel-update-stack is the other emergency command: it stops an update that is still running (UPDATE_IN_PROGRESS) and rolls it back - useful when you see a bad change start, useless after the fact.

Drift: when someone fixed it in the console

try-web is a healthy stack - but healthy only means the last operation worked. Drift detection checks the live resources against the template:

$ aws cloudformation detect-stack-drift --stack-name try-web --query StackDriftDetectionId --output text
2b94665d-715b-4c0d-ae63-592218303513
$ sleep 10; aws cloudformation describe-stacks --stack-name try-web --query 'Stacks[0].DriftInformation'
{
    "StackDriftStatus": "DRIFTED",
    "LastCheckTimestamp": "2026-09-22T20:00:51.100000+00:00"
}
$ aws cloudformation describe-stack-resource-drifts --stack-name try-web --stack-resource-drift-status-filters MODIFIED DELETED --query 'StackResourceDrifts[].[LogicalResourceId,StackResourceDriftStatus]' --output text
WebSg	MODIFIED
Workers	MODIFIED
$ aws cloudformation describe-stack-resource-drifts --stack-name try-web --stack-resource-drift-status-filters MODIFIED --query 'StackResourceDrifts[].PropertyDifferences[]'
[
    {
        "PropertyPath": "/SecurityGroupIngress/0",
        "ExpectedValue": "null",
        "ActualValue": "{\"IpProtocol\":\"tcp\",\"FromPort\":22,\"ToPort\":22,\"CidrIp\":\"0.0.0.0/0\",\"Description\":\"debug - remove later\"}",
        "DifferenceType": "ADD"
    },
    {
        "PropertyPath": "/Value",
        "ExpectedValue": "8",
        "ActualValue": "16",
        "DifferenceType": "NOT_EQUAL"
    }
]

Someone opened SSH to the world ("debug - remove later") and doubled the worker count. Each PropertyDifference gives the path, the expected value (the template) and the actual one, with DifferenceType ADD, REMOVE or NOT_EQUAL.

What drift detection is not:

  • not continuous - it runs when you ask (or on a schedule you build: AWS Config has a managed rule, cloudformation-stack-drift-detection-check, that does it for you);
  • not a fix - update-stack with the same template says "No updates are to be performed", because CloudFormation compares templates, not reality. You fix drift by changing the resource back by hand, or by changing the template to match reality and updating;
  • not complete - only properties you set in the template are compared, and some resource types are not supported (NOT_CHECKED).

The decision is a team one: if the console change was right, put it in the template; if it was a shortcut, undo it. What must not happen is leaving it - the next update that touches the security group silently removes the SSH rule, or fails because of it.

On the exam (SOA-C03 3.1): "identify and remediate CloudFormation errors": the first CREATE_FAILED event is the cause; ROLLBACK_COMPLETE stacks must be deleted and recreated; UPDATE_ROLLBACK_FAILED is fixed with ContinueUpdateRollback (fix the cause, or skip the resource); a stack's resources changed outside CloudFormation are found with drift detection. Insufficient IAM permissions of the person or of the stack's service role are a classic cause of both failures.

In an interview: "A CloudFormation stack is stuck in UPDATE_ROLLBACK_FAILED. What do you do?" - "Find why the rollback failed: describe-stack-events, the UPDATE_FAILED events after UPDATE_ROLLBACK_IN_PROGRESS - usually a resource deleted or changed by hand, or a permission the role lost. Fix that cause if I can and run continue-update-rollback; if the resource cannot come back, continue-update-rollback with --resources-to-skip for it. Then the stack is UPDATE_ROLLBACK_COMPLETE, and I run drift detection and an update that brings the skipped resource back in line."

You can now: find the first failure in a stack's events, explain ROLLBACK_COMPLETE, UPDATE_ROLLBACK_COMPLETE and UPDATE_ROLLBACK_FAILED, get a stuck stack out with continue-update-rollback and --resources-to-skip, know when cancel-update-stack helps, and detect and read drift.

Why it helps

Failed stacks are where CloudFormation incidents happen, and the skills are specific: read events oldest first and find the first real failure, know which states can be updated, and get a stack out of UPDATE_ROLLBACK_FAILED with continue-update-rollback - skipping what cannot come back and repairing it afterwards. Drift detection explains many of those failures, because out-of-band changes are what break rollbacks. Knowing whether to put reality back or change the template is the judgement call the exam and real outages both ask for.

Commands in this lesson

aws cd sleep

FAQ

How do I tell the cause from the noise in stack events?

Read them oldest first with reverse(StackEvents) and filter for statuses ending in _FAILED. The first one whose reason is not "Resource creation cancelled" or "Resource update cancelled" is the cause. Later failures are CloudFormation stopping the work that was running in parallel and cleaning up.

What does --resources-to-skip actually do?

It tells continue-update-rollback to give up on rolling back the named resources and mark them as rolled back. The stack finishes in UPDATE_ROLLBACK_COMPLETE, but CloudFormation now believes something that may not be true about those resources. Follow it with drift detection and an update that repairs them, often with a new logical ID.

Can I cancel an update that is taking too long?

Yes, with cancel-update-stack while the stack is in UPDATE_IN_PROGRESS. CloudFormation then rolls back to the previous state, exactly as if the update had failed. It does not work for creates or deletes, and the rollback still has to succeed for the stack to be usable.

Which resources can drift detection check?

Most common resource types support it, and for each resource it compares the properties set in the template with the live values. Properties the template does not set are not compared, and resources of unsupported types show NOT_CHECKED. A resource deleted outside CloudFormation shows DELETED.

Is drift always a mistake to revert?

No. Sometimes the console change was the right fix during an incident, and the template is what is out of date. Decide which side is correct: put reality back with the CLI, or put the change into the template and update the stack. What you must not do is leave it, because the next update can overwrite or fail on it.

In an interview Mid

A stack is in UPDATE_ROLLBACK_FAILED and every deployment fails. What do you do?

First find out why the rollback failed: read the events oldest first - the first UPDATE_FAILED is the original cause, and the failure during the rollback is why it is stuck, usually a resource changed or deleted out of band or a lost permission. If the cause can be fixed, continue-update-rollback retries. If the resource cannot come back, continue-update-rollback --resources-to-skip finishes the rollback, and then I repair the stack - a new logical ID for the lost resource - and run detect-stack-drift to confirm it is IN_SYNC.

Also asked: What is the difference between ROLLBACK_COMPLETE and UPDATE_ROLLBACK_COMPLETE? · How do you detect and fix drift in CloudFormation? · Why can an out-of-band change break a stack update?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.