Change sets, replacement, and protecting what matters
Creating a stack is the easy day. The dangerous moment is the update: one changed property can mean CloudFormation modifies a resource in place, interrupts it, or replaces it - creates a new one, points everything at it, and deletes the old one, data included. A renamed bucket is a new, empty bucket. A changed security group description is a new security group. This lesson is about seeing that before it happens, and about the four guards that stop it from happening by accident.
Need to know: a change set is CloudFormation's terraform plan: create-change-set with the new template and parameters, describe-change-set to read what would happen, then execute-change-set - or delete-change-set and walk away. Every row says Action (Add / Modify / Remove) and Replacement (True, False, or Conditional). The guards: DeletionPolicy and UpdateReplacePolicy (Retain / Snapshot: keep the old resource when it would be deleted or replaced), a stack policy (deny Update:Replace / Update:Delete on chosen resources), and termination protection (no delete-stack at all).
Three kinds of update
Each property in the CloudFormation resource reference says what changing it does - its update requires line:
- No interruption - changed in place (a log group's retention, a parameter's value, tags);
- Some interruption - changed in place but the resource restarts or blinks (an EC2 instance type: stop, change, start);
- Replacement - a new physical resource: anything that is part of the resource's identity, usually its name (
BucketName,LogGroupName,RoleName), a VPC'sCidrBlock, a security group'sGroupDescriptionorVpcId, an instance'sImageIdorSubnetId.
That is also the hidden cost of giving resources names in a template: a named resource can never be replaced without a conflict ("already exists"), and renaming it is always a replacement.
The whole path of an update, from the change set to the cleanup - and where it can go wrong:
Reading a change set
The lab's stack try-data holds an exports bucket, its log group and a parameter that tells the app where the bucket is. A developer's new version renames the bucket and raises the retention:
$ cd ~/oncall-lab/labs/aws/try4
$ grep -n -e BucketName -e 'Default:' data.yaml data-v2.yaml
data.yaml:5: Default: 14
data.yaml:10: BucketName: oncall-try-exports-111122223333
data-v2.yaml:5: Default: 30
data-v2.yaml:10: BucketName: oncall-try-exports-v2-111122223333
$ aws cloudformation create-change-set --stack-name try-data --change-set-name v2 --template-body file://data-v2.yaml --query Id --output text
arn:aws:cloudformation:eu-central-1:111122223333:changeSet/v2/17c50581-9197-4d32-b715-16573408300c
$ aws cloudformation wait change-set-create-complete --stack-name try-data --change-set-name v2
$ aws cloudformation describe-change-set --stack-name try-data --change-set-name v2 --query 'Changes[].ResourceChange.[Action,LogicalResourceId,ResourceType,Replacement]' --output table
-----------------------------------------------------
| DescribeChangeSet |
+--------+--------+-----------------------+---------+
| Modify| Data | AWS::S3::Bucket | True |
| Modify| Logs | AWS::Logs::LogGroup | False |
| Modify| Where | AWS::SSM::Parameter | False |
+--------+--------+-----------------------+---------+
One True in that table is the whole story: the bucket would be replaced, so every object in it would be left behind in the old bucket - which CloudFormation then tries to delete. The Details say why, property by property:
$ aws cloudformation describe-change-set --stack-name try-data --change-set-name v2 --query "Changes[?ResourceChange.LogicalResourceId=='Data'].ResourceChange.Details[]"
[
{
"Target": {
"Attribute": "Properties",
"Name": "BucketName",
"RequiresRecreation": "Always"
},
"Evaluation": "Static",
"ChangeSource": "DirectModification"
}
]
$ aws cloudformation describe-change-set --stack-name try-data --change-set-name v2 --query "Changes[?ResourceChange.LogicalResourceId=='Where'].ResourceChange.Details[]"
[
{
"Target": {
"Attribute": "Properties",
"Name": "Value",
"RequiresRecreation": "Never"
},
"Evaluation": "Dynamic",
"ChangeSource": "ResourceReference",
"CausingEntity": "Data"
}
]
RequiresRecreationis the property's rule:Always(this change replaces),Never,Conditionally(depends on the value - e.g. some changes to an RDS instance);EvaluationStatic= known now;Dynamic= only known while the update runs, because it depends on another resource's new value;ChangeSource:DirectModification(you edited the property),ParameterReference(a parameter changed),ResourceReference/ResourceAttribute(another resource it refers to changes -CausingEntitynames it).Wherechanges only becauseDatagets a new name.
Replacement: Conditional on a row means "it depends on a Dynamic value" - treat it as True until you know better. A change set is read-only: nothing has happened yet.
Guard 1: DeletionPolicy and UpdateReplacePolicy
Two resource attributes (next to Type, not inside Properties) decide what happens to the old physical resource:
Data:
Type: AWS::S3::Bucket
DeletionPolicy: Retain # delete-stack, or removed from the template: keep it
UpdateReplacePolicy: Retain # replaced by an update: keep the old one
Delete is the default for almost everything; Retain keeps the resource (it leaves the stack and is yours to clean up); Snapshot takes a final snapshot first (EBS volumes, RDS, ElastiCache, Neptune, Redshift). For anything that holds data - buckets, tables, databases, KMS keys - the habit is Retain on both. They do not stop the change; they make it survivable.
Guard 2: a stack policy
A stack policy is a JSON document on the stack that says which update actions are allowed on which logical resources: Update:Modify, Update:Replace, Update:Delete, Update:*. Once a stack has one, everything not explicitly allowed is denied - so the usual shape is "allow everything, deny the dangerous ones on the data":
$ cat policy.json
{
"Statement": [
{
"Effect": "Allow",
"Action": "Update:*",
"Principal": "*",
"Resource": "*"
},
{
"Effect": "Deny",
"Action": [
"Update:Replace",
"Update:Delete"
],
"Principal": "*",
"Resource": "LogicalResourceId/Data"
}
]
}
$ aws cloudformation set-stack-policy --stack-name try-data --stack-policy-body file://policy.json
$ aws cloudformation execute-change-set --stack-name try-data --change-set-name v2
$ aws cloudformation wait stack-update-complete --stack-name try-data; echo "exit $?"
aws: [ERROR]: Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state: For expression "Stacks[].StackStatus" we matched expected path: "UPDATE_ROLLBACK_COMPLETE" at least once
exit 255
$ aws cloudformation describe-stack-events --stack-name try-data --query 'StackEvents[?ResourceStatus==`UPDATE_FAILED`].[LogicalResourceId,ResourceStatusReason]' --output text
Data Action denied by stack policy: Statement [#2] does not allow [Update:Replace] for resource [LogicalResourceId/Data];
$ aws cloudformation describe-stacks --stack-name try-data --query 'Stacks[0].StackStatus' --output text
UPDATE_ROLLBACK_COMPLETE
The update reached the bucket, the policy refused the replacement, and the stack rolled back to exactly what it was: UPDATE_ROLLBACK_COMPLETE is a healthy state - the stack works and can be updated again. The change set is spent (EXECUTE_FAILED); a fixed template needs a new one.
A stack policy is not IAM: it protects against mistakes in templates, applies to everyone including administrators, and is overridden for one update with --stack-policy-during-update-body - deliberately, in the command line, where a reviewer sees it. It cannot be removed, only replaced with a more permissive one.
Guard 3: termination protection
$ aws cloudformation update-termination-protection --stack-name try-data --enable-termination-protection --query StackId --output text
arn:aws:cloudformation:eu-central-1:111122223333:stack/try-data/ab52fb50-2fa7-427b-b614-a46734a53a52
$ aws cloudformation delete-stack --stack-name try-data
aws: [ERROR]: An error occurred (ValidationError) when calling the DeleteStack operation: Stack [try-data] cannot be deleted while TerminationProtection is enabled
On every stack that matters - and off only by someone who can call update-termination-protection --no-enable-termination-protection, which IAM can restrict.
Guard 4: review the change set in the pipeline
The reliable guard is a process: the pipeline creates a change set, prints it, fails (or asks a person) when any row has Replacement True or Conditional or Action Remove on a resource that holds data, and executes only that reviewed change set. aws cloudformation deploy does the create-and-execute in one step (--no-execute-changeset stops after creating it), which is convenient for low-risk stacks and wrong for the database:
$ aws cloudformation describe-change-set --stack-name try-data --change-set-name v2 --query 'length(Changes[?ResourceChange.Replacement!=`False`])'
1
After a replacement goes wrong, the old resource is gone unless a policy kept it - CloudFormation has no "undo" for deleted data, only for its own configuration.
On the exam (SOA-C03 3.1, 2.3): "update a stack without data loss" = a change set first, then
DeletionPolicy: Retain/SnapshotandUpdateReplacePolicyon stateful resources; "prevent an update from replacing or deleting the database" = a stack policy denyingUpdate:ReplaceandUpdate:Deleteon its logical ID; "prevent the stack from being deleted" = termination protection. Know thatReplacement: Conditionalmeans it depends on a dynamic value.
In an interview: "How do you change production infrastructure with CloudFormation safely?" - "Never update directly: create a change set and read it - Action and Replacement per resource, and the Details for why. Anything stateful gets DeletionPolicy and UpdateReplacePolicy Retain or Snapshot, the stack has a stack policy that denies Update:Replace and Update:Delete on those resources, and termination protection. If the update fails it rolls back by itself; if a replacement is really intended, the stack policy is overridden for that one update."
You can now: say which property changes replace a resource, create and read a change set (Action, Replacement, RequiresRecreation, Evaluation, ChangeSource), keep data with DeletionPolicy and UpdateReplacePolicy, write a stack policy that refuses replacing or deleting the data, recognise a stack-policy denial and UPDATE_ROLLBACK_COMPLETE, and switch on termination protection.