OnCallReadyStart learning

AWS Lesson 37.4 · AWS IV: Operations Automation · 18 min read

Change sets, replacement, and protecting what matters

In plain words

Before a builder changes your house, a good one shows you a plan: we will repaint this wall, move that door, and - careful - knock down the garage and build a new one. A change set is that plan for a stack. Nothing happens until you say go.

Some changes can be made in place, like repainting. Others mean tearing something down and building it again, like renaming a bucket. The plan marks those as replacements, and for anything holding data that is the line to read twice.

Change sets, replacement, and protecting what matters

Creating a stack is the easy day. The dangerous moment is the update: one changed property can mean CloudFormation modifies a resource in place, interrupts it, or replaces it - creates a new one, points everything at it, and deletes the old one, data included. A renamed bucket is a new, empty bucket. A changed security group description is a new security group. This lesson is about seeing that before it happens, and about the four guards that stop it from happening by accident.

Need to know: a change set is CloudFormation's terraform plan: create-change-set with the new template and parameters, describe-change-set to read what would happen, then execute-change-set - or delete-change-set and walk away. Every row says Action (Add / Modify / Remove) and Replacement (True, False, or Conditional). The guards: DeletionPolicy and UpdateReplacePolicy (Retain / Snapshot: keep the old resource when it would be deleted or replaced), a stack policy (deny Update:Replace / Update:Delete on chosen resources), and termination protection (no delete-stack at all).

Three kinds of update

Each property in the CloudFormation resource reference says what changing it does - its update requires line:

  • No interruption - changed in place (a log group's retention, a parameter's value, tags);
  • Some interruption - changed in place but the resource restarts or blinks (an EC2 instance type: stop, change, start);
  • Replacement - a new physical resource: anything that is part of the resource's identity, usually its name (BucketName, LogGroupName, RoleName), a VPC's CidrBlock, a security group's GroupDescription or VpcId, an instance's ImageId or SubnetId.

That is also the hidden cost of giving resources names in a template: a named resource can never be replaced without a conflict ("already exists"), and renaming it is always a replacement.

The whole path of an update, from the change set to the cleanup - and where it can go wrong:

Reading a change set

The lab's stack try-data holds an exports bucket, its log group and a parameter that tells the app where the bucket is. A developer's new version renames the bucket and raises the retention:

$ cd ~/oncall-lab/labs/aws/try4
$ grep -n -e BucketName -e 'Default:' data.yaml data-v2.yaml
data.yaml:5:    Default: 14
data.yaml:10:      BucketName: oncall-try-exports-111122223333
data-v2.yaml:5:    Default: 30
data-v2.yaml:10:      BucketName: oncall-try-exports-v2-111122223333
$ aws cloudformation create-change-set --stack-name try-data --change-set-name v2 --template-body file://data-v2.yaml --query Id --output text
arn:aws:cloudformation:eu-central-1:111122223333:changeSet/v2/17c50581-9197-4d32-b715-16573408300c
$ aws cloudformation wait change-set-create-complete --stack-name try-data --change-set-name v2
$ aws cloudformation describe-change-set --stack-name try-data --change-set-name v2 --query 'Changes[].ResourceChange.[Action,LogicalResourceId,ResourceType,Replacement]' --output table
-----------------------------------------------------
|                 DescribeChangeSet                 |
+--------+--------+-----------------------+---------+
|  Modify|  Data  |  AWS::S3::Bucket      |  True   |
|  Modify|  Logs  |  AWS::Logs::LogGroup  |  False  |
|  Modify|  Where |  AWS::SSM::Parameter  |  False  |
+--------+--------+-----------------------+---------+

One True in that table is the whole story: the bucket would be replaced, so every object in it would be left behind in the old bucket - which CloudFormation then tries to delete. The Details say why, property by property:

$ aws cloudformation describe-change-set --stack-name try-data --change-set-name v2 --query "Changes[?ResourceChange.LogicalResourceId=='Data'].ResourceChange.Details[]"
[
    {
        "Target": {
            "Attribute": "Properties",
            "Name": "BucketName",
            "RequiresRecreation": "Always"
        },
        "Evaluation": "Static",
        "ChangeSource": "DirectModification"
    }
]
$ aws cloudformation describe-change-set --stack-name try-data --change-set-name v2 --query "Changes[?ResourceChange.LogicalResourceId=='Where'].ResourceChange.Details[]"
[
    {
        "Target": {
            "Attribute": "Properties",
            "Name": "Value",
            "RequiresRecreation": "Never"
        },
        "Evaluation": "Dynamic",
        "ChangeSource": "ResourceReference",
        "CausingEntity": "Data"
    }
]
  • RequiresRecreation is the property's rule: Always (this change replaces), Never, Conditionally (depends on the value - e.g. some changes to an RDS instance);
  • Evaluation Static = known now; Dynamic = only known while the update runs, because it depends on another resource's new value;
  • ChangeSource: DirectModification (you edited the property), ParameterReference (a parameter changed), ResourceReference / ResourceAttribute (another resource it refers to changes - CausingEntity names it). Where changes only because Data gets a new name.

Replacement: Conditional on a row means "it depends on a Dynamic value" - treat it as True until you know better. A change set is read-only: nothing has happened yet.

Guard 1: DeletionPolicy and UpdateReplacePolicy

Two resource attributes (next to Type, not inside Properties) decide what happens to the old physical resource:

  Data:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain          # delete-stack, or removed from the template: keep it
    UpdateReplacePolicy: Retain     # replaced by an update: keep the old one

Delete is the default for almost everything; Retain keeps the resource (it leaves the stack and is yours to clean up); Snapshot takes a final snapshot first (EBS volumes, RDS, ElastiCache, Neptune, Redshift). For anything that holds data - buckets, tables, databases, KMS keys - the habit is Retain on both. They do not stop the change; they make it survivable.

Guard 2: a stack policy

A stack policy is a JSON document on the stack that says which update actions are allowed on which logical resources: Update:Modify, Update:Replace, Update:Delete, Update:*. Once a stack has one, everything not explicitly allowed is denied - so the usual shape is "allow everything, deny the dangerous ones on the data":

$ cat policy.json
{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "Update:*",
      "Principal": "*",
      "Resource": "*"
    },
    {
      "Effect": "Deny",
      "Action": [
        "Update:Replace",
        "Update:Delete"
      ],
      "Principal": "*",
      "Resource": "LogicalResourceId/Data"
    }
  ]
}
$ aws cloudformation set-stack-policy --stack-name try-data --stack-policy-body file://policy.json
$ aws cloudformation execute-change-set --stack-name try-data --change-set-name v2
$ aws cloudformation wait stack-update-complete --stack-name try-data; echo "exit $?"
aws: [ERROR]: Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state: For expression "Stacks[].StackStatus" we matched expected path: "UPDATE_ROLLBACK_COMPLETE" at least once
exit 255
$ aws cloudformation describe-stack-events --stack-name try-data --query 'StackEvents[?ResourceStatus==`UPDATE_FAILED`].[LogicalResourceId,ResourceStatusReason]' --output text
Data	Action denied by stack policy: Statement [#2] does not allow [Update:Replace] for resource [LogicalResourceId/Data];
$ aws cloudformation describe-stacks --stack-name try-data --query 'Stacks[0].StackStatus' --output text
UPDATE_ROLLBACK_COMPLETE

The update reached the bucket, the policy refused the replacement, and the stack rolled back to exactly what it was: UPDATE_ROLLBACK_COMPLETE is a healthy state - the stack works and can be updated again. The change set is spent (EXECUTE_FAILED); a fixed template needs a new one.

A stack policy is not IAM: it protects against mistakes in templates, applies to everyone including administrators, and is overridden for one update with --stack-policy-during-update-body - deliberately, in the command line, where a reviewer sees it. It cannot be removed, only replaced with a more permissive one.

Guard 3: termination protection

$ aws cloudformation update-termination-protection --stack-name try-data --enable-termination-protection --query StackId --output text
arn:aws:cloudformation:eu-central-1:111122223333:stack/try-data/ab52fb50-2fa7-427b-b614-a46734a53a52
$ aws cloudformation delete-stack --stack-name try-data
aws: [ERROR]: An error occurred (ValidationError) when calling the DeleteStack operation: Stack [try-data] cannot be deleted while TerminationProtection is enabled

On every stack that matters - and off only by someone who can call update-termination-protection --no-enable-termination-protection, which IAM can restrict.

Guard 4: review the change set in the pipeline

The reliable guard is a process: the pipeline creates a change set, prints it, fails (or asks a person) when any row has Replacement True or Conditional or Action Remove on a resource that holds data, and executes only that reviewed change set. aws cloudformation deploy does the create-and-execute in one step (--no-execute-changeset stops after creating it), which is convenient for low-risk stacks and wrong for the database:

$ aws cloudformation describe-change-set --stack-name try-data --change-set-name v2 --query 'length(Changes[?ResourceChange.Replacement!=`False`])'
1

After a replacement goes wrong, the old resource is gone unless a policy kept it - CloudFormation has no "undo" for deleted data, only for its own configuration.

On the exam (SOA-C03 3.1, 2.3): "update a stack without data loss" = a change set first, then DeletionPolicy: Retain / Snapshot and UpdateReplacePolicy on stateful resources; "prevent an update from replacing or deleting the database" = a stack policy denying Update:Replace and Update:Delete on its logical ID; "prevent the stack from being deleted" = termination protection. Know that Replacement: Conditional means it depends on a dynamic value.

In an interview: "How do you change production infrastructure with CloudFormation safely?" - "Never update directly: create a change set and read it - Action and Replacement per resource, and the Details for why. Anything stateful gets DeletionPolicy and UpdateReplacePolicy Retain or Snapshot, the stack has a stack policy that denies Update:Replace and Update:Delete on those resources, and termination protection. If the update fails it rolls back by itself; if a replacement is really intended, the stack policy is overridden for that one update."

You can now: say which property changes replace a resource, create and read a change set (Action, Replacement, RequiresRecreation, Evaluation, ChangeSource), keep data with DeletionPolicy and UpdateReplacePolicy, write a stack policy that refuses replacing or deleting the data, recognise a stack-policy denial and UPDATE_ROLLBACK_COMPLETE, and switch on termination protection.

Why it helps

Most CloudFormation disasters are updates that replaced a stateful resource: a renamed bucket, a changed database identifier, a new subnet for an instance. Change sets show Replacement True before anything happens, which is why pipelines create them and a person reads them. The lesson adds the layers that protect what matters when someone does not read: stack policies that refuse replacement or deletion of named resources, DeletionPolicy and UpdateReplacePolicy Retain or Snapshot, and termination protection - and explains why UPDATE_ROLLBACK_COMPLETE is a healthy outcome.

Commands in this lesson

cd grep aws cat

FAQ

What does Replacement Conditional mean?

The property can sometimes change in place and sometimes not, depending on the values involved or other properties - CloudFormation decides during the update. Treat Conditional like True for anything that holds data: check the resource's documentation for that property, or test the change on a copy of the stack.

Does a stack policy stop people from deleting resources in the console?

No. A stack policy only applies to stack updates made through CloudFormation, and only to the stack's own resources. Someone with the permissions can still delete the bucket in the console or with the CLI. Protect against that with IAM, service control policies, bucket policies and termination or deletion protection.

What is the difference between DeletionPolicy and UpdateReplacePolicy?

DeletionPolicy decides what happens to the physical resource when it is removed from the template or the stack is deleted. UpdateReplacePolicy decides what happens to the old physical resource when an update replaces it. Both accept Delete, Retain and, for snapshot-capable types, Snapshot; production data should usually have Retain or Snapshot on both.

Can I review changes from a pipeline without executing them?

Yes. aws cloudformation deploy --no-execute-changeset creates the change set and stops, and describe-change-set prints the rows for a review step or an approval gate. A pipeline can also fail itself when any row shows Replacement True for resource types you consider stateful.

Why is UPDATE_ROLLBACK_COMPLETE called healthy?

Because the update failed and CloudFormation put everything back the way it was. The stack is consistent and can be updated again once the template is fixed. The state to worry about is UPDATE_ROLLBACK_FAILED, where even the rollback could not finish and the stack refuses further updates.

In an interview Mid

How do you stop a CloudFormation update from replacing a production database or bucket?

Several independent layers. Every update goes through a change set that someone reads, and the pipeline stops on rows with Replacement True for stateful types. A stack policy denies Update:Replace and Update:Delete on that logical ID, so even an approved change needs a deliberate override. DeletionPolicy and UpdateReplacePolicy set to Retain or Snapshot keep the data if the resource is removed or replaced anyway, and termination protection guards the whole stack. IAM and SCPs cover the console.

Also asked: Which kinds of property changes cause a resource replacement? · What does a stack policy protect against, and what not? · How do you review CloudFormation changes in a pipeline?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.