OnCallReadyStart learning

AWS Lesson 37.10 · AWS IV: Operations Automation · 19 min read

Exports, pipelines and StackSets: CloudFormation at scale

In plain words

One stack builds things in one account and one Region. A company with forty accounts and three Regions wants the same alarm topic, the same guardrail role, the same Config rules everywhere. A StackSet is a stamp: one template, pressed into every account and Region you choose, one at a time or a few at a time, stopping if a stamp smudges.

Exports and imports are the smaller version of sharing: one stack publishes a value, such as a topic name, and another stack in the same place reads it.

Exports, pipelines and StackSets: CloudFormation at scale

One stack is a tidy unit. A real account has dozens - a network stack, a stack per service, a monitoring stack - and an organization has dozens of accounts in several Regions that all need the same baseline: the same alarm topic, the same Config rules, the same roles for the security team. This lesson connects stacks to each other, puts them behind a pipeline, and rolls one template out to many accounts and Regions with StackSets.

Need to know: a stack shares a value with an Output that has an Export name; another stack in the same account and Region reads it with !ImportValue, and from then on the exporting stack cannot delete or change that export. aws cloudformation deploy is the pipeline command (change set + execute + wait; --no-fail-on-empty-changeset), aws cloudformation package uploads local artifacts (Lambda code, nested templates) to S3 first. A StackSet is one template plus a list of stack instances - (account, Region) pairs - each a normal stack. Self-managed StackSets use two IAM roles you create; service-managed StackSets use AWS Organizations, target OUs, and can deploy automatically to accounts that join.

Exports and imports

try-shared exports the topic every service sends its alerts to; try-consumer subscribes the team's address to it by importing that value:

$ aws cloudformation list-exports --query 'Exports[?starts_with(Name,`try-`)].[Name,Value]' --output text
try-shared-AlertsTopic	arn:aws:sns:eu-central-1:111122223333:try-shared-alerts
$ aws cloudformation list-imports --export-name try-shared-AlertsTopic
{
    "Imports": [
        "try-consumer"
    ]
}
$ aws cloudformation get-template --stack-name try-consumer --query TemplateBody --output text
Description: try - the checkout team listens to the shared alerts topic
Resources:
  TeamMail:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !ImportValue try-shared-AlertsTopic
      Protocol: email
      Endpoint: [email protected]

The coupling is strong on purpose:

While anything imports it, the export is pinned:

$ aws cloudformation delete-stack --stack-name try-shared
$ sleep 5; aws cloudformation describe-stacks --stack-name try-shared --query 'Stacks[0].[StackStatus,StackStatusReason]' --output text
DELETE_FAILED	Export try-shared-AlertsTopic cannot be deleted as it is in use by try-consumer

DELETE_FAILED - and the stack is still fine, nothing was deleted. Exports are for values that almost never change (VPC and subnet IDs, a shared topic, a KMS key ARN); for anything that changes, an SSM parameter the other stack reads with an AWS::SSM::Parameter::Value<String> parameter type is the looser coupling. Nested stacks (AWS::CloudFormation::Stack with a TemplateURL) are the other way to split a big template: the parent owns the children and passes parameters down; package uploads the child templates for you.

The pipeline command

$ cd ~/oncall-lab/labs/aws/try4
$ aws cloudformation deploy --template-file shared.yaml --stack-name try-pipeline

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - try-pipeline
$ aws cloudformation deploy --template-file shared.yaml --stack-name try-pipeline --no-fail-on-empty-changeset; echo "exit $?"

Waiting for changeset to be created..

No changes to deploy. Stack try-pipeline is up to date
exit 0

deploy creates a change set, executes it and waits; with nothing to change it exits 255 unless --no-fail-on-empty-changeset says that is fine - the flag every CI job wants. Other options worth knowing: --parameter-overrides Key=Value, --capabilities, --tags, --role-arn (a service role: the stack acts with that role's permissions, not the pipeline's), and --no-execute-changeset to stop after creating the change set for a review.

StackSets, self-managed: two roles and a template

A self-managed StackSet works anywhere, with or without Organizations. The administration role in the account that owns the StackSet is assumed by CloudFormation; it assumes the execution role in every target account, which does the actual work:

$ aws iam create-role --role-name AWSCloudFormationStackSetAdministrationRole --assume-role-policy-document file://admin-trust.json --query Role.Arn --output text
arn:aws:iam::111122223333:role/AWSCloudFormationStackSetAdministrationRole
$ aws iam put-role-policy --role-name AWSCloudFormationStackSetAdministrationRole --policy-name assume-execution --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"sts:AssumeRole","Resource":"arn:aws:iam::*:role/AWSCloudFormationStackSetExecutionRole"}]}'
$ aws iam create-role --role-name AWSCloudFormationStackSetExecutionRole --assume-role-policy-document file://exec-trust.json --query Role.Arn --output text
arn:aws:iam::111122223333:role/AWSCloudFormationStackSetExecutionRole
$ aws iam attach-role-policy --role-name AWSCloudFormationStackSetExecutionRole --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

Those two names are the defaults; in a real organization the execution role is created in every target account (often by the account factory), and its policy is narrowed to what the StackSets deploy. Now the template, and two instances - this account, two Regions:

$ cat baseline.yaml
Description: ops baseline - an alerts topic and a marker parameter in every account and Region
Resources:
  OpsAlerts:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: try-ops-alerts
  Marker:
    Type: AWS::SSM::Parameter
    Properties:
      Name: /try/baseline/version
      Type: String
      Value: '1'
$ aws cloudformation create-stack-set --stack-set-name try-baseline --template-body file://baseline.yaml
{
    "StackSetId": "try-baseline:738733f2-9797-4889-b755-34dc3d07b9e6"
}
$ aws cloudformation create-stack-instances --stack-set-name try-baseline --accounts 111122223333 --regions eu-central-1 eu-west-1 --operation-preferences MaxConcurrentCount=1,FailureToleranceCount=0
{
    "OperationId": "de7a293f-5384-45b8-a4e1-2325236e8761"
}
$ sleep 60
$ aws cloudformation list-stack-instances --stack-set-name try-baseline --query 'Summaries[].[Account,Region,Status,StackInstanceStatus.DetailedStatus]' --output text
111122223333	eu-central-1	CURRENT	SUCCEEDED
111122223333	eu-west-1	CURRENT	SUCCEEDED
$ aws sns list-topics --region eu-west-1 --query 'Topics[].TopicArn' --output text
arn:aws:sns:eu-west-1:111122223333:try-ops-alerts

Each instance is an ordinary stack named StackSet-try-baseline-<id> in its Region - its events are where you look when an instance is OUTDATED (its last operation failed) or INOPERABLE (CloudFormation cannot reach it, usually a missing or untrusting execution role). Operation preferences control the blast radius: MaxConcurrentCount / MaxConcurrentPercentage (how many at once), FailureToleranceCount / FailureTolerancePercentage (how many may fail before the operation stops), RegionOrder, RegionConcurrencyType.

StackSets, service-managed: Organizations does the roles

With AWS Organizations, a StackSet with --permission-model SERVICE_MANAGED needs no roles: the organization's trusted access creates them. You target OUs instead of account IDs, and with automatic deployment an account that joins the OU gets the stack, one that leaves loses it. It is created from the management account or a delegated administrator for StackSets (--call-as DELEGATED_ADMIN) - this account is one:

$ aws cloudformation create-stack-set --stack-set-name try-org-guard --template-body file://baseline.yaml --permission-model SERVICE_MANAGED --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false
aws: [ERROR]: An error occurred (ValidationError) when calling the CreateStackSet operation: You must be the management account or a delegated administrator to create a stack set with service-managed permissions. Specify --call-as DELEGATED_ADMIN if you are a delegated administrator.
$ aws cloudformation create-stack-set --stack-set-name try-org-guard --template-body file://baseline.yaml --permission-model SERVICE_MANAGED --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false --call-as DELEGATED_ADMIN
{
    "StackSetId": "try-org-guard:640b9852-513a-4736-a772-18408932d147"
}
$ aws cloudformation create-stack-instances --stack-set-name try-org-guard --deployment-targets OrganizationalUnitIds=ou-f6g7-8m3tw1ab,ou-f6g7-2c9vx5df --regions eu-central-1 --call-as DELEGATED_ADMIN --query OperationId --output text
a8289c96-6e26-4328-9ca1-54cffcf4a893
$ sleep 60
$ aws cloudformation list-stack-instances --stack-set-name try-org-guard --call-as DELEGATED_ADMIN --query 'Summaries[].[Account,OrganizationalUnitId,Status]' --output text
111122223333	ou-f6g7-8m3tw1ab	OUTDATED
777788889999	ou-f6g7-6j1rn8gh	CURRENT
222233334444	ou-f6g7-3p7ys4kl	CURRENT

One instance is OUTDATED: this account's stack failed. Its reason, and the stack's own first failure:

$ aws cloudformation list-stack-instances --stack-set-name try-org-guard --call-as DELEGATED_ADMIN --stack-instance-account 111122223333 --query 'Summaries[0].StatusReason' --output text
ResourceLogicalId:Marker, ResourceType:AWS::SSM::Parameter, ResourceStatusReason:Resource handler returned message: "The parameter already exists. To overwrite this value, set the overwrite option in the request to true. (Service: Ssm, Status Code: 400, Request ID: e32aef55-0cb9-4879-b73b-a813f61ba647)" (RequestToken: 413285ae-1372-4994-b961-b5bbea533a41, HandlerErrorCode: AlreadyExists).
$ S=$(aws cloudformation list-stack-instances --stack-set-name try-org-guard --call-as DELEGATED_ADMIN --stack-instance-account 111122223333 --query 'Summaries[0].StackId' --output text); aws cloudformation describe-stack-events --stack-name $S --query 'reverse(StackEvents)[?ends_with(ResourceStatus,`_FAILED`)] | [0].ResourceStatusReason' --output text
Resource handler returned message: "The parameter already exists. To overwrite this value, set the overwrite option in the request to true. (Service: Ssm, Status Code: 400, Request ID: e32aef55-0cb9-4879-b73b-a813f61ba647)" (RequestToken: 413285ae-1372-4994-b961-b5bbea533a41, HandlerErrorCode: AlreadyExists)

The self-managed try-baseline already made /try/baseline/version here: two stacks cannot own one named resource. The classic StackSet failure in real organizations is the same - a resource made by hand (or by an older StackSet) under the name the template wants.

Service-managed StackSets never deploy to the management account itself - that account is handled separately, and should hold as little as possible. (In the lab, the other accounts of the organization are simulated: their instances succeed without a modelled stack.)

Around CloudFormation: CDK, RAM, Image Builder

Three neighbours the exam expects you to place:

  • AWS CDK - infrastructure in TypeScript/Python/Java; cdk synth produces a CloudFormation template, cdk deploy deploys it as a stack (bootstrapped per account and Region with cdk bootstrap). Everything in this lesson - events, rollbacks, drift - applies unchanged;
  • AWS RAM (Resource Access Manager) - shares one resource (a subnet, a Transit Gateway, a Route 53 Resolver rule) with other accounts, instead of deploying a copy into each; StackSets deploy copies, RAM shares one;
  • EC2 Image Builder - pipelines that build and test golden AMIs and container images on a schedule, from a recipe (base image + components) and distribute them to Regions and accounts. The AMI it produces is then a parameter (often an SSM public or private parameter) that stacks and launch templates read. (Concept here: the lab builds nothing with it.)

On the exam (SOA-C03 3.1): "provision and share resources across multiple Regions and accounts" - StackSets (self-managed: the AWSCloudFormationStackSetAdministrationRole and AWSCloudFormationStackSetExecutionRole; service-managed: Organizations, OUs, automatic deployment, delegated administrator) vs RAM for sharing one resource. Cross-stack references are Outputs with Export + Fn::ImportValue, same Region only, and an imported export cannot be deleted or changed.

In an interview: "How would you deploy the same baseline to forty accounts in three Regions?" - "A CloudFormation StackSet with service-managed permissions from the delegated administrator account, targeting the OUs, with automatic deployment so new accounts get it, rolled out with operation preferences - a small MaxConcurrentPercentage and a FailureTolerance of zero - so a bad template stops after the first failures. Each instance is a normal stack, so a failed one is debugged with its own events."

You can now: share values between stacks with exports and imports and know why they pin each other, run cloudformation deploy safely in a pipeline, set up the two self-managed StackSet roles, deploy a StackSet to several Regions with operation preferences, read stack instance statuses, deploy to OUs with a service-managed StackSet as a delegated administrator, and place CDK, RAM and Image Builder.

Why it helps

Multi-account operations are the normal case on AWS, and the CloudOps exam expects you to roll out a baseline across accounts and Regions safely. The lesson covers sharing between stacks with exports (and why they pin each other), cloudformation deploy in pipelines, the two self-managed StackSet roles and their trust, operation preferences that limit the blast radius, and service-managed StackSets with Organizations and a delegated administrator - including the classic failure of a name already taken in one target account.

Commands in this lesson

aws sleep cd cat

FAQ

Why can I not delete a stack that has an export?

Because another stack imports it. CloudFormation refuses to delete or change an export while any stack uses it, so the provider and the consumer are tied together. list-imports shows who uses an export. For looser coupling, put the value in Parameter Store and let the consumer read it.

What do operation preferences control?

How a StackSet operation walks its targets: how many accounts per Region at once (MaxConcurrentCount or Percentage), how many failures are tolerated before it stops (FailureToleranceCount or Percentage), the Region order, and whether Regions run in sequence or in parallel. Small numbers mean a bad template reaches few places.

What does INOPERABLE mean for a stack instance?

CloudFormation can no longer manage that instance, typically because the execution role in the target account is missing, does not trust the administration role, or the account left the organization. Fix the role or remove the instance with delete-stack-instances and --retain-stacks if the stack must stay.

Why do service-managed StackSets skip the management account?

By design: the management account should hold as little as possible, and service-managed deployment targets the member accounts in the OUs you choose. Anything the management account itself needs is deployed separately, usually by a self-managed StackSet or a plain stack.

How does cloudformation deploy differ from create-stack and update-stack?

deploy works out whether the stack exists, creates a change set, executes it and waits, in one command. With --no-fail-on-empty-changeset an unchanged template is not an error, and --no-execute-changeset stops after the change set for a review. It is what most pipelines call.

In an interview Mid

How would you roll out the same baseline to every account and Region in an organization?

A service-managed StackSet from the delegated administrator account, targeting the OUs, with automatic deployment so new accounts get the baseline when they join. Operation preferences limit the blast radius - one Region at a time, a small MaxConcurrentCount, zero failure tolerance at first - and I watch list-stack-instances for OUTDATED or INOPERABLE instances, whose stack events say why. Without Organizations, a self-managed StackSet needs the administration and execution roles in place first.

Also asked: What are CloudFormation exports and imports, and what is their downside? · What is the difference between self-managed and service-managed StackSets? · How do you limit the blast radius of a StackSet update?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.