Exports, pipelines and StackSets: CloudFormation at scale
One stack is a tidy unit. A real account has dozens - a network stack, a stack per service, a monitoring stack - and an organization has dozens of accounts in several Regions that all need the same baseline: the same alarm topic, the same Config rules, the same roles for the security team. This lesson connects stacks to each other, puts them behind a pipeline, and rolls one template out to many accounts and Regions with StackSets.
Need to know: a stack shares a value with an Output that has an Export name; another stack in the same account and Region reads it with !ImportValue, and from then on the exporting stack cannot delete or change that export. aws cloudformation deploy is the pipeline command (change set + execute + wait; --no-fail-on-empty-changeset), aws cloudformation package uploads local artifacts (Lambda code, nested templates) to S3 first. A StackSet is one template plus a list of stack instances - (account, Region) pairs - each a normal stack. Self-managed StackSets use two IAM roles you create; service-managed StackSets use AWS Organizations, target OUs, and can deploy automatically to accounts that join.
Exports and imports
try-shared exports the topic every service sends its alerts to; try-consumer subscribes the team's address to it by importing that value:
$ aws cloudformation list-exports --query 'Exports[?starts_with(Name,`try-`)].[Name,Value]' --output text
try-shared-AlertsTopic arn:aws:sns:eu-central-1:111122223333:try-shared-alerts
$ aws cloudformation list-imports --export-name try-shared-AlertsTopic
{
"Imports": [
"try-consumer"
]
}
$ aws cloudformation get-template --stack-name try-consumer --query TemplateBody --output text
Description: try - the checkout team listens to the shared alerts topic
Resources:
TeamMail:
Type: AWS::SNS::Subscription
Properties:
TopicArn: !ImportValue try-shared-AlertsTopic
Protocol: email
Endpoint: [email protected]
The coupling is strong on purpose:
While anything imports it, the export is pinned:
$ aws cloudformation delete-stack --stack-name try-shared
$ sleep 5; aws cloudformation describe-stacks --stack-name try-shared --query 'Stacks[0].[StackStatus,StackStatusReason]' --output text
DELETE_FAILED Export try-shared-AlertsTopic cannot be deleted as it is in use by try-consumer
DELETE_FAILED - and the stack is still fine, nothing was deleted. Exports are for values that almost never change (VPC and subnet IDs, a shared topic, a KMS key ARN); for anything that changes, an SSM parameter the other stack reads with an AWS::SSM::Parameter::Value<String> parameter type is the looser coupling. Nested stacks (AWS::CloudFormation::Stack with a TemplateURL) are the other way to split a big template: the parent owns the children and passes parameters down; package uploads the child templates for you.
The pipeline command
$ cd ~/oncall-lab/labs/aws/try4
$ aws cloudformation deploy --template-file shared.yaml --stack-name try-pipeline
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - try-pipeline
$ aws cloudformation deploy --template-file shared.yaml --stack-name try-pipeline --no-fail-on-empty-changeset; echo "exit $?"
Waiting for changeset to be created..
No changes to deploy. Stack try-pipeline is up to date
exit 0
deploy creates a change set, executes it and waits; with nothing to change it exits 255 unless --no-fail-on-empty-changeset says that is fine - the flag every CI job wants. Other options worth knowing: --parameter-overrides Key=Value, --capabilities, --tags, --role-arn (a service role: the stack acts with that role's permissions, not the pipeline's), and --no-execute-changeset to stop after creating the change set for a review.
StackSets, self-managed: two roles and a template
A self-managed StackSet works anywhere, with or without Organizations. The administration role in the account that owns the StackSet is assumed by CloudFormation; it assumes the execution role in every target account, which does the actual work:
$ aws iam create-role --role-name AWSCloudFormationStackSetAdministrationRole --assume-role-policy-document file://admin-trust.json --query Role.Arn --output text
arn:aws:iam::111122223333:role/AWSCloudFormationStackSetAdministrationRole
$ aws iam put-role-policy --role-name AWSCloudFormationStackSetAdministrationRole --policy-name assume-execution --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"sts:AssumeRole","Resource":"arn:aws:iam::*:role/AWSCloudFormationStackSetExecutionRole"}]}'
$ aws iam create-role --role-name AWSCloudFormationStackSetExecutionRole --assume-role-policy-document file://exec-trust.json --query Role.Arn --output text
arn:aws:iam::111122223333:role/AWSCloudFormationStackSetExecutionRole
$ aws iam attach-role-policy --role-name AWSCloudFormationStackSetExecutionRole --policy-arn arn:aws:iam::aws:policy/AdministratorAccess
Those two names are the defaults; in a real organization the execution role is created in every target account (often by the account factory), and its policy is narrowed to what the StackSets deploy. Now the template, and two instances - this account, two Regions:
$ cat baseline.yaml
Description: ops baseline - an alerts topic and a marker parameter in every account and Region
Resources:
OpsAlerts:
Type: AWS::SNS::Topic
Properties:
TopicName: try-ops-alerts
Marker:
Type: AWS::SSM::Parameter
Properties:
Name: /try/baseline/version
Type: String
Value: '1'
$ aws cloudformation create-stack-set --stack-set-name try-baseline --template-body file://baseline.yaml
{
"StackSetId": "try-baseline:738733f2-9797-4889-b755-34dc3d07b9e6"
}
$ aws cloudformation create-stack-instances --stack-set-name try-baseline --accounts 111122223333 --regions eu-central-1 eu-west-1 --operation-preferences MaxConcurrentCount=1,FailureToleranceCount=0
{
"OperationId": "de7a293f-5384-45b8-a4e1-2325236e8761"
}
$ sleep 60
$ aws cloudformation list-stack-instances --stack-set-name try-baseline --query 'Summaries[].[Account,Region,Status,StackInstanceStatus.DetailedStatus]' --output text
111122223333 eu-central-1 CURRENT SUCCEEDED
111122223333 eu-west-1 CURRENT SUCCEEDED
$ aws sns list-topics --region eu-west-1 --query 'Topics[].TopicArn' --output text
arn:aws:sns:eu-west-1:111122223333:try-ops-alerts
Each instance is an ordinary stack named StackSet-try-baseline-<id> in its Region - its events are where you look when an instance is OUTDATED (its last operation failed) or INOPERABLE (CloudFormation cannot reach it, usually a missing or untrusting execution role). Operation preferences control the blast radius: MaxConcurrentCount / MaxConcurrentPercentage (how many at once), FailureToleranceCount / FailureTolerancePercentage (how many may fail before the operation stops), RegionOrder, RegionConcurrencyType.
StackSets, service-managed: Organizations does the roles
With AWS Organizations, a StackSet with --permission-model SERVICE_MANAGED needs no roles: the organization's trusted access creates them. You target OUs instead of account IDs, and with automatic deployment an account that joins the OU gets the stack, one that leaves loses it. It is created from the management account or a delegated administrator for StackSets (--call-as DELEGATED_ADMIN) - this account is one:
$ aws cloudformation create-stack-set --stack-set-name try-org-guard --template-body file://baseline.yaml --permission-model SERVICE_MANAGED --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false
aws: [ERROR]: An error occurred (ValidationError) when calling the CreateStackSet operation: You must be the management account or a delegated administrator to create a stack set with service-managed permissions. Specify --call-as DELEGATED_ADMIN if you are a delegated administrator.
$ aws cloudformation create-stack-set --stack-set-name try-org-guard --template-body file://baseline.yaml --permission-model SERVICE_MANAGED --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false --call-as DELEGATED_ADMIN
{
"StackSetId": "try-org-guard:640b9852-513a-4736-a772-18408932d147"
}
$ aws cloudformation create-stack-instances --stack-set-name try-org-guard --deployment-targets OrganizationalUnitIds=ou-f6g7-8m3tw1ab,ou-f6g7-2c9vx5df --regions eu-central-1 --call-as DELEGATED_ADMIN --query OperationId --output text
a8289c96-6e26-4328-9ca1-54cffcf4a893
$ sleep 60
$ aws cloudformation list-stack-instances --stack-set-name try-org-guard --call-as DELEGATED_ADMIN --query 'Summaries[].[Account,OrganizationalUnitId,Status]' --output text
111122223333 ou-f6g7-8m3tw1ab OUTDATED
777788889999 ou-f6g7-6j1rn8gh CURRENT
222233334444 ou-f6g7-3p7ys4kl CURRENT
One instance is OUTDATED: this account's stack failed. Its reason, and the stack's own first failure:
$ aws cloudformation list-stack-instances --stack-set-name try-org-guard --call-as DELEGATED_ADMIN --stack-instance-account 111122223333 --query 'Summaries[0].StatusReason' --output text
ResourceLogicalId:Marker, ResourceType:AWS::SSM::Parameter, ResourceStatusReason:Resource handler returned message: "The parameter already exists. To overwrite this value, set the overwrite option in the request to true. (Service: Ssm, Status Code: 400, Request ID: e32aef55-0cb9-4879-b73b-a813f61ba647)" (RequestToken: 413285ae-1372-4994-b961-b5bbea533a41, HandlerErrorCode: AlreadyExists).
$ S=$(aws cloudformation list-stack-instances --stack-set-name try-org-guard --call-as DELEGATED_ADMIN --stack-instance-account 111122223333 --query 'Summaries[0].StackId' --output text); aws cloudformation describe-stack-events --stack-name $S --query 'reverse(StackEvents)[?ends_with(ResourceStatus,`_FAILED`)] | [0].ResourceStatusReason' --output text
Resource handler returned message: "The parameter already exists. To overwrite this value, set the overwrite option in the request to true. (Service: Ssm, Status Code: 400, Request ID: e32aef55-0cb9-4879-b73b-a813f61ba647)" (RequestToken: 413285ae-1372-4994-b961-b5bbea533a41, HandlerErrorCode: AlreadyExists)
The self-managed try-baseline already made /try/baseline/version here: two stacks cannot own one named resource. The classic StackSet failure in real organizations is the same - a resource made by hand (or by an older StackSet) under the name the template wants.
Service-managed StackSets never deploy to the management account itself - that account is handled separately, and should hold as little as possible. (In the lab, the other accounts of the organization are simulated: their instances succeed without a modelled stack.)
Around CloudFormation: CDK, RAM, Image Builder
Three neighbours the exam expects you to place:
- AWS CDK - infrastructure in TypeScript/Python/Java;
cdk synthproduces a CloudFormation template,cdk deploydeploys it as a stack (bootstrapped per account and Region withcdk bootstrap). Everything in this lesson - events, rollbacks, drift - applies unchanged; - AWS RAM (Resource Access Manager) - shares one resource (a subnet, a Transit Gateway, a Route 53 Resolver rule) with other accounts, instead of deploying a copy into each; StackSets deploy copies, RAM shares one;
- EC2 Image Builder - pipelines that build and test golden AMIs and container images on a schedule, from a recipe (base image + components) and distribute them to Regions and accounts. The AMI it produces is then a parameter (often an SSM public or private parameter) that stacks and launch templates read. (Concept here: the lab builds nothing with it.)
On the exam (SOA-C03 3.1): "provision and share resources across multiple Regions and accounts" - StackSets (self-managed: the AWSCloudFormationStackSetAdministrationRole and AWSCloudFormationStackSetExecutionRole; service-managed: Organizations, OUs, automatic deployment, delegated administrator) vs RAM for sharing one resource. Cross-stack references are Outputs with
Export+Fn::ImportValue, same Region only, and an imported export cannot be deleted or changed.
In an interview: "How would you deploy the same baseline to forty accounts in three Regions?" - "A CloudFormation StackSet with service-managed permissions from the delegated administrator account, targeting the OUs, with automatic deployment so new accounts get it, rolled out with operation preferences - a small MaxConcurrentPercentage and a FailureTolerance of zero - so a bad template stops after the first failures. Each instance is a normal stack, so a failed one is debugged with its own events."
You can now: share values between stacks with exports and imports and know why they pin each other, run cloudformation deploy safely in a pipeline, set up the two self-managed StackSet roles, deploy a StackSet to several Regions with operation preferences, read stack instance statuses, deploy to OUs with a service-managed StackSet as a delegated administrator, and place CDK, RAM and Image Builder.