OnCallReadyStart learning

AWS Lesson 37.12 · AWS IV: Operations Automation · 27 min read

Systems Manager: managed instances, Run Command and Automation runbooks

In plain words

Systems Manager is a remote control for your servers and your account. Run Command presses the same button on many machines at once. An Automation runbook is a recipe with steps - check the machine is on, clean the disk, write a note - that the remote control follows by itself, with a badge (a role) that says what it may touch.

You can ask it to do one machine at a time, to stop at the first problem, or to wait until a person says yes. The wiki page with six commands becomes a recipe anyone can run.

Systems Manager: managed instances, Run Command and Automation runbooks

Every team has a wiki page called something like "web node full disk - what to do": six commands, two of them out of date, run by hand at 3 am by whoever is on call. AWS Systems Manager (SSM) is where those pages become code. AWS II used two of its features to get a shell without SSH (Session Manager) and to run a command on a fleet (Run Command). This chapter uses the rest: the Automation runbooks in this lesson, Patch Manager, State Manager and maintenance windows in the next, and the services that start runbooks by themselves (EventBridge, AWS Config) after that.

Need to know: Systems Manager works on managed instances - machines whose SSM Agent has registered. Everything it does is described by a document: a Command document (AWS-RunShellScript, AWS-RunPatchBaseline) runs on an instance through the agent; an Automation document (a runbook: AWS-RestartEC2Instance, or your own) runs in the service and calls AWS APIs, step by step, with the permissions of a role you give it (AutomationAssumeRole). AWS ships hundreds of both (names starting AWS- and AWSConfigRemediation-); you can write your own in YAML.

Managed instances

An instance is managed when three things are true: the agent runs (it is preinstalled on Ubuntu, Amazon Linux and Windows AMIs), the instance has an instance profile whose role allows the agent to talk to the service (the managed policy AmazonSSMManagedInstanceCore), and it can reach the SSM endpoints (a public IP, a NAT gateway, or VPC interface endpoints). The setup launched three, tagged by environment and patch group:

$ cd ~/oncall-lab/labs/aws/try4
$ aws ssm describe-instance-information --query 'InstanceInformationList[].[InstanceId,PingStatus,PlatformName,PlatformVersion,AgentVersion]' --output table
--------------------------------------------------------------------
|                    DescribeInstanceInformation                   |
+----------------------+---------+---------+--------+--------------+
|  i-0e51940f154311175 |  Online |  Ubuntu |  26.04 |  3.3.5538.0  |
|  i-0063b3a8a739818a8 |  Online |  Ubuntu |  26.04 |  3.3.5538.0  |
|  i-09c564d5a0109f19a |  Online |  Ubuntu |  26.04 |  3.3.5538.0  |
+----------------------+---------+---------+--------+--------------+
$ aws ec2 describe-instances --filters Name=tag:Name,Values='try-*' Name=instance-state-name,Values=running --query 'Reservations[].Instances[].[InstanceId,Tags[?Key==`Name`]|[0].Value,Tags[?Key==`PatchGroup`]|[0].Value]' --output text
i-0e51940f154311175	try-web-1	web
i-0063b3a8a739818a8	try-web-2	web
i-09c564d5a0109f19a	try-db-1	db

An instance that is running but missing from describe-instance-information (or shows ConnectionLost) is the first thing to check when a command fails with TargetNotConnected or a runbook step never finishes: the profile, the agent, the route to the endpoints - in that order.

Run Command, the short version

AWS II ran AWS-RunShellScript on one instance. On a fleet you target tags, and two numbers decide how much damage a bad command can do: --max-concurrency (how many instances at once, a number or a percentage) and --max-errors (how many failures stop the rest):

$ C=$(aws ssm send-command --document-name AWS-RunShellScript --targets Key=tag:PatchGroup,Values=web --parameters 'commands=["df -h / | tail -1"]' --max-concurrency 1 --max-errors 0 --comment 'disk check' --query Command.CommandId --output text); sleep 5
$ aws ssm list-command-invocations --command-id $C --details --query 'CommandInvocations[].[InstanceId,Status,CommandPlugins[0].Output]' --output text
i-0e51940f154311175	Success	/dev/root   6.8G  1.9G  4.6G  29% /

i-0063b3a8a739818a8	Success	/dev/root   6.8G  1.9G  4.6G  29% /

Run Command is a remote shell with an audit trail: every invocation is in CloudTrail and list-commands, the output can go to S3 or CloudWatch Logs, and nobody needed SSH. What it does not do is decide anything: one document, one script, the same on every target.

Automation runbooks

A runbook is a list of steps, each an action: aws:executeAwsApi (call any API), aws:changeInstanceState, aws:waitForAwsResourceProperty and aws:assertAwsResourceProperty (wait for / insist on a state), aws:runCommand (a Command document on instances), aws:approve (stop until a person approves), aws:branch, aws:sleep, aws:executeScript (Python or PowerShell inline). AWS-owned runbooks cover the common jobs:

$ aws ssm list-documents --filters Key=Owner,Values=Amazon Key=DocumentType,Values=Automation --query 'DocumentIdentifiers[?starts_with(Name, `AWS-`)].Name' --output text | tr '\t' '\n' | head -12
AWS-RestartEC2Instance
AWS-StopEC2Instance
AWS-StartEC2Instance
AWS-TerminateEC2Instance
AWS-StopEC2InstanceWithApproval
AWS-ResizeInstance
AWS-CreateSnapshot
AWS-ConfigureS3BucketVersioning
AWS-DisableS3BucketPublicReadWrite
AWS-EnableS3BucketEncryption
AWS-DisablePublicAccessForSecurityGroup
AWS-DisableIncomingSSHOnPort22
$ aws ssm describe-document --name AWS-RestartEC2Instance --query 'Document.Parameters[].[Name,Type,DefaultValue]' --output table
------------------------------------------------
|               DescribeDocument               |
+-----------------------+--------------+-------+
|  AutomationAssumeRole |  String      |       |
|  InstanceId           |  StringList  |  None |
+-----------------------+--------------+-------+

AutomationAssumeRole is the role the runbook works with. Leave it empty and the runbook uses the permissions of whoever started it - fine at a terminal, impossible for a service (EventBridge, Config, a maintenance window) that starts it on its own. The role must trust ssm.amazonaws.com.

A role made for EC2 does not trust it. Here is one made that way, on purpose:

$ aws iam create-role --role-name try-automation --assume-role-policy-document file://ec2-trust.json --query Role.Arn --output text
arn:aws:iam::111122223333:role/try-automation
$ aws iam put-role-policy --role-name try-automation --policy-name ops --policy-document file://ops-policy.json
$ I=$(aws ec2 describe-instances --filters Name=tag:Name,Values=try-db-1 Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].InstanceId' --output text)
$ X=$(aws ssm start-automation-execution --document-name AWS-RestartEC2Instance --parameters InstanceId=$I,AutomationAssumeRole=arn:aws:iam::111122223333:role/try-automation --query AutomationExecutionId --output text); sleep 2
$ aws ssm get-automation-execution --automation-execution-id $X --query 'AutomationExecution.[AutomationExecutionStatus,FailureMessage]' --output text
Failed	The defined assume role is unable to be assumed.

start-automation-execution succeeded - the API only checks the document and its parameters. The execution failed before its first step. The fix is the trust policy, not the permissions:

$ aws iam update-assume-role-policy --role-name try-automation --policy-document file://ssm-trust.json
$ X=$(aws ssm start-automation-execution --document-name AWS-RestartEC2Instance --parameters InstanceId=$I,AutomationAssumeRole=arn:aws:iam::111122223333:role/try-automation --query AutomationExecutionId --output text)
$ sleep 60
$ aws ssm describe-automation-step-executions --automation-execution-id $X --query 'StepExecutions[].[StepName,Action,StepStatus]' --output table
----------------------------------------------------------
|            DescribeAutomationStepExecutions            |
+-----------------+---------------------------+----------+
|  stopInstances  |  aws:changeInstanceState  |  Success |
|  startInstances |  aws:changeInstanceState  |  Success |
+-----------------+---------------------------+----------+
$ aws ssm get-automation-execution --automation-execution-id $X --query 'AutomationExecution.[AutomationExecutionStatus,ExecutedBy]' --output text
Success	arn:aws:iam::111122223333:user/learner

Two aws:changeInstanceState steps, each waiting until EC2 reports the state: the instance really stopped and started. ExecutedBy is who started it; the API calls themselves were made by the role - CloudTrail records StopInstances by the assumed role try-automation, with ssm.amazonaws.com as the source. That split is what you want in an audit: a person asked, a narrow role acted.

Rate control and approvals

A runbook can target many resources the same way Run Command does: --targets resolves to values of one parameter (--target-parameter-name), and the execution becomes a parent with one child execution per target, run under --max-concurrency / --max-errors:

$ P=$(aws ssm start-automation-execution --document-name AWS-RestartEC2Instance --targets Key=tag:PatchGroup,Values=web --target-parameter-name InstanceId --max-concurrency 1 --max-errors 0 --parameters AutomationAssumeRole=arn:aws:iam::111122223333:role/try-automation --query AutomationExecutionId --output text)
$ sleep 120
$ aws ssm describe-automation-executions --filters Key=ParentExecutionId,Values=$P --query 'AutomationExecutionMetadataList[].[Target,AutomationExecutionStatus,ExecutionStartTime]' --output text
i-0063b3a8a739818a8	Success	2026-09-22T20:02:04.100000+00:00
i-0e51940f154311175	Success	2026-09-22T20:01:12.400000+00:00

One web node at a time (newest first in the list): the second child started only when the first had finished. That is the difference between "restart the web tier" and "take the web tier down". With --max-errors 0 the first failed child would have stopped the rest.

Some steps should not happen without a person. AWS-StopEC2InstanceWithApproval waits in an aws:approve step until enough approvers say yes (or anyone says no):

$ A=$(aws ssm start-automation-execution --document-name AWS-StopEC2InstanceWithApproval --parameters InstanceId=$I,Approvers=learner,AutomationAssumeRole=arn:aws:iam::111122223333:role/try-automation --query AutomationExecutionId --output text); sleep 3
$ aws ssm get-automation-execution --automation-execution-id $A --query 'AutomationExecution.[AutomationExecutionStatus,CurrentStepName]' --output text
Waiting	approve
$ aws ssm send-automation-signal --automation-execution-id $A --signal-type Reject --payload Comment='not during business hours'
$ sleep 3; aws ssm get-automation-execution --automation-execution-id $A --query 'AutomationExecution.AutomationExecutionStatus' --output text
Failed

A rejection fails the execution at the approval step - nothing was stopped. An approval can also notify an SNS topic (NotificationArn) so the approvers know they are needed - otherwise a runbook waits, silently, until its step times out.

Your own runbook

The wiki page "web node full disk" as a runbook: check the instance is running, vacuum the journal through Run Command, record what was done in a tag:

$ cat vacuum.yaml
schemaVersion: '0.3'
description: Free disk space on an instance - vacuum the journal, then record it in a tag
assumeRole: '{{ AutomationAssumeRole }}'
parameters:
  AutomationAssumeRole:
    type: String
    description: The role Automation assumes
  InstanceId:
    type: String
  MaxJournal:
    type: String
    default: 100M
    allowedPattern: '^[0-9]+[KMG]$'
mainSteps:
  - name: mustRun
    action: aws:assertAwsResourceProperty
    inputs:
      Service: ec2
      Api: DescribeInstances
      InstanceIds: ['{{ InstanceId }}']
      PropertySelector: $.Reservations[0].Instances[0].State.Name
      DesiredValues: [running]
  - name: vacuum
    action: aws:runCommand
    inputs:
      DocumentName: AWS-RunShellScript
      InstanceIds: ['{{ InstanceId }}']
      Parameters:
        commands: ['journalctl --vacuum-size={{ MaxJournal }}']
  - name: record
    action: aws:executeAwsApi
    inputs:
      Service: ec2
      Api: CreateTags
      Resources: ['{{ InstanceId }}']
      Tags:
        - Key: last-vacuum
          Value: '{{ automation:EXECUTION_ID }}'
    isEnd: true
$ aws ssm create-document --name try-vacuum-journal --document-type Automation --document-format YAML --content file://vacuum.yaml --query 'DocumentDescription.[Name,Status,DocumentVersion]' --output text
try-vacuum-journal	Active	1
$ V=$(aws ssm start-automation-execution --document-name try-vacuum-journal --parameters InstanceId=$I,MaxJournal=50M,AutomationAssumeRole=arn:aws:iam::111122223333:role/try-automation --query AutomationExecutionId --output text); sleep 20
$ aws ssm describe-automation-step-executions --automation-execution-id $V --query 'StepExecutions[].[StepName,StepStatus]' --output text
mustRun	Success
vacuum	Success
record	Success
$ aws ec2 describe-tags --filters Name=resource-id,Values=$I Name=key,Values=last-vacuum --query 'Tags[0].Value' --output text
3f918c2c-9a55-42a3-811a-d421879f050a

{{ InstanceId }} is a parameter, {{ automation:EXECUTION_ID }} a built-in variable, and a step's declared outputs can feed later steps as {{ stepName.OutputName }}. Parameters are checked before anything runs - names are case-sensitive, and a parameter without a default is required (allowedValues and allowedPattern are checked too):

$ aws ssm start-automation-execution --document-name try-vacuum-journal --parameters InstanceID=$I
aws: [ERROR]: An error occurred (InvalidAutomationExecutionParametersException) when calling the StartAutomationExecution operation: Parameter "InstanceID" is not defined in the document.
$ aws ssm start-automation-execution --document-name try-vacuum-journal --parameters InstanceId=$I
aws: [ERROR]: An error occurred (InvalidAutomationExecutionParametersException) when calling the StartAutomationExecution operation: Parameter "AutomationAssumeRole" is required but not provided.

update-document makes a new version (--document-version '$LATEST'), and update-document-default-version decides which version runs when nobody names one - the same "review before it becomes live" idea as a change set.

OpsCenter: where operational work items live

OpsCenter collects OpsItems - "something needs an operator" - from CloudWatch alarms (an alarm action can create one), EventBridge rules, Config, Security Hub, or by hand, with the related resources and the runbooks that can fix them:

$ OI=$(aws ssm create-ops-item --title 'try-db-1 journal filled /var' --description 'Vacuumed to 50M by try-vacuum-journal' --source try --priority 3 --operational-data '{"/aws/resources":{"Value":"[{\"arn\":\"arn:aws:ec2:eu-central-1:111122223333:instance/'$I'\"}]","Type":"SearchableString"}}' --query OpsItemId --output text)
$ aws ssm update-ops-item --ops-item-id $OI --status Resolved
$ aws ssm describe-ops-items --ops-item-filters Key=Status,Values=Resolved,Operator=Equal --query 'OpsItemSummaries[].[OpsItemId,Title,Status]' --output text
oi-8a7565f22114	try-db-1 journal filled /var	Resolved

It is a ticket queue built into the account. Teams with a real incident tool (PagerDuty, Jira) usually send alarms there instead, and use OpsCenter for what nobody else watches - the OpsItems Config and Security Hub raise.

Parameter Store and Secrets Manager

Both store configuration the same way (aws ssm get-parameter, aws secretsmanager get-secret-value), and both encrypt with KMS. Parameter Store is free for standard parameters, hierarchical (/shop/prod/db/host), with SecureString for encrypted values and {{resolve:ssm:...}} in CloudFormation. Secrets Manager costs per secret per month and adds what a password needs: rotation on a schedule, versions with stages (AWSCURRENT, AWSPREVIOUS), resource policies, and cross-Region replication. Rule of thumb: settings and feature flags in Parameter Store, credentials that should change in Secrets Manager - the secrets lesson later in this chapter does rotation in full.

On the exam (SOA-C03 1.2 and 3.2): "create or run custom and predefined Systems Manager Automation runbooks", "automate the management of existing resources": the document types, AutomationAssumeRole and its trust on ssm.amazonaws.com, rate control (targets, max-concurrency, max-errors), approvals, and which managed runbook fixes what (AWS-RestartEC2Instance, AWS-CreateSnapshot, AWS-DisablePublicAccessForSecurityGroup). SAA-C03 asks the Parameter Store vs Secrets Manager question in almost every set.

In an interview: "How would you automate a fix you keep doing by hand on EC2?" - "As a Systems Manager Automation runbook: the steps as aws:executeAwsApi, aws:runCommand and assert/wait actions, run with an AutomationAssumeRole that trusts ssm.amazonaws.com and has only the permissions the steps need. I'd run it against tags with max-concurrency and max-errors, add an aws:approve step if it is disruptive, and then let EventBridge or a CloudWatch alarm start it so nobody has to."

You can now: say what makes an instance managed and check it, run a command on a fleet with rate control, read the AWS-owned runbooks and their parameters, give a runbook a role it can assume, follow an execution step by step, run it across tagged targets one at a time, approve or reject a waiting step, write and run your own runbook, record work in OpsCenter, and choose between Parameter Store and Secrets Manager.

Why it helps

On-call work is full of repeated fixes - restart the service, free disk space, take a snapshot, close a port. Turning them into Systems Manager Automation runbooks makes them repeatable, audited and safe to run on a fleet: rate control, approvals, outputs that feed later steps, and a role with only the permissions the steps need. The same runbooks are what EventBridge, AWS Config and maintenance windows start by themselves later in this chapter, so writing and debugging them is the foundation of auto-remediation.

Commands in this lesson

cd aws sleep cat

FAQ

Why does my instance not appear in Systems Manager?

It is not a managed instance yet. Check three things in order: the instance profile's role has AmazonSSMManagedInstanceCore, the SSM Agent is running, and the instance can reach the SSM endpoints through a public IP, a NAT gateway or interface VPC endpoints. describe-instance-information lists only instances that registered.

What does "The defined assume role is unable to be assumed." mean?

The runbook's AutomationAssumeRole does not trust ssm.amazonaws.com, does not exist, or has a typo in its ARN. The start-automation-execution call itself succeeds because it only validates the document and parameters; the execution fails before its first step. Fix the role's trust policy, not its permissions.

Should I write runbooks or use the AWS ones?

Use the AWS-owned runbooks where they fit - restarting instances, creating snapshots, closing ports, configuring buckets - because they are maintained and documented. Write your own for team-specific procedures, built from the same actions, and keep them versioned and reviewed like code with update-document and a default version.

What is the difference between max-concurrency and max-errors?

max-concurrency limits how many targets run at the same time, as a number or a percentage. max-errors decides how many failed targets stop the rest of the run. Restarting a web tier with max-concurrency 1 and max-errors 0 keeps capacity up and stops at the first node that does not come back.

Is Parameter Store enough for secrets?

For secrets that never rotate, a SecureString parameter encrypted with KMS is often enough and costs nothing in the standard tier. As soon as a credential should change on a schedule, or needs versions you can roll back, a resource policy or replication, Secrets Manager is the right tool.

In an interview Mid

How would you automate a fix you keep doing by hand on EC2 instances?

As a Systems Manager Automation runbook: the steps as aws:assertAwsResourceProperty to check the instance is in the right state, aws:runCommand for the work on the machine, and aws:executeAwsApi for the AWS side, run with an AutomationAssumeRole that trusts ssm.amazonaws.com and has only the permissions the steps need. On a fleet I run it against tags with --max-concurrency and --max-errors, add an aws:approve step if it is disruptive, and let EventBridge or an alarm start it so nobody has to.

Also asked: What makes an EC2 instance a managed instance in Systems Manager? · What is the difference between a Command document and an Automation runbook? · When would you choose Parameter Store over Secrets Manager?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.