OnCallReadyStart learning

AWS Lesson 37.15 · AWS IV: Operations Automation · 24 min read

Patch Manager, maintenance windows and State Manager

In plain words

Patching is like getting a whole team their vaccinations. First you decide which jabs count (the baseline). Then you group people by job, because nurses need different jabs from drivers (patch groups). You check who is missing what without jabbing anyone (Scan), and you book the actual jabs at a quiet time, a few people at a time, so the hospital keeps running (a maintenance window).

Someone who got the jab but has not rested yet is not protected yet - that is a patch waiting for a reboot.

Patch Manager, maintenance windows and State Manager

"Are we patched?" is a question an auditor, a security team and a CVE announcement all ask, and "I ran apt upgrade on most of them last month" is not an answer. Patch Manager answers it per instance, against a rule you wrote down; maintenance windows decide when the disruptive part (installing, rebooting) may happen; State Manager keeps things true continuously (the inventory collected, the agent up to date, a setting applied); Inventory tells you what is installed where.

Need to know: a patch baseline says which patches are approved for an operating system (AWS ships one per OS; you can write your own). Instances pick a baseline through their patch group - the tag PatchGroup (or Patch Group) - or get the OS default. The Command document AWS-RunPatchBaseline does the work on the instance: Operation=Scan only reports, Operation=Install installs what is missing and, with the default RebootOption=RebootIfNeeded, reboots when a package needs it. The results are patch compliance: per instance, how many patches are installed, missing, failed, or installed but waiting for a reboot.

Baselines and patch groups

$ cd ~/oncall-lab/labs/aws/try4
$ aws ssm describe-patch-baselines --filters Key=OWNER,Values=AWS --query 'BaselineIdentities[?OperatingSystem==`UBUNTU`].[BaselineName,BaselineDescription,DefaultBaseline]' --output text
AWS-UbuntuDefaultPatchBaseline	Default Patch Baseline for Ubuntu Provided by AWS.	True
$ B=$(aws ssm get-default-patch-baseline --operating-system UBUNTU --query BaselineId --output text); aws ssm get-patch-baseline --baseline-id $B --query 'ApprovalRules.PatchRules[0].PatchFilterGroup.PatchFilters' --output json
[
    {
        "Key": "PRIORITY",
        "Values": [
            "Required",
            "Important",
            "Standard",
            "Optional",
            "Extra"
        ]
    }
]

On Ubuntu, a "patch" is a package update, and the predefined baseline approves updates by the package's priority (Required, Important, Standard, Optional, Extra) - there are no release dates on Ubuntu, so the "approve N days after release" delay you see for Amazon Linux or Windows is not available here. First, what the web group is missing - a scan changes nothing on the instance:

$ C=$(aws ssm send-command --document-name AWS-RunPatchBaseline --targets Key=tag:PatchGroup,Values=web --parameters Operation=Scan --query Command.CommandId --output text); sleep 10
$ aws ssm describe-instance-patch-states-for-patch-group --patch-group web --query 'InstancePatchStates[].[InstanceId,Operation,MissingCount,InstalledPendingRebootCount,FailedCount]' --output table
------------------------------------------------
|   DescribeInstancePatchStatesForPatchGroup   |
+----------------------+-------+----+----+-----+
|  i-0e51940f154311175 |  Scan |  7 |  0 |  0  |
|  i-0063b3a8a739818a8 |  Scan |  7 |  0 |  0  |
+----------------------+-------+----+----+-----+
$ W1=$(aws ec2 describe-instances --filters Name=tag:Name,Values=try-web-1 Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].InstanceId' --output text)
$ aws ssm describe-instance-patches --instance-id $W1 --filters Key=State,Values=Missing --query 'Patches[].[Title,Classification,Severity,State]' --output table
---------------------------------------------------------
|                DescribeInstancePatches                |
+--------------+----------------+------------+----------+
|  libssl3t64  |  libs          |  Required  |  Missing |
|  openssl     |  utils         |  Required  |  Missing |
|  systemd     |  admin         |  Important |  Missing |
|  libsystemd0 |  libs          |  Important |  Missing |
|  udev        |  admin         |  Important |  Missing |
|  libudev1    |  libs          |  Important |  Missing |
|  tzdata      |  localization  |  Standard  |  Missing |
+--------------+----------------+------------+----------+

The database team wants something stricter: only Required updates, never a new tzdata mid-quarter. Their own baseline, registered for their patch group:

$ DB=$(aws ssm create-patch-baseline --name try-db-baseline --operating-system UBUNTU --approval-rules 'PatchRules=[{PatchFilterGroup={PatchFilters=[{Key=PRIORITY,Values=[Required]},{Key=SECTION,Values=[*]}]}}]' --rejected-patches tzdata --description 'db: security only' --query BaselineId --output text)
$ aws ssm register-patch-baseline-for-patch-group --baseline-id $DB --patch-group db
{
    "BaselineId": "pb-06c3378e6fc2ffa4a",
    "PatchGroup": "db"
}
$ aws ssm describe-patch-groups --query 'Mappings[].[PatchGroup,BaselineIdentity.BaselineName]' --output text
db	try-db-baseline
$ D1=$(aws ec2 describe-instances --filters Name=tag:Name,Values=try-db-1 Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].InstanceId' --output text)
$ aws ssm send-command --document-name AWS-RunPatchBaseline --instance-ids $D1 --parameters Operation=Scan --query Command.CommandId --output text >/dev/null; sleep 5
$ aws ssm describe-instance-patch-states --instance-ids $D1 --query 'InstancePatchStates[0].[PatchGroup,MissingCount]' --output text
db	2
$ aws ssm describe-instance-patches --instance-id $D1 --query 'Patches[].[Title,State]' --output text
libssl3t64	Missing
openssl	Missing
systemd	InstalledOther
libsystemd0	InstalledOther
udev	InstalledOther
libudev1	InstalledOther
tzdata	InstalledRejected

The same packages, a different answer: compliance is always measured against the instance's own baseline. Updates its baseline does not approve are not "missing" - the installed package is InstalledOther, outside the baseline - and a rejected one is InstalledRejected. Two identical machines can be compliant and non-compliant at the same time, which is exactly what you want when the tiers have different rules.

Install, and the reboot question

Install is what changes machines. First a guess:

Run by hand on try-web-1, with the reboot deferred:

$ aws ssm send-command --document-name AWS-RunPatchBaseline --instance-ids $W1 --parameters Operation=Install,RebootOption=NoReboot --query Command.CommandId --output text > /tmp/cid; sleep 10
$ aws ssm get-command-invocation --command-id $(cat /tmp/cid) --instance-id $W1 --query StandardOutputContent --output text | tail -4
Setting up libudev1:arm64 (259.5-0ubuntu3.4) ...
Setting up tzdata:arm64 (2026b-0ubuntu0.26.04) ...
2026-09-22 20:00:20 INFO Report: Installed 519, InstalledPendingReboot 4, Missing 0, Failed 0
$ aws ssm describe-instance-patch-states --instance-ids $W1 --query 'InstancePatchStates[0].[Operation,InstalledCount,MissingCount,InstalledPendingRebootCount,RebootOption]' --output text
Install	519	0	4	NoReboot

InstalledPendingReboot means the packages are on disk but the running system still uses the old ones - a new systemd or kernel does nothing until the machine restarts, and compliance reports those patches as not compliant until it does. So NoReboot is not "patched", it is "patched later"; the reboot still has to happen in a window someone agreed to.

Maintenance windows: when the disruptive part may happen

A maintenance window is a schedule (cron or rate), a duration in hours, a cutoff (stop starting new tasks this many hours before the end), targets and tasks. The cron is in UTC unless the window has a ScheduleTimezone - the single most common reason a window runs at the wrong hour:

$ MW=$(aws ssm create-maintenance-window --name try-web-sunday --schedule 'cron(0 2 ? * SUN *)' --schedule-timezone Europe/Berlin --duration 3 --cutoff 1 --no-allow-unassociated-targets --query WindowId --output text)
$ T=$(aws ssm register-target-with-maintenance-window --window-id $MW --resource-type INSTANCE --targets Key=tag:PatchGroup,Values=web --name web-hosts --query WindowTargetId --output text)
$ aws ssm register-task-with-maintenance-window --window-id $MW --task-type RUN_COMMAND --task-arn AWS-RunPatchBaseline --targets Key=WindowTargetIds,Values=$T --task-invocation-parameters '{"RunCommand":{"Parameters":{"Operation":["Install"],"RebootOption":["RebootIfNeeded"]}}}' --max-concurrency 1 --max-errors 1 --priority 1 --name install --query WindowTaskId --output text
ba03c238-b92c-4699-9f36-1a3496703e21
$ aws ssm describe-maintenance-windows --query 'WindowIdentities[].[Name,Schedule,ScheduleTimezone,Duration,Cutoff,NextExecutionTime]' --output table
---------------------------------------------------------------------------------------------------
|                                   DescribeMaintenanceWindows                                    |
+---------------------+----------------------+----------------+----+-----+------------------------+
|  try-db-nightly-scan|  cron(0 3 ? * * *)   |  Europe/Berlin |  2 |  1  |  2026-09-23T01:00:00Z  |
|  try-web-sunday     |  cron(0 2 ? * SUN *) |  Europe/Berlin |  3 |  1  |  2026-09-27T00:00:00Z  |
+---------------------+----------------------+----------------+----+-----+------------------------+

Read every field before trusting a window: who (the targets: a tag on prod and staging alike patches both), what (the task's Operation and RebootOption - get-maintenance-window-task shows them), how many at once (--max-concurrency 1 keeps one web node up while the other reboots) and when (cron + time zone). The setup left one more window, a nightly scan of the db group that has already run twice:

$ OLD=$(aws ssm describe-maintenance-windows --filters Key=Name,Values=try-db-nightly-scan --query 'WindowIdentities[0].WindowId' --output text)
$ aws ssm describe-maintenance-window-executions --window-id $OLD --query 'WindowExecutions[].[Status,StartTime]' --output text
SUCCESS	2026-09-22T01:00:00.000000+00:00
SUCCESS	2026-09-21T01:00:00.000000+00:00

NextExecutionTime is always shown in UTC: 03:00 in Berlin is 01:00Z in summer time and 02:00Z in winter. A window with a ScheduleTimezone follows daylight saving time; a plain UTC cron does not, so a "03:00" window written in UTC drifts by an hour twice a year.

State Manager: keep it true

A maintenance window does something at a time. A State Manager association keeps something true: it applies a document to targets when they appear and then again on its schedule, and reports compliance per association. The classic one collects the software inventory:

$ aws ssm create-association --association-name try-inventory --name AWS-GatherSoftwareInventory --targets Key=InstanceIds,Values='*' --schedule-expression 'rate(1 day)' --query 'AssociationDescription.[Name,Overview.Status]' --output text
AWS-GatherSoftwareInventory	Success
$ sleep 5; aws ssm describe-association-executions --association-id $(aws ssm list-associations --association-filter-list key=AssociationName,value=try-inventory --query 'Associations[0].AssociationId' --output text) --query 'AssociationExecutions[0].[Status,ResourceCountByStatus]' --output text
Success	{Success=3}
$ for n in try-web-1 try-db-1; do i=$(aws ec2 describe-instances --filters Name=tag:Name,Values=$n Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].InstanceId' --output text); echo "$n $(aws ssm list-inventory-entries --instance-id $i --type-name AWS:Application --filters Key=AWS:Application.Name,Values=openssl,Type=Equal --query 'Entries[0].Version' --output text)"; done
try-web-1 3.5.5-1ubuntu3.7
try-db-1 3.5.5-1ubuntu3

Inventory answers "where is the vulnerable openssl still installed?" across the fleet without logging in anywhere: try-web-1 has the fixed version, try-db-1 does not (its baseline approves the update, but nobody has run Install there yet). Other associations worth knowing: AWS-UpdateSSMAgent weekly, AWS-RunPatchBaseline with Operation=Scan daily (so compliance is never stale), AWS-ConfigureAWSPackage to install the CloudWatch agent, and your own Command document to enforce a setting.

The fleet view

Compliance from patching and from associations lands in one place, per resource:

$ aws ssm list-resource-compliance-summaries --query 'ResourceComplianceSummaryItems[].[ResourceId,ComplianceType,Status,NonCompliantSummary.NonCompliantCount]' --output text
i-09c564d5a0109f19a	Patch	NON_COMPLIANT	2
i-0e51940f154311175	Patch	NON_COMPLIANT	4
i-0063b3a8a739818a8	Patch	NON_COMPLIANT	7
i-0e51940f154311175	Association	COMPLIANT	0
i-0063b3a8a739818a8	Association	COMPLIANT	0
i-09c564d5a0109f19a	Association	COMPLIANT	0
$ aws ssm list-compliance-summaries --query 'ComplianceSummaryItems[].[ComplianceType,CompliantSummary.CompliantCount,NonCompliantSummary.NonCompliantCount]' --output text
Patch	0	3
Association	3	0

That is the answer to "are we patched?": a number per patch group, the instances behind it, and the patches behind each instance. AWS Config can turn it into a rule (EC2_MANAGEDINSTANCE_PATCH_COMPLIANCE_STATUS_CHECK, later in this chapter), and Security Hub shows it next to everything else.

In new accounts the console's Quick Setup creates all of this from one patch policy: baselines per OS, a scan schedule, an install schedule, across accounts and Regions of an organization. Underneath it is the same: baselines, associations and AWS-RunPatchBaseline.

On the exam (SOA-C03 3.2): "Automate the management of existing resources" - Patch Manager (baselines, patch groups, Scan vs Install, RebootOption), maintenance windows (schedule, time zone, duration and cutoff, targets, tasks, concurrency), State Manager associations and Inventory. Typical question: "patch a fleet with minimal downtime" = a maintenance window with rate control behind a load balancer, one AZ or one instance at a time.

In an interview: "How do you patch a fleet of EC2 instances?" - "With Systems Manager Patch Manager: a baseline that defines what is approved, patch groups via tags so each tier gets the right baseline, a daily Scan association so compliance is always current, and the Install with RebootIfNeeded in a maintenance window in the team's time zone, max-concurrency so only part of each tier reboots at once. Then I check compliance - including InstalledPendingReboot, which is not patched yet."

You can now: read the predefined baselines and write your own, put instances in patch groups by tag, scan and install with AWS-RunPatchBaseline and read what it did, explain why a pending reboot is not compliant, build a maintenance window with targets, a task, rate control and a time zone, read its executions, keep inventory with a State Manager association, and answer "are we patched?" for a whole fleet.

Why it helps

"Are we patched?" is asked by auditors, security teams and every CVE announcement, and Patch Manager is how AWS answers it per instance against a written rule. The lesson builds the whole chain: baselines and approval rules, patch groups by tag, Scan versus Install, the reboot question and why InstalledPendingReboot is not compliant, maintenance windows with time zones, duration, cutoff and rate control, State Manager associations that keep compliance current, and Inventory for "where is the vulnerable package still installed".

Commands in this lesson

cd aws sleep

FAQ

Why can I not delay approvals by seven days on Ubuntu?

Auto-approval delays rely on a release date for each patch, and Ubuntu's package metadata does not carry one Patch Manager can use. On Ubuntu, approval rules filter by priority and section instead, and you control timing with the maintenance window - staging a few days before production.

What happens with RebootOption NoReboot?

The packages are installed, but anything that needs a restart to take effect - the kernel, systemd, core libraries - keeps running the old version. Patch Manager reports those patches as InstalledPendingReboot, and the instance stays non-compliant until it reboots. NoReboot only moves the reboot to a time you choose.

Why did my maintenance window run at the wrong hour?

A window's cron is in UTC unless the window has a ScheduleTimezone. A schedule copied from a team in another time zone, or written in local time without the zone, runs hours off. NextExecutionTime is always shown in UTC, so convert it before trusting the window.

When do I use State Manager instead of a maintenance window?

State Manager associations keep something true all the time: collect inventory daily, keep the agent updated, scan for missing patches, enforce a setting. Maintenance windows are for disruptive work at an agreed time with a duration, a cutoff and rate control, such as installing patches with reboots.

Do I need maintenance windows if I use Quick Setup patch policies?

Quick Setup creates the same building blocks for you - baselines, scan and install schedules, targets - across accounts and Regions from one policy. Underneath it still uses AWS-RunPatchBaseline and associations, so knowing the parts helps when a patch policy behaves unexpectedly.

In an interview Mid

How do you patch a fleet of EC2 instances?

With Patch Manager: a patch baseline per tier that defines what is approved, patch groups via the PatchGroup tag so each tier gets the right baseline, a daily Scan association so compliance is always current, and Install with RebootIfNeeded in a maintenance window in the team's time zone (ScheduleTimezone), with max-concurrency so only part of each tier reboots at once - staging first, production a few days later. Then I check describe-instance-patch-states, including InstalledPendingReboot, which is not patched yet.

Also asked: What is the difference between a Scan and an Install patch operation? · Why would a maintenance window run at the wrong time? · How do you find every instance that still has a vulnerable package installed?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.