OnCallReady

Lesson 31.14 · AWS III: CloudWatch, CloudTrail, Cost & Incidents · 16 min read

CloudTrail: event history, trails and reading a record

In plain words

CloudTrail is the building's security camera. Every time anyone - a person, a script or an AWS service - asks AWS to do something, the camera writes it down: who asked, what they asked for, from where, when, and whether it worked.

The last 90 days of the important requests are always kept for free. If you want to keep the recordings longer, or record the smaller things too, you set up a trail that copies everything into a bucket and a log group.

CloudTrail: who did what, when, from where

Half of all incidents start with a change. On AWS every change is an API call, and AWS CloudTrail records API calls: who made it (the user, the role and the session behind it), what it was, on which resource, from which IP and tool, and whether it worked. AWS I used CloudTrail's event history to chase a leaked key; this lesson reads records properly, follows an assumed role back to the person, and sets up a trail so the record lasts longer than 90 days and can be queried with Logs Insights.

Need to know: event history is free and always on: 90 days of management events (control-plane calls), per Region, searched with lookup-events - and IAM, Organizations, STS's global endpoint and the other global services are recorded in us-east-1. A trail delivers the same events (plus, if you ask, data events such as S3 object reads) as files to an S3 bucket and optionally to a CloudWatch Logs group, from every Region, kept as long as you like. A record's userIdentity says who; for an assumed role the session name is usually the person.

Event history, in the right Region

Someone deleted the role try-report-reader half an hour ago. The first search finds nothing:

$ aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteRole --query 'Events[].[EventTime,Username]' --output text
$ aws cloudtrail lookup-events --region us-east-1 --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteRole --query 'Events[].[EventTime,Username,Resources[0].ResourceName]' --output text
2026-09-22T19:30:03+00:00	[email protected]	try-report-reader

The profile's Region is eu-central-1; IAM is global and its events are in us-east-1. The row gives a time and a Username - for an assumed role, that is the session name. The full record is the JSON string in CloudTrailEvent:

$ aws cloudtrail lookup-events --region us-east-1 --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteRole --max-items 1 --query 'Events[0].CloudTrailEvent' --output text | jq '{eventTime, eventName, who: .userIdentity.arn, type: .userIdentity.type, issuer: .userIdentity.sessionContext.sessionIssuer.userName, mfa: .userIdentity.sessionContext.attributes.mfaAuthenticated, ip: .sourceIPAddress, agent: .userAgent, params: .requestParameters}'
{
  "eventTime": "2026-09-22T19:30:03Z",
  "eventName": "DeleteRole",
  "who": "arn:aws:sts::111122223333:assumed-role/AWSReservedSSO_AdministratorAccess_ef134df5b3cd1957/[email protected]",
  "type": "AssumedRole",
  "issuer": "AWSReservedSSO_AdministratorAccess_ef134df5b3cd1957",
  "mfa": "false",
  "ip": "203.0.113.77",
  "agent": "aws-cli/2.37.10 md/awscrt#0.37.0 ua/2.1 os/linux#7.0.0-31-generic md/arch#aarch64 lang/python#3.14.6 md/pyimpl#CPython m/E,b cfg/retry-mode#standard md/installer#exe md/distrib#ubuntu.26 md/prompt#off md/command#iam.delete-role",
  "params": {
    "roleName": "try-report-reader"
  }
}

How to read it:

The life of a resource

ResourceName follows one resource through its history - here, a role that was created, given a policy, stripped and deleted by two different people:

$ aws cloudtrail lookup-events --region us-east-1 --lookup-attributes AttributeKey=ResourceName,AttributeValue=try-report-reader --query 'reverse(Events)[].[EventTime,EventName,Username]' --output text
2026-09-22T19:10:03+00:00	CreateRole	[email protected]
2026-09-22T19:11:03+00:00	AttachRolePolicy	[email protected]
2026-09-22T19:29:03+00:00	DetachRolePolicy	[email protected]
2026-09-22T19:30:03+00:00	DeleteRole	[email protected]

Event history lists newest first; reverse() makes it a timeline. A role cannot be deleted while policies are attached (IAM answers DeleteConflict), so a DeleteRole is always preceded by the Detach/DeleteRolePolicy calls - and those carry the policy ARNs you need to rebuild it. lookup-events takes one attribute at a time: EventName, Username, ResourceName, ResourceType, EventSource, AccessKeyId, ReadOnly, EventId, plus --start-time / --end-time. Anything more is --query or jq:

$ aws cloudtrail lookup-events --region us-east-1 --start-time $(date -u -d '-1 hour' +%FT%TZ) --lookup-attributes AttributeKey=ReadOnly,AttributeValue=false --query 'Events[].CloudTrailEvent' --output text | jq -r 'select(.eventSource == "iam.amazonaws.com") | [.eventTime, .eventName, (.userIdentity.arn | split("/") | last)] | @tsv'
2026-09-22T19:30:03Z	DeleteRole	[email protected]
2026-09-22T19:29:03Z	DetachRolePolicy	[email protected]
2026-09-22T19:11:03Z	AttachRolePolicy	[email protected]
2026-09-22T19:10:03Z	CreateRole	[email protected]
$ aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=GetBucketPolicy --query 'Events[].CloudTrailEvent' --output text | jq -r '[.eventName, .errorCode, .errorMessage] | @tsv'
GetBucketPolicy	NoSuchBucketPolicy	The bucket policy does not exist

Event history has limits that matter on a real investigation: 90 days only, one Region per call, management events only (no S3 GetObject, no Lambda Invoke), and events typically appear about 5 minutes after the call. For more, you need a trail.

Trails

A trail writes every event as gzipped JSON files to an S3 bucket - and, if configured, to a CloudWatch Logs group, where metric filters and Logs Insights can use them:

$ aws cloudtrail describe-trails --query 'trailList[].[Name,IsMultiRegionTrail,S3BucketName,CloudWatchLogsLogGroupArn]' --output text
management-events	True	aws-cloudtrail-logs-111122223333-5f1e2c3d	arn:aws:logs:eu-central-1:111122223333:log-group:aws-cloudtrail-logs-111122223333-5f1e2c3d:*
$ aws cloudtrail get-trail-status --name management-events --query '{logging: IsLogging, started: StartLoggingTime, lastS3: LatestDeliveryTime, lastLogs: LatestCloudWatchLogsDeliveryTime}'
{
    "logging": true,
    "started": "2026-06-24T20:00:03.000000+00:00",
    "lastS3": "2026-09-22T19:56:04.700000+00:00",
    "lastLogs": "2026-09-22T19:59:04.700000+00:00"
}
$ aws cloudtrail get-event-selectors --trail-name management-events --query 'EventSelectors[0]'
{
    "ReadWriteType": "All",
    "IncludeManagementEvents": true,
    "DataResources": [],
    "ExcludeManagementEventSources": []
}

A multi-Region trail with log file validation (signed digest files: you can prove the logs were not edited) and global service events is the baseline every account should have - in an organization, one organization trail from the management account covers every account into a central log-archive bucket nobody in the member accounts can delete.

With the trail in a log group, the questions get easy - every IAM change of the last hour, by whom:

$ Q=$(aws logs start-query --log-group-name aws-cloudtrail-logs-111122223333-5f1e2c3d --start-time $(date -d '-1 hour' +%s) --end-time $(date +%s) --query-string 'filter eventSource = "iam.amazonaws.com" and eventName like /^(Create|Delete|Attach|Detach|Put|Update)/ | parse userIdentity.arn "*/*/*" as kind, role, person | stats count(*) as calls by eventName, person, sourceIPAddress | sort calls desc' --query queryId --output text); sleep 2
$ aws logs get-query-results --query-id $Q --query 'results[*][*].value' --output text
CreateRole	[email protected]	198.51.100.24	1
AttachRolePolicy	[email protected]	198.51.100.24	1
DetachRolePolicy	[email protected]	203.0.113.77	1
DeleteRole	[email protected]	203.0.113.77	1

Nested JSON fields are addressed with dots (userIdentity.arn, userIdentity.sessionContext.sessionIssuer.userName), exactly as they are named in the record, and parse works on any field, not only @message: here it cuts the session name out of the assumed-role ARN.

When the trail goes quiet

The first thing an attacker with admin rights does is stop the recording: StopLogging, DeleteTrail, or an event selector that excludes everything. Two defences, both in the next labs:

In an interview: "Someone deleted a production IAM role. How do you find out who?" - "CloudTrail: lookup-events for EventName DeleteRole in us-east-1, because IAM is global. The record's userIdentity gives the principal - for an assumed role the session name and the session issuer, which with Identity Center is the person - plus the source IP and the user agent. The earlier Detach and DeleteRolePolicy events show what it had, so it can be rebuilt; beyond 90 days the trail's files or log group have the CreateRole."

You can now: search CloudTrail event history in the right Region, read a record's userIdentity, session name, source and errors, follow a resource through its history, query a trail's log group with Logs Insights, check that a trail is logging, and explain how you would notice it being switched off.

Why it helps

"What changed?" is the question that ends most incidents, and on AWS every change is an API call that CloudTrail recorded. Reading a record - the role, the session name behind it, the IP and the tool - is how you go from "someone deleted the role" to a person, a script and a time.

The traps are the ones this lesson drills: global services are recorded in us-east-1, event history stops at 90 days, and an attacker's first move is to switch the trail off.

Commands in this lesson

aws

FAQ

Why does CloudTrail show nothing for my IAM change?

IAM is a global service and its events are recorded in us-east-1. If your CLI's Region is eu-central-1, lookup-events searches eu-central-1 and returns an empty list. Add --region us-east-1 for IAM, Organizations and the other global services.

How do I find the person behind an assumed role?

The record's userIdentity.arn has the form assumed-role/ROLE/SESSION. With IAM Identity Center, the session name is the person's user name; for CI roles it is whatever the pipeline chose, which is why a pipeline should set the run ID as its session name. The sessionContext shows the role that issued the session.

What is the difference between event history and a trail?

Event history is always on: 90 days of management events per Region, searchable with lookup-events. A trail delivers the events to S3 (and optionally CloudWatch Logs) from every Region, for as long as you keep them, and can include data events such as S3 object reads.

Are failed calls recorded?

Yes. A failed call has an errorCode and errorMessage in its record. A burst of AccessDenied errors from an unknown address is what someone probing stolen credentials looks like, so failed calls are often the most interesting ones.

How quickly do events appear?

Usually within about five minutes in event history, and a few minutes in a trail's S3 files and log group. Do not conclude that nothing happened when a call made two minutes ago is missing. In the lab they appear at once (simulator).

In an interview Mid

Someone deleted a production IAM role. How do you find out who?

CloudTrail lookup-events for EventName DeleteRole with --region us-east-1, because IAM is global. The record's userIdentity gives the principal - for an assumed role the session issuer and the session name, which with Identity Center is the person - plus the source IP and the user agent. The Detach and DeleteRolePolicy events just before show what the role had; beyond 90 days the trail's files or log group have the CreateRole.

Also asked: How would you notice that someone stopped your CloudTrail trail? · What are CloudTrail data events, and when do you need them? · Why would you send CloudTrail to a CloudWatch Logs group?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.