Lesson 35.18 · AWS I: CLI, IAM, S3 & KMS · 17 min read

Access keys: rotation, leaks and CloudTrail

Q: A developer pushed an access key to a public repository. What do you do?

Contain first: deactivate the key at once with aws iam update-access-key --status Inactive, as a different principal. Identify the owner and last use with get-access-key-last-used. Investigate with CloudTrail lookup-events by AccessKeyId in us-east-1 (IAM, STS) and in every Region used: new users, keys, roles, instances, policy changes, and the source IP. Remove what the attacker created, issue a replacement (ideally a role or OIDC instead of a key), delete the leaked key and only then detach the AWSCompromisedKeyQuarantineV3 policy. Removing the commit is not enough; the key is burned.

In plain words

An access key is like a permanent password on a sticky note that a program uses to log in. It never expires on its own. If someone photographs the note, they can use it from anywhere until you notice and throw it away. Rotating means writing a new note, putting it where the program looks, checking the program uses it, and only then burning the old one.

CloudTrail is the building's visitor log: it records who used which note, when, from where and what they tried, so after a leak you can see exactly what happened.

Access keys, leaks and CloudTrail

An access key is a password that never expires, works from anywhere on the internet, and is very easy to paste into the wrong place. Leaked keys are the most common way AWS accounts get broken into, and bots harvest them from public GitHub within minutes of a push. This lesson is the lifecycle of a key - create, rotate, retire - what happens when one leaks, and CloudTrail, the record you investigate with.

Need to know: a long-term key is an ID (AKIA..., not secret) plus a 40-character secret shown once, at creation. A user can have two, each Active or Inactive. get-access-key-last-used and the credential report show which keys are alive. Rotate by overlapping: create a second key, deploy it, check the old one stopped being used, deactivate, delete. When a key leaks: deactivate it first, investigate second - CloudTrail event history has 90 days of management events per Region, and IAM's events are in us-east-1.

The key's lifecycle

$ aws iam list-access-keys --user-name try-backup-bot --output table
-------------------------------------------------------------------------------------
|                                  ListAccessKeys                                   |
+-----------------------------------------------------------------------------------+
||                                AccessKeyMetadata                                ||
|+-----------------------+-----------------------------+---------+-----------------+|
||      AccessKeyId      |         CreateDate          | Status  |    UserName     ||
|+-----------------------+-----------------------------+---------+-----------------+|
||  AKIALGOOLXVECZTLWTOB |  2025-08-06T20:00:03+00:00  |  Active |  try-backup-bot ||
||  AKIAGFYOXXCC2KJIZZUI |  2026-09-19T20:00:03+00:00  |  Active |  try-backup-bot ||
|+-----------------------+-----------------------------+---------+-----------------+|
$ aws iam get-access-key-last-used --access-key-id $(aws iam list-access-keys --user-name try-backup-bot --query 'AccessKeyMetadata[0].AccessKeyId' --output text)
{
    "UserName": "try-backup-bot",
    "AccessKeyLastUsed": {
        "LastUsedDate": "2026-09-22T14:00:03+00:00",
        "ServiceName": "s3",
        "Region": "eu-central-1"
    }
}
$ aws iam get-access-key-last-used --access-key-id $(aws iam list-access-keys --user-name try-backup-bot --query 'AccessKeyMetadata[1].AccessKeyId' --output text)
{
    "UserName": "try-backup-bot",
    "AccessKeyLastUsed": {
        "ServiceName": "N/A",
        "Region": "N/A"
    }
}
$ aws iam create-access-key --user-name try-backup-bot
aws: [ERROR]: An error occurred (LimitExceeded) when calling the CreateAccessKey operation: Cannot exceed quota for AccessKeysPerUser: 2

Three facts from three commands:

list-access-keys never shows a secret: AWS does not keep it in a form it can give back. Lose the secret and the only fix is a new key.
The second key was created three days ago and never used (N/A): someone started a rotation and did not finish it - common, and dangerous, because now two keys are valid.
A user cannot have a third key (LimitExceeded). The quota is what makes rotation a dance: there is always room for exactly one new key.

The credential report is the account-wide view: one CSV row per user with password and key ages and last use. Auditors ask for it; you can read it in the shell:

$ aws iam generate-credential-report
{
    "State": "STARTED",
    "Description": "No report exists. Starting a new report generation task"
}
$ sleep 6
$ aws iam generate-credential-report
{
    "State": "COMPLETE"
}
$ aws iam get-credential-report --query Content --output text | base64 -d | cut -d, -f1,9,10,11 | column -t -s,
user            access_key_1_active  access_key_1_last_rotated  access_key_1_last_used_date
<root_account>  false                N/A                        N/A
learner         true                 2026-08-23T20:00:03+00:00  2026-09-22T20:00:10+00:00
try-backup-bot  true                 2025-08-06T20:00:03+00:00  2026-09-22T14:00:03+00:00

Rotating without an outage

The safe rotation of a key that a running system uses:

Create a second key (create-access-key) - the old one keeps working.
Deploy the new key to every place that uses the old one (the secret store, the CI variable, the config file).
Verify: get-access-key-last-used of the new key shows a recent date, of the old one stops moving. Wait at least one full cycle of the job (a nightly job: a night).
Deactivate the old key (update-access-key --status Inactive) - reversible in one command if something still breaks.
Delete it after a quiet period (delete-access-key).

Skipping step 3 is how a rotation becomes an incident at 02:00. Skipping step 5 leaves a valid credential lying around. Better than all of it: make the key unnecessary - people through IAM Identity Center, workloads through roles, CI through OIDC (the previous lesson). AWS's own guidance is to rotate keys that must exist regularly and to remove unused ones; Access Analyzer's unused access findings list keys nobody has used for a while.

When a key leaks

Bots scan public repositories, package registries and paste sites continuously; a key pushed to public GitHub is typically tried within minutes. AWS scans too. When it finds one of your keys in public, it:

sends an email ("Your AWS Access Key is Exposed") and creates an AWS Health event of type AWS_RISK_CREDENTIALS_EXPOSED, and opens a support case;
attaches the AWS managed policy AWSCompromisedKeyQuarantineV3 to the user - explicit denies on what attackers do first (creating users and keys, ec2:RunInstances, changing policies, s3:GetObject, s3:ListBucket, cloudtrail:LookupEvents...).

$ aws iam get-policy-version --policy-arn arn:aws:iam::aws:policy/AWSCompromisedKeyQuarantineV3 --version-id v3 --query 'PolicyVersion.Document.Statement[0].Action[?starts_with(@, `iam:`)] | length(@)'
26
$ aws iam get-policy-version --policy-arn arn:aws:iam::aws:policy/AWSCompromisedKeyQuarantineV3 --version-id v3 --query 'PolicyVersion.Document.Statement[0].Action' --output text | tr '\t' '\n' | grep -c .
98

The quarantine is damage limitation, not a fix: it blocks a list of actions, and anything not on the list (reading other services' data, s3:PutObject) still works with the key. Your response, in order:

Contain: deactivate the key (update-access-key --status Inactive) as a different principal - the quarantine policy denies the leaked user's own iam:UpdateAccessKey.
Investigate: CloudTrail by AccessKeyId, in every Region you use and us-east-1 (IAM, STS's global endpoint), for the window since the push. Look for what succeeded: new users, keys, roles, instances, changed policies.
Eradicate: delete what the attacker created; issue a new key (or better, replace the key with a role); delete the leaked key.
Recover and learn: confirm the service works with the new credential, check what data the key could read or write (management events do not show object reads), remove the secret from the repository - and accept that git history rewriting does not un-leak it: clones exist.

aws sts get-access-key-info tells you which account a key ID belongs to - useful when a key turns up in a log or a repo and nobody knows whose it is:

$ aws sts get-access-key-info --access-key-id $(aws iam list-access-keys --user-name try-backup-bot --query 'AccessKeyMetadata[0].AccessKeyId' --output text)
{
    "Account": "111122223333"
}
$ aws iam get-access-key-last-used --access-key-id $(aws iam list-access-keys --user-name try-backup-bot --query 'AccessKeyMetadata[0].AccessKeyId' --output text) --query UserName --output text
try-backup-bot

CloudTrail: who did what, when, from where

CloudTrail records API calls. Without any setup, every account has event history: the last 90 days of management events (control-plane calls: CreateUser, PutBucketPolicy, RunInstances, AssumeRole...) per Region, searchable with aws cloudtrail lookup-events. For longer retention, all Regions in one place, or data events (S3 object reads and writes, Lambda invocations) you create a trail that delivers log files to S3 (or use CloudTrail Lake) - AWS III.

$ aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=CreateAccessKey --query 'Events[].[EventTime,Username,EventName]' --output text
$ aws cloudtrail lookup-events --region us-east-1 --lookup-attributes AttributeKey=EventName,AttributeValue=CreateAccessKey --query 'Events[].[EventTime,Username,EventName]' --output text
2026-09-22T20:00:04+00:00	learner	CreateAccessKey

Nothing in eu-central-1, the event in us-east-1: IAM is global and its events live there. "CloudTrail shows nothing" is very often "you looked in the wrong Region". Each event carries the full record as a JSON string in CloudTrailEvent:

$ aws cloudtrail lookup-events --region us-east-1 --max-items 1 --lookup-attributes AttributeKey=EventName,AttributeValue=CreateAccessKey --query 'Events[0].CloudTrailEvent' --output text | jq '{who: .userIdentity.arn, key: .userIdentity.accessKeyId, from: .sourceIPAddress, agent: .userAgent, error: .errorCode, created: .responseElements.accessKey.accessKeyId}'
{
  "who": "arn:aws:iam::111122223333:user/learner",
  "key": "AKIAQ3EGPLAB4LEARNER",
  "from": "198.51.100.24",
  "agent": "aws-cli/2.37.10 md/awscrt#0.37.0 ua/2.1 os/linux#7.0.0-31-generic md/arch#aarch64 lang/python#3.14.6 md/pyimpl#CPython m/E,b cfg/retry-mode#standard md/installer#exe md/distrib#ubuntu.26 md/prompt#off md/command#iam.create-access-key",
  "error": "LimitExceeded",
  "created": null
}
$ aws cloudtrail lookup-events --lookup-attributes AttributeKey=Username,AttributeValue=try-backup-bot --query 'Events[].[EventTime,EventSource,EventName]' --output text
2026-09-22T14:00:03+00:00	s3.amazonaws.com	ListBuckets

The fields you read in an investigation: userIdentity (type, ARN, the access key, for roles the session issuer), eventSource + eventName (the API call), sourceIPAddress and userAgent (a laptop's CLI, a Python script, an AWS service), errorCode/errorMessage (failed attempts are recorded too - a burst of AccessDenied from an unknown IP is the signature of someone probing a stolen key), and requestParameters/responseElements.

lookup-events takes one lookup attribute at a time (EventName, Username, AccessKeyId, ResourceName, ResourceType, EventSource, ReadOnly, EventId) plus a time window (--start-time, --end-time); combine further with --query or jq. Events usually appear within about 5 minutes of the call.

Quick check

$ aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=CreateAccessKey --query 'length(Events)'

Your profile's Region is eu-central-1, and an access key was just created there. How many events?

1
0
an error: lookup-events needs --region us-east-1

Answer

0 - CreateAccessKey is an IAM call. IAM is global, so CloudTrail records it in us-east-1, whatever Region the CLI was set to.

In an interview: "A developer pushed an access key to a public repo. What do you do?" - "Contain first: deactivate the key immediately. Then investigate with CloudTrail by access key ID in every Region and us-east-1 for what it did - new users, keys, roles, instances. Delete what the attacker created, issue a replacement (ideally a role or OIDC instead of a key), delete the leaked key, and check what data it could reach. Removing the commit is not enough; the key is burned the moment it is public."

You can now: read a user's keys and their last use, run a rotation without an outage, pull the credential report, explain what AWS does with an exposed key and what you must do yourself, and search CloudTrail event history in the right Region and read an event record.

Why it helps

Leaked access keys are one of the most common causes of real AWS breaches: a key in a public repository is found by scanners within minutes, and the first move is usually to start expensive instances or create a new user for persistence. Knowing the containment order (deactivate, investigate, clean up, replace, delete) is incident-response muscle memory.

Rotation without downtime is the everyday version of the same skill, and CloudTrail lookups by access key ID, in the right Region, are how you answer "what did this credential do?" in any investigation.

Commands in this lesson

aws sleep

FAQ

Why does a user get two access keys?

So you can rotate without downtime: create the second key, deploy it everywhere, verify with get-access-key-last-used that the new key is the one in use, deactivate the old one, watch, then delete it. With one key you would have a moment with no valid credential. Two active keys for a long time is itself a finding.

What does AWS do when it finds a leaked key?

AWS scans public repositories and other sources. When it finds a key it sends a notification (an AWS Health event and email) and attaches the AWSCompromisedKeyQuarantineV3 managed policy to the user, which denies risky actions such as starting instances or creating users and keys. It is a seatbelt, not a fix: you still have to deactivate and replace the key.

Why search CloudTrail in us-east-1?

CloudTrail event history is per Region. Calls to global services such as IAM and STS (for the global endpoint) are recorded in us-east-1, regional calls in their Region. An investigation that only looks at the default Region misses half of what an attacker did. lookup-events takes one lookup attribute per call, for example AccessKeyId.

Does CloudTrail record S3 object reads?

Not in event history. The 90 days of free history contain management events only (CreateBucket, PutBucketPolicy, AssumeRole...). GetObject and PutObject are data events and are recorded only by a trail (or an event data store) that you configure to capture them, usually to an S3 bucket in a separate logging account.

Is deleting the commit enough after a key leak?

No. The key was public the moment it was pushed; clones, forks, caches and scanners already have it. Removing it from the history is good hygiene but the key must be treated as burned: deactivate it at once, check what it did, replace it, and delete it. Better still, replace keys with roles or OIDC so there is nothing to leak.

In an interview Mid

A developer pushed an access key to a public repository. What do you do?

Contain first: deactivate the key at once with aws iam update-access-key --status Inactive, as a different principal. Identify the owner and last use with get-access-key-last-used. Investigate with CloudTrail lookup-events by AccessKeyId in us-east-1 (IAM, STS) and in every Region used: new users, keys, roles, instances, policy changes, and the source IP. Remove what the attacker created, issue a replacement (ideally a role or OIDC instead of a key), delete the leaked key and only then detach the AWSCompromisedKeyQuarantineV3 policy. Removing the commit is not enough; the key is burned.

Also asked: How do you rotate an access key without downtime? · What does CloudTrail event history contain, and what not? · What is the credential report?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.