OnCallReadyStart learning

AWS Lesson 37.29 · AWS IV: Operations Automation · 19 min read

Secrets and certificates: Parameter Store, Secrets Manager rotation, KMS and ACM

In plain words

Secrets Manager is a safe for passwords that changes its own combination on a schedule and tells the door the new one. Each password has versions with labels: the current one, the previous one, and during a change the pending one. If the change goes wrong, the current label stays where it was.

ACM is a certificate shop that renews your certificates by itself, as long as the little proof it left in your DNS stays there. Both lock their contents with keys from KMS, so whoever cannot use the key cannot open the safe.

Secrets and certificates: Parameter Store, Secrets Manager rotation, KMS and ACM

The database password in a .env file on three servers, last changed when the person who set it left; the TLS certificate that expired on a Saturday because the renewal reminder went to a shared mailbox nobody reads. Both are operations problems with managed answers: Secrets Manager stores credentials and rotates them on a schedule; AWS Certificate Manager (ACM) issues public certificates and renews them by itself, as long as the validation record stays in DNS. Both lean on KMS, which AWS I introduced: whoever cannot kms:Decrypt with the key cannot read the secret, whatever the secret's own permissions say.

Need to know: a secret has versions; staging labels point at them - AWSCURRENT (what get-secret-value returns by default), AWSPREVIOUS (the one before), and during a rotation AWSPENDING. A rotation is four steps run by a Lambda function (AWS supplies them for RDS, Aurora, Redshift, DocumentDB): createSecret (a new password as AWSPENDING), setSecret (set it in the database), testSecret (log in with it), finishSecret (move AWSCURRENT). If any step fails, AWSCURRENT does not move. The application must read the secret through the API (and cache it), never copy it into a file - a rotated secret is useless to an app that read it once at boot.

Parameter Store: SecureString

For settings and the occasional secret that never rotates, Parameter Store is free (standard tier, 4 KB values) and encrypts SecureString values with KMS (alias/aws/ssm unless you name a key):

$ cd ~/oncall-lab/labs/aws/try4
$ aws ssm get-parameter --name /try/shop/api-key --query 'Parameter.[Type,Version]' --output text
SecureString	1
$ aws ssm get-parameter --name /try/shop/api-key --query Parameter.Value --output text | cut -c1-40
AQICAHhap1kxUsQdQa72v2yZR1sOUNiNbUBx/wlw
$ aws ssm get-parameter --name /try/shop/api-key --with-decryption --query Parameter.Value --output text
sk_live_51Hx9TRY0000000000000000

Without --with-decryption you get the ciphertext; with it, the caller needs ssm:GetParameter and kms:Decrypt on the key. The advanced tier (paid) allows 8 KB values and parameter policies (expiry, "notify if not changed in 90 days") - the closest Parameter Store gets to rotation.

Secrets Manager: versions and labels

$ aws secretsmanager create-secret --name try/shop/db --description 'shop app database login' --secret-string file://db.json --query '[Name,ARN]' --output text
try/shop/db	arn:aws:secretsmanager:eu-central-1:111122223333:secret:try/shop/db-Xtujok
$ aws secretsmanager get-secret-value --secret-id try/shop/db --query SecretString --output text | jq -c '{username, host}'
{"username":"shop","host":"shop-db.oncall-lab.internal"}
$ aws secretsmanager put-secret-value --secret-id try/shop/db --secret-string "$(jq -c '.password = "second-Pa55word"' db.json)" --query VersionStages
[
    "AWSCURRENT"
]
$ aws secretsmanager list-secret-version-ids --secret-id try/shop/db --query 'Versions[].[VersionId,join(`,`, VersionStages)]' --output text
79755698-8446-4f96-a2f9-3b7061123668	AWSPREVIOUS
c037f125-6c95-455a-a7d0-8b4121731802	AWSCURRENT
$ aws secretsmanager get-secret-value --secret-id try/shop/db --version-stage AWSPREVIOUS --query SecretString --output text | jq -r .password
initial-Pa55word

The ARN ends with a hyphen and six random characters: a secret deleted and created again under the same name gets a new ARN, so a policy naming the old ARN does not silently grant access to the new secret (policies use try/shop/db-?????? to match any).

Rotation

Turn rotation on with a function and a schedule. Without a function, Secrets Manager has nothing to run:

With the AWS-supplied single-user PostgreSQL function, the four steps run, and the stages move:

$ aws secretsmanager rotate-secret --secret-id try/shop/db --rotation-lambda-arn arn:aws:lambda:eu-central-1:111122223333:function:SecretsManagerRDSPostgreSQLRotationSingleUser --rotation-rules '{"ScheduleExpression":"rate(30 days)","Duration":"2h"}' --query VersionId --output text
34146a76-b942-4531-8876-952a59ce7659
$ sleep 3; aws secretsmanager list-secret-version-ids --secret-id try/shop/db --query 'Versions[].[VersionId,join(`,`, VersionStages)]' --output text
79755698-8446-4f96-a2f9-3b7061123668	AWSPREVIOUS
c037f125-6c95-455a-a7d0-8b4121731802	AWSCURRENT
34146a76-b942-4531-8876-952a59ce7659	AWSPENDING
$ sleep 20; aws secretsmanager describe-secret --secret-id try/shop/db --query '[RotationEnabled,RotationRules.ScheduleExpression,LastRotatedDate!=`null`,NextRotationDate!=`null`]' --output text
True	rate(30 days)	True	True
$ aws secretsmanager list-secret-version-ids --secret-id try/shop/db --query 'Versions[].[VersionId,join(`,`, VersionStages)]' --output text
c037f125-6c95-455a-a7d0-8b4121731802	AWSPREVIOUS
34146a76-b942-4531-8876-952a59ce7659	AWSCURRENT
$ aws secretsmanager get-secret-value --secret-id try/shop/db --query SecretString --output text | jq -r '.password | length'
32

The new password is 32 random characters. Two strategies decide what happens to the old one:

  • Single user - one database user whose password changes. Between setSecret and the moment every app has read the new value, connections with the old password fail. Fine for apps that re-read the secret on an authentication error (the AWS caching clients do); painful for apps that read it once at boot.
  • Alternating users - two users (shop, shop_clone); each rotation changes the user that is not current, so AWSPREVIOUS keeps working until the next rotation. No failed logins, at the cost of a second user and a superuser secret for the function.

For RDS and Aurora there is also managed rotation (--manage-master-user-password): RDS owns the secret, rotates it every 7 days by default, and you cannot change the function. A failed rotation is visible: the secret keeps AWSPENDING on a version, rotate-secret refuses with "A previous rotation isn't complete", and CloudTrail has a RotationFailed event (an EventBridge rule on it is the alarm you want).

KMS, policies and deletion

A secret is encrypted with aws/secretsmanager unless you name a customer managed key. Use your own key when another account must read the secret (the AWS managed key cannot be shared) or when you want the key policy as a second lock:

$ KID=$(aws kms create-key --description 'try secrets' --query KeyMetadata.KeyId --output text) && aws kms create-alias --alias-name alias/try-secrets --target-key-id $KID
$ aws secretsmanager create-secret --name try/shop/stripe --kms-key-id alias/try-secrets --secret-string 'sk_live_51Hx9TRY1111111111111111' --query Name --output text
try/shop/stripe
$ aws kms put-key-policy --key-id $KID --policy-name default --policy '{"Version":"2012-10-17","Statement":[{"Sid":"root","Effect":"Allow","Principal":{"AWS":"arn:aws:iam::111122223333:root"},"Action":"kms:*","Resource":"*"},{"Sid":"nobody-but-the-app","Effect":"Deny","Principal":{"AWS":"arn:aws:iam::111122223333:user/learner"},"Action":"kms:Decrypt","Resource":"*"}]}'
$ aws secretsmanager get-secret-value --secret-id try/shop/stripe
aws: [ERROR]: An error occurred (AccessDeniedException) when calling the GetSecretValue operation: Access to KMS is not allowed
$ aws secretsmanager describe-secret --secret-id try/shop/stripe --query '[Name,KmsKeyId]' --output text
try/shop/stripe	alias/try-secrets

describe-secret still works - only reading the value needs the key. That is the error to recognise at 3 am: "Access to KMS is not allowed" means the key refused, not Secrets Manager. A secret can also carry a resource policy (put-resource-policy; --block-public-policy refuses one that would open it to the world). Deleting is never immediate:

$ aws secretsmanager delete-secret --secret-id try/shop/stripe --recovery-window-in-days 3
aws: [ERROR]: An error occurred (InvalidParameterException) when calling the DeleteSecret operation: The RecoveryWindowInDays value must be between 7 and 30 days (inclusive).
$ aws secretsmanager delete-secret --secret-id try/shop/stripe --recovery-window-in-days 7 --query DeletionDate --output text
2026-09-29T20:00:28.800000+00:00
$ aws secretsmanager get-secret-value --secret-id try/shop/stripe
aws: [ERROR]: An error occurred (InvalidRequestException) when calling the GetSecretValue operation: You can't perform this operation on the secret because it was marked for deletion.
$ aws secretsmanager restore-secret --secret-id try/shop/stripe --query Name --output text
try/shop/stripe

--force-delete-without-recovery exists and is what the cleanup script that took production down used. CloudFormation and Terraform read secrets without copying them into templates: a dynamic reference {{resolve:secretsmanager:try/shop/db:SecretString:password}} is resolved at deploy time and never stored in the template or the stack's parameters.

ACM: certificates that renew themselves

ACM issues public certificates for free for use on AWS endpoints (load balancers, CloudFront, API Gateway); you never see the private key. You prove you control the domain by DNS validation - a CNAME ACM gives you - or by email. With DNS validation and the record left in place, renewal is automatic:

$ ZID=$(aws route53 list-hosted-zones-by-name --dns-name try.oncall-lab.example --query 'HostedZones[0].Id' --output text)
$ ARN=$(aws acm request-certificate --domain-name shop.try.oncall-lab.example --subject-alternative-names 'api.try.oncall-lab.example' --validation-method DNS --query CertificateArn --output text); sleep 6
$ aws acm describe-certificate --certificate-arn $ARN --query 'Certificate.[Status,DomainValidationOptions[].ResourceRecord.Name]' --output json
[
    "PENDING_VALIDATION",
    [
        "_3ffd92b0816d446b77645b800e569591.shop.try.oncall-lab.example.",
        "_7449333a878093b58c338c38398475b7.api.try.oncall-lab.example."
    ]
]
$ RR=$(aws acm describe-certificate --certificate-arn $ARN --query 'Certificate.DomainValidationOptions[0].ResourceRecord'); aws route53 change-resource-record-sets --hosted-zone-id $ZID --change-batch "{\"Changes\":[{\"Action\":\"UPSERT\",\"ResourceRecordSet\":{\"Name\":$(echo $RR | jq .Name),\"Type\":\"CNAME\",\"TTL\":300,\"ResourceRecords\":[{\"Value\":$(echo $RR | jq .Value)}]}}]}" --query ChangeInfo.Status --output text
PENDING
$ RR=$(aws acm describe-certificate --certificate-arn $ARN --query 'Certificate.DomainValidationOptions[1].ResourceRecord'); aws route53 change-resource-record-sets --hosted-zone-id $ZID --change-batch "{\"Changes\":[{\"Action\":\"UPSERT\",\"ResourceRecordSet\":{\"Name\":$(echo $RR | jq .Name),\"Type\":\"CNAME\",\"TTL\":300,\"ResourceRecords\":[{\"Value\":$(echo $RR | jq .Value)}]}}]}" --query ChangeInfo.Status --output text
PENDING
$ aws acm describe-certificate --certificate-arn $ARN --query 'Certificate.DomainValidationOptions[].[DomainName,ValidationStatus]' --output text
shop.try.oncall-lab.example	SUCCESS
api.try.oncall-lab.example	SUCCESS
$ sleep 30
$ aws acm describe-certificate --certificate-arn $ARN --query 'Certificate.[Status,Type,RenewalEligibility,NotBefore,NotAfter]' --output text
ISSUED	AMAZON_ISSUED	INELIGIBLE	2026-09-22T20:00:55.500000+00:00	2027-04-08T20:00:55.500000+00:00

In a script, aws acm wait certificate-validated --certificate-arn $ARN polls until it is issued (once a minute). RenewalEligibility is INELIGIBLE for now: ACM renews a certificate only while it is in use (attached to a load balancer, a distribution, an API) - an unused certificate simply expires. Each name needs its own validation record (a wildcard *.try.oncall-lab.example shares the record of try.oncall-lab.example). Public certificates are valid for about 200 days since the CA/B Forum's 2026 limit, and ACM starts renewing well before expiry: the CNAME must still resolve then, which is why deleting "that weird _3f2a... record" from the zone breaks a renewal months later (ACM sends an ACM Certificate Approaching Expiration event to EventBridge - page on it). Rules the exam likes:

  • A certificate is regional; for CloudFront it must be requested in us-east-1.
  • Imported certificates (from another CA) are stored and deployed but never renewed by ACM.
  • A certificate in use cannot be deleted (ResourceInUseException names the user).
  • AWS Private CA issues private certificates for internal names; ACM can manage and renew them.

On the exam (SOA-C03 4.2, SAA-C03 1.3): "protect data at rest and in transit": Secrets Manager vs Parameter Store (rotation is the deciding word), rotation and its Lambda, the staging labels, KMS customer managed keys for cross-account access, ACM DNS validation and automatic renewal, us-east-1 for CloudFront, and "store credentials securely for an application on EC2" = an instance role reading Secrets Manager, never keys in user data.

In an interview: "How do you handle database credentials for an app on AWS?" - "Secrets Manager, encrypted with a customer managed KMS key, with automatic rotation - alternating users if the app can't tolerate a failed login. The app gets the secret through its role at runtime and caches it with a refresh, retrying on an authentication error so a rotation never needs a restart. Nothing in files, user data or environment variables, and an alarm on RotationFailed."

You can now: store and read SecureString parameters, create secrets and read versions by stage, turn on rotation and follow AWSPENDING to AWSCURRENT, choose single-user or alternating rotation, explain what an application must do to survive one, encrypt a secret with your own key and recognise a KMS denial, delete and restore within the recovery window, reference secrets from templates, and request, validate and keep renewing ACM certificates.

Why it helps

Leaked and expired credentials cause more outages and breaches than most bugs: the password in a file from two years ago, the certificate that expired on a weekend. The lesson covers Parameter Store SecureString, Secrets Manager versions and staging labels, how rotation works step by step and what the application must do to survive it, customer managed KMS keys and recognising a KMS denial, deletion with a recovery window, dynamic references in templates, and ACM's DNS validation, managed renewal and regional rules.

Commands in this lesson

cd aws sleep

FAQ

What happens if a rotation fails halfway?

AWSCURRENT does not move, so applications keep the working password. The new version keeps the AWSPENDING label, rotate-secret refuses with "A previous rotation isn't complete", and CloudTrail records RotationFailed. Fix the cause - often network access or permissions of the rotation function - then retry, or clear AWSPENDING with update-secret-version-stage.

Single-user or alternating-users rotation?

Single user changes one user's password: simple, but connections using the old password fail until every client has read the new one. Alternating users keeps two users and changes the one that is not current, so the previous credentials stay valid until the next rotation - smoother for clients, at the cost of a second user and a superuser secret.

Why use a customer managed KMS key for a secret?

The AWS managed key aws/secretsmanager cannot be shared with other accounts and its policy cannot be changed. A customer managed key gives you a key policy as a second lock, lets another account decrypt a shared secret, and shows every decryption in CloudTrail under a key you control.

Can I delete a secret immediately?

Normally a deletion waits for a recovery window of 7 to 30 days, during which restore-secret brings it back and every other operation is refused. --force-delete-without-recovery deletes at once and cannot be undone, which is why cleanup scripts should never use it on production secrets.

Why does my ACM certificate say RenewalEligibility INELIGIBLE?

ACM only renews certificates that are in use - associated with a load balancer, a CloudFront distribution or another integrated service. An unused certificate simply expires. Once attached, renewal also needs the DNS validation records to still exist in the zone.

In an interview Mid

How do you handle database credentials for an application on AWS?

Secrets Manager, encrypted with a customer managed KMS key, with automatic rotation - alternating users if the app cannot tolerate a failed login. The app gets the secret through its instance or task role at runtime, with a caching client that refreshes and re-fetches on an authentication error, so a rotation never needs a restart. The role may read that one secret ARN and decrypt with that one key. Nothing in files, user data or environment variables, and an alarm on RotationFailed.

Also asked: What are the four steps of a Secrets Manager rotation? · How does ACM DNS validation keep a certificate renewing? · When would you choose Parameter Store SecureString over Secrets Manager?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.