Secrets and certificates: Parameter Store, Secrets Manager rotation, KMS and ACM
The database password in a .env file on three servers, last changed when the person who set it left; the TLS certificate that expired on a Saturday because the renewal reminder went to a shared mailbox nobody reads. Both are operations problems with managed answers: Secrets Manager stores credentials and rotates them on a schedule; AWS Certificate Manager (ACM) issues public certificates and renews them by itself, as long as the validation record stays in DNS. Both lean on KMS, which AWS I introduced: whoever cannot kms:Decrypt with the key cannot read the secret, whatever the secret's own permissions say.
Need to know: a secret has versions; staging labels point at them - AWSCURRENT (what get-secret-value returns by default), AWSPREVIOUS (the one before), and during a rotation AWSPENDING. A rotation is four steps run by a Lambda function (AWS supplies them for RDS, Aurora, Redshift, DocumentDB): createSecret (a new password as AWSPENDING), setSecret (set it in the database), testSecret (log in with it), finishSecret (move AWSCURRENT). If any step fails, AWSCURRENT does not move. The application must read the secret through the API (and cache it), never copy it into a file - a rotated secret is useless to an app that read it once at boot.
Parameter Store: SecureString
For settings and the occasional secret that never rotates, Parameter Store is free (standard tier, 4 KB values) and encrypts SecureString values with KMS (alias/aws/ssm unless you name a key):
$ cd ~/oncall-lab/labs/aws/try4
$ aws ssm get-parameter --name /try/shop/api-key --query 'Parameter.[Type,Version]' --output text
SecureString 1
$ aws ssm get-parameter --name /try/shop/api-key --query Parameter.Value --output text | cut -c1-40
AQICAHhap1kxUsQdQa72v2yZR1sOUNiNbUBx/wlw
$ aws ssm get-parameter --name /try/shop/api-key --with-decryption --query Parameter.Value --output text
sk_live_51Hx9TRY0000000000000000
Without --with-decryption you get the ciphertext; with it, the caller needs ssm:GetParameter and kms:Decrypt on the key. The advanced tier (paid) allows 8 KB values and parameter policies (expiry, "notify if not changed in 90 days") - the closest Parameter Store gets to rotation.
Secrets Manager: versions and labels
$ aws secretsmanager create-secret --name try/shop/db --description 'shop app database login' --secret-string file://db.json --query '[Name,ARN]' --output text
try/shop/db arn:aws:secretsmanager:eu-central-1:111122223333:secret:try/shop/db-Xtujok
$ aws secretsmanager get-secret-value --secret-id try/shop/db --query SecretString --output text | jq -c '{username, host}'
{"username":"shop","host":"shop-db.oncall-lab.internal"}
$ aws secretsmanager put-secret-value --secret-id try/shop/db --secret-string "$(jq -c '.password = "second-Pa55word"' db.json)" --query VersionStages
[
"AWSCURRENT"
]
$ aws secretsmanager list-secret-version-ids --secret-id try/shop/db --query 'Versions[].[VersionId,join(`,`, VersionStages)]' --output text
79755698-8446-4f96-a2f9-3b7061123668 AWSPREVIOUS
c037f125-6c95-455a-a7d0-8b4121731802 AWSCURRENT
$ aws secretsmanager get-secret-value --secret-id try/shop/db --version-stage AWSPREVIOUS --query SecretString --output text | jq -r .password
initial-Pa55word
The ARN ends with a hyphen and six random characters: a secret deleted and created again under the same name gets a new ARN, so a policy naming the old ARN does not silently grant access to the new secret (policies use try/shop/db-?????? to match any).
Rotation
Turn rotation on with a function and a schedule. Without a function, Secrets Manager has nothing to run:
With the AWS-supplied single-user PostgreSQL function, the four steps run, and the stages move:
$ aws secretsmanager rotate-secret --secret-id try/shop/db --rotation-lambda-arn arn:aws:lambda:eu-central-1:111122223333:function:SecretsManagerRDSPostgreSQLRotationSingleUser --rotation-rules '{"ScheduleExpression":"rate(30 days)","Duration":"2h"}' --query VersionId --output text
34146a76-b942-4531-8876-952a59ce7659
$ sleep 3; aws secretsmanager list-secret-version-ids --secret-id try/shop/db --query 'Versions[].[VersionId,join(`,`, VersionStages)]' --output text
79755698-8446-4f96-a2f9-3b7061123668 AWSPREVIOUS
c037f125-6c95-455a-a7d0-8b4121731802 AWSCURRENT
34146a76-b942-4531-8876-952a59ce7659 AWSPENDING
$ sleep 20; aws secretsmanager describe-secret --secret-id try/shop/db --query '[RotationEnabled,RotationRules.ScheduleExpression,LastRotatedDate!=`null`,NextRotationDate!=`null`]' --output text
True rate(30 days) True True
$ aws secretsmanager list-secret-version-ids --secret-id try/shop/db --query 'Versions[].[VersionId,join(`,`, VersionStages)]' --output text
c037f125-6c95-455a-a7d0-8b4121731802 AWSPREVIOUS
34146a76-b942-4531-8876-952a59ce7659 AWSCURRENT
$ aws secretsmanager get-secret-value --secret-id try/shop/db --query SecretString --output text | jq -r '.password | length'
32
The new password is 32 random characters. Two strategies decide what happens to the old one:
- Single user - one database user whose password changes. Between setSecret and the moment every app has read the new value, connections with the old password fail. Fine for apps that re-read the secret on an authentication error (the AWS caching clients do); painful for apps that read it once at boot.
- Alternating users - two users (
shop,shop_clone); each rotation changes the user that is not current, so AWSPREVIOUS keeps working until the next rotation. No failed logins, at the cost of a second user and a superuser secret for the function.
For RDS and Aurora there is also managed rotation (--manage-master-user-password): RDS owns the secret, rotates it every 7 days by default, and you cannot change the function. A failed rotation is visible: the secret keeps AWSPENDING on a version, rotate-secret refuses with "A previous rotation isn't complete", and CloudTrail has a RotationFailed event (an EventBridge rule on it is the alarm you want).
KMS, policies and deletion
A secret is encrypted with aws/secretsmanager unless you name a customer managed key. Use your own key when another account must read the secret (the AWS managed key cannot be shared) or when you want the key policy as a second lock:
$ KID=$(aws kms create-key --description 'try secrets' --query KeyMetadata.KeyId --output text) && aws kms create-alias --alias-name alias/try-secrets --target-key-id $KID
$ aws secretsmanager create-secret --name try/shop/stripe --kms-key-id alias/try-secrets --secret-string 'sk_live_51Hx9TRY1111111111111111' --query Name --output text
try/shop/stripe
$ aws kms put-key-policy --key-id $KID --policy-name default --policy '{"Version":"2012-10-17","Statement":[{"Sid":"root","Effect":"Allow","Principal":{"AWS":"arn:aws:iam::111122223333:root"},"Action":"kms:*","Resource":"*"},{"Sid":"nobody-but-the-app","Effect":"Deny","Principal":{"AWS":"arn:aws:iam::111122223333:user/learner"},"Action":"kms:Decrypt","Resource":"*"}]}'
$ aws secretsmanager get-secret-value --secret-id try/shop/stripe
aws: [ERROR]: An error occurred (AccessDeniedException) when calling the GetSecretValue operation: Access to KMS is not allowed
$ aws secretsmanager describe-secret --secret-id try/shop/stripe --query '[Name,KmsKeyId]' --output text
try/shop/stripe alias/try-secrets
describe-secret still works - only reading the value needs the key. That is the error to recognise at 3 am: "Access to KMS is not allowed" means the key refused, not Secrets Manager. A secret can also carry a resource policy (put-resource-policy; --block-public-policy refuses one that would open it to the world). Deleting is never immediate:
$ aws secretsmanager delete-secret --secret-id try/shop/stripe --recovery-window-in-days 3
aws: [ERROR]: An error occurred (InvalidParameterException) when calling the DeleteSecret operation: The RecoveryWindowInDays value must be between 7 and 30 days (inclusive).
$ aws secretsmanager delete-secret --secret-id try/shop/stripe --recovery-window-in-days 7 --query DeletionDate --output text
2026-09-29T20:00:28.800000+00:00
$ aws secretsmanager get-secret-value --secret-id try/shop/stripe
aws: [ERROR]: An error occurred (InvalidRequestException) when calling the GetSecretValue operation: You can't perform this operation on the secret because it was marked for deletion.
$ aws secretsmanager restore-secret --secret-id try/shop/stripe --query Name --output text
try/shop/stripe
--force-delete-without-recovery exists and is what the cleanup script that took production down used. CloudFormation and Terraform read secrets without copying them into templates: a dynamic reference {{resolve:secretsmanager:try/shop/db:SecretString:password}} is resolved at deploy time and never stored in the template or the stack's parameters.
ACM: certificates that renew themselves
ACM issues public certificates for free for use on AWS endpoints (load balancers, CloudFront, API Gateway); you never see the private key. You prove you control the domain by DNS validation - a CNAME ACM gives you - or by email. With DNS validation and the record left in place, renewal is automatic:
$ ZID=$(aws route53 list-hosted-zones-by-name --dns-name try.oncall-lab.example --query 'HostedZones[0].Id' --output text)
$ ARN=$(aws acm request-certificate --domain-name shop.try.oncall-lab.example --subject-alternative-names 'api.try.oncall-lab.example' --validation-method DNS --query CertificateArn --output text); sleep 6
$ aws acm describe-certificate --certificate-arn $ARN --query 'Certificate.[Status,DomainValidationOptions[].ResourceRecord.Name]' --output json
[
"PENDING_VALIDATION",
[
"_3ffd92b0816d446b77645b800e569591.shop.try.oncall-lab.example.",
"_7449333a878093b58c338c38398475b7.api.try.oncall-lab.example."
]
]
$ RR=$(aws acm describe-certificate --certificate-arn $ARN --query 'Certificate.DomainValidationOptions[0].ResourceRecord'); aws route53 change-resource-record-sets --hosted-zone-id $ZID --change-batch "{\"Changes\":[{\"Action\":\"UPSERT\",\"ResourceRecordSet\":{\"Name\":$(echo $RR | jq .Name),\"Type\":\"CNAME\",\"TTL\":300,\"ResourceRecords\":[{\"Value\":$(echo $RR | jq .Value)}]}}]}" --query ChangeInfo.Status --output text
PENDING
$ RR=$(aws acm describe-certificate --certificate-arn $ARN --query 'Certificate.DomainValidationOptions[1].ResourceRecord'); aws route53 change-resource-record-sets --hosted-zone-id $ZID --change-batch "{\"Changes\":[{\"Action\":\"UPSERT\",\"ResourceRecordSet\":{\"Name\":$(echo $RR | jq .Name),\"Type\":\"CNAME\",\"TTL\":300,\"ResourceRecords\":[{\"Value\":$(echo $RR | jq .Value)}]}}]}" --query ChangeInfo.Status --output text
PENDING
$ aws acm describe-certificate --certificate-arn $ARN --query 'Certificate.DomainValidationOptions[].[DomainName,ValidationStatus]' --output text
shop.try.oncall-lab.example SUCCESS
api.try.oncall-lab.example SUCCESS
$ sleep 30
$ aws acm describe-certificate --certificate-arn $ARN --query 'Certificate.[Status,Type,RenewalEligibility,NotBefore,NotAfter]' --output text
ISSUED AMAZON_ISSUED INELIGIBLE 2026-09-22T20:00:55.500000+00:00 2027-04-08T20:00:55.500000+00:00
In a script, aws acm wait certificate-validated --certificate-arn $ARN polls until it is issued (once a minute). RenewalEligibility is INELIGIBLE for now: ACM renews a certificate only while it is in use (attached to a load balancer, a distribution, an API) - an unused certificate simply expires. Each name needs its own validation record (a wildcard *.try.oncall-lab.example shares the record of try.oncall-lab.example). Public certificates are valid for about 200 days since the CA/B Forum's 2026 limit, and ACM starts renewing well before expiry: the CNAME must still resolve then, which is why deleting "that weird _3f2a... record" from the zone breaks a renewal months later (ACM sends an ACM Certificate Approaching Expiration event to EventBridge - page on it). Rules the exam likes:
- A certificate is regional; for CloudFront it must be requested in us-east-1.
- Imported certificates (from another CA) are stored and deployed but never renewed by ACM.
- A certificate in use cannot be deleted (
ResourceInUseExceptionnames the user). - AWS Private CA issues private certificates for internal names; ACM can manage and renew them.
On the exam (SOA-C03 4.2, SAA-C03 1.3): "protect data at rest and in transit": Secrets Manager vs Parameter Store (rotation is the deciding word), rotation and its Lambda, the staging labels, KMS customer managed keys for cross-account access, ACM DNS validation and automatic renewal, us-east-1 for CloudFront, and "store credentials securely for an application on EC2" = an instance role reading Secrets Manager, never keys in user data.
In an interview: "How do you handle database credentials for an app on AWS?" - "Secrets Manager, encrypted with a customer managed KMS key, with automatic rotation - alternating users if the app can't tolerate a failed login. The app gets the secret through its role at runtime and caches it with a refresh, retrying on an authentication error so a rotation never needs a restart. Nothing in files, user data or environment variables, and an alarm on RotationFailed."
You can now: store and read SecureString parameters, create secrets and read versions by stage, turn on rotation and follow AWSPENDING to AWSCURRENT, choose single-user or alternating rotation, explain what an application must do to survive one, encrypt a secret with your own key and recognise a KMS denial, delete and restore within the recovery window, reference secrets from templates, and request, validate and keep renewing ACM certificates.