Lesson 35.29 · AWS I: CLI, IAM, S3 & KMS · 22 min read

KMS: keys, key policies, grants and the kms:Decrypt gotcha

Q: What is envelope encryption, and why does KMS use it?

KMS generates a data key and returns it twice: in plaintext and encrypted under the KMS key (aws kms generate-data-key). You encrypt the data locally with the plaintext data key, discard it, and store the encrypted data key next to the data. To read, KMS decrypts only the small data key. The master key never leaves KMS, KMS only handles up to 4 KB per call, large data never travels to KMS, and every kms:Decrypt is authorised by the key policy and logged in CloudTrail. S3 with SSE-KMS does exactly this for every object.

In plain words

KMS is a bank vault where the master keys never leave the building. You cannot take a master key home; you hand the bank a small envelope and it locks or unlocks it for you, and writes every visit in a ledger. For big things, the bank gives you a fresh small key for your suitcase plus a locked copy of that small key; you lock the suitcase yourself and keep the locked copy with it.

Each vault key has its own rules on the door (the key policy) saying who may use it, and those rules come first.

KMS: keys, key policies and the kms:Decrypt gotcha

AWS Key Management Service holds the keys that protect almost everything else: S3 objects, EBS volumes, RDS databases, Secrets Manager secrets, CloudTrail logs. You never see the keys - you ask KMS to use them. That design makes KMS the one service where the resource's own policy is in charge, and it produces the most baffling AccessDenied in AWS: a role that may read the bucket, cannot read the file, and the error is about kms:Decrypt.

Need to know: a KMS key never leaves KMS's hardware security modules; you call Encrypt, Decrypt and GenerateDataKey with it. Every key has exactly one key policy, and it decides: IAM policies only count if the key policy delegates to the account (the default statement "Enable IAM User Permissions", principal arn:aws:iam::<acct>:root). S3 with SSE-KMS calls KMS as you: writing needs kms:GenerateDataKey, reading needs kms:Decrypt, on the key, in addition to the S3 permissions. Keys are regional; deleting one destroys every piece of data encrypted under it, so deletion waits 7-30 days.

Keys and aliases

$ cd ~/oncall-lab/labs/aws
$ aws kms create-key --description 'try: reports encryption' --tags TagKey=team,TagValue=platform | tee /tmp/key.json
{
    "KeyMetadata": {
        "AWSAccountId": "111122223333",
        "KeyId": "317afb64-316b-4d0f-a549-8418f71e61c9",
        "Arn": "arn:aws:kms:eu-central-1:111122223333:key/317afb64-316b-4d0f-a549-8418f71e61c9",
        "CreationDate": "2026-09-22T20:00:04.100000+00:00",
        "Enabled": true,
        "Description": "try: reports encryption",
        "KeyUsage": "ENCRYPT_DECRYPT",
        "KeyState": "Enabled",
        "Origin": "AWS_KMS",
        "KeyManager": "CUSTOMER",
        "CustomerMasterKeySpec": "SYMMETRIC_DEFAULT",
        "KeySpec": "SYMMETRIC_DEFAULT",
        "EncryptionAlgorithms": [
            "SYMMETRIC_DEFAULT"
        ],
        "MultiRegion": false
    }
}
$ KEY=$(jq -r .KeyMetadata.KeyId /tmp/key.json)
$ aws kms create-alias --alias-name alias/try-reports --target-key-id $KEY
$ aws kms describe-key --key-id alias/try-reports --query 'KeyMetadata.[KeyId,KeyState,KeyManager,KeySpec]' --output text
317afb64-316b-4d0f-a549-8418f71e61c9	Enabled	CUSTOMER	SYMMETRIC_DEFAULT
$ aws kms list-aliases --query 'Aliases[?starts_with(AliasName, `alias/try`)].[AliasName,TargetKeyId]' --output text
alias/try-lab	62310569-68ef-465b-a669-e71b43f1542e
alias/try-reports	317afb64-316b-4d0f-a549-8418f71e61c9

Customer managed keys are yours: you own the key policy, rotation, tags, deletion. About $1 a month each, plus requests.
AWS managed keys (aws/s3, aws/ebs, aws/rds...) are created by a service the first time you use its default encryption. Free, rotated every year, but their key policy is fixed: you cannot let another account use aws/s3.
An alias (alias/name) is a friendly name pointing at a key, per Region. Code should use aliases, so a key can be swapped without changing code. Aliases do not appear in key policies: permissions are on the key itself.
KeySpec SYMMETRIC_DEFAULT is an AES-256-GCM key, the kind S3, EBS and RDS use. KMS also has asymmetric (RSA, ECC) and HMAC keys for signing and MACs.

Encrypt, decrypt, and the 4 KB limit

$ aws kms encrypt --key-id alias/try-reports --plaintext 'hello'
aws: [ERROR]: Invalid base64: "hello"
$ aws kms encrypt --key-id alias/try-reports --plaintext fileb://try/db-password.txt --query CiphertextBlob --output text | base64 -d > /tmp/db-password.enc
$ aws kms decrypt --ciphertext-blob fileb:///tmp/db-password.enc --query Plaintext --output text | base64 -d
pg-orders-Z8q2-prod

The first command fails on purpose: in CLI v2 blob parameters are base64, so hello is not a valid value. Pass binary with fileb://, or set cli_binary_format = raw-in-base64-out. Outputs are base64 too - hence --output text | base64 -d. Decrypt needs no --key-id: the ciphertext says which key made it.

KMS encrypts at most 4 KB directly. For anything bigger the pattern is envelope encryption: GenerateDataKey returns a fresh data key twice - in plaintext and encrypted under the KMS key. You encrypt the data locally with the plaintext key, throw that key away, and store the encrypted data key next to the data. To read, you ask KMS to decrypt the small data key. S3, EBS and every AWS SDK's encryption client work this way.

$ aws kms generate-data-key --key-id alias/try-reports --key-spec AES_256 --query '{Plain: Plaintext, Encrypted: CiphertextBlob}' --output yaml | cut -c 1-60
Plain: QVZ2SElxdVA2d21iN3dQMU0rdEFBTVZMamVKS05wTE0=
Encrypted: KktNUzEzMTdhZmI2NC0zMTZiLTRkMGYtYTU0OS04NDE4ZjcxZ

Encryption context

An encryption context is a set of key-value pairs bound to the ciphertext: decrypt must present the same pairs, it appears in CloudTrail, and key policies can require it (kms:EncryptionContext:<key>). S3 uses aws:s3:arn = the object (or bucket) ARN.

$ aws kms encrypt --key-id alias/try-reports --plaintext fileb://try/db-password.txt --encryption-context app=orders --query CiphertextBlob --output text | base64 -d > /tmp/ctx.enc
$ aws kms decrypt --ciphertext-blob fileb:///tmp/ctx.enc --encryption-context app=orders --query Plaintext --output text | base64 -d
pg-orders-Z8q2-prod
$ aws kms decrypt --ciphertext-blob fileb:///tmp/ctx.enc --encryption-context app=payments
aws: [ERROR]: An error occurred (InvalidCiphertextException) when calling the Decrypt operation:

InvalidCiphertextException with an empty message: KMS says nothing about why, on purpose. Wrong context, wrong key, corrupted bytes all look the same.

Quick check

$ aws kms encrypt --key-id alias/try-lab --plaintext fileb:///etc/hostname --encryption-context k=1 --query CiphertextBlob --output text | base64 -d > /tmp/p.enc && aws kms decrypt --ciphertext-blob fileb:///tmp/p.enc --query Plaintext --output text

The ciphertext was made with the encryption context k=1. Decrypt is called without one. What happens?

it decrypts: the context is only metadata
InvalidCiphertextException: the context is part of what was encrypted
AccessDenied: the key policy requires the context

Answer

InvalidCiphertextException: the context is part of what was encrypted - The encryption context is authenticated data. Without the same pairs, decryption fails - and KMS does not tell you why.

The key policy decides

$ aws kms get-key-policy --key-id alias/try-reports --query Policy --output text
{
    "Version": "2012-10-17",
    "Id": "key-default-1",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::111122223333:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        }
    ]
}

That is the default key policy a key gets when you create it without --policy. Its one statement allows kms:* to the account (:root), which does not grant anything by itself: it hands the decision to IAM, so users and roles with an IAM policy allowing kms:Decrypt can decrypt. Remove that statement and IAM policies stop mattering for this key - even AdministratorAccess cannot use it. KMS protects you from doing that by accident:

$ cat try/key-policy-lockout.json
{
    "Version": "2012-10-17",
    "Id": "users-only",
    "Statement": [
        {
            "Sid": "ReportsUsersOnly",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::111122223333:user/learner"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        }
    ]
}
$ aws kms put-key-policy --key-id alias/try-reports --policy file://try/key-policy-lockout.json
aws: [ERROR]: An error occurred (MalformedPolicyDocumentException) when calling the PutKeyPolicy operation: The new key policy will not allow you to update the key policy in the future.

The lockout safety check: KMS refuses a key policy that would stop the caller from changing the key policy later (--bypass-policy-lockout-safety-check overrides it; only with a very good reason, because a key nobody can manage needs AWS Support to recover).

A typical customer managed key policy has statements for: the account (delegation to IAM), key administrators (manage, not use: kms:Create*, kms:Put*, kms:ScheduleKeyDeletion...), key users (kms:Encrypt, kms:Decrypt, kms:GenerateDataKey*, kms:DescribeKey), and services that need grants. Conditions such as kms:ViaService ("only when S3 in eu-central-1 calls on your behalf") and kms:CallerAccount narrow them further.

S3 + SSE-KMS: the kms:Decrypt gotcha

When a bucket encrypts with a KMS key, S3 calls KMS on behalf of the caller - with the caller's identity. So the caller needs permissions on both services:

Operation	S3 permission	KMS permission on the key
upload (PutObject)	`s3:PutObject`	`kms:GenerateDataKey`
download (GetObject)	`s3:GetObject`	`kms:Decrypt`
multipart upload	`s3:PutObject`	`kms:GenerateDataKey` and `kms:Decrypt`

$ aws s3 mb s3://try-reports-kms-111122223333
make_bucket: try-reports-kms-111122223333
$ aws s3api put-bucket-encryption --bucket try-reports-kms-111122223333 --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms","KMSMasterKeyID":"alias/try-reports"},"BucketKeyEnabled":true}]}'
$ echo 'q3 revenue: confidential' | aws s3 cp - s3://try-reports-kms-111122223333/q3.txt
$ aws iam create-role --role-name try-reports-reader --assume-role-policy-document file://try/trust-learner.json --query Role.Arn --output text
arn:aws:iam::111122223333:role/try-reports-reader
$ aws iam put-role-policy --role-name try-reports-reader --policy-name read --policy-document file://try/s3-read-only.json
$ aws configure set role_arn arn:aws:iam::111122223333:role/try-reports-reader --profile try-reader
$ aws configure set source_profile default --profile try-reader
$ aws s3 cp s3://try-reports-kms-111122223333/q3.txt - --profile try-reader
download failed: s3://try-reports-kms-111122223333/q3.txt to - An error occurred (AccessDenied) when calling the GetObject operation: User: arn:aws:sts::111122223333:assumed-role/try-reports-reader/botocore-session-1790107206 is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:eu-central-1:111122223333:key/317afb64-316b-4d0f-a549-8418f71e61c9 because no identity-based policy allows the kms:Decrypt action

The role may read the bucket - HeadObject worked, it needs no KMS - and the download failed with an error about kms:Decrypt, not S3. People lose hours here because they look at the bucket policy. Two fixes, depending on who owns what:

the key policy delegates to IAM (as here): add kms:Decrypt on the key's ARN to the role's policy;
the key policy does not delegate (keys owned by a security team often do not): add the role to the key policy (the message then says "because no resource-based policy allows"), or give it a grant.

$ aws iam put-role-policy --role-name try-reports-reader --policy-name decrypt --policy-document "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":\"kms:Decrypt\",\"Resource\":\"$(aws kms describe-key --key-id alias/try-reports --query KeyMetadata.Arn --output text)\",\"Condition\":{\"StringEquals\":{\"kms:ViaService\":\"s3.eu-central-1.amazonaws.com\"}}}]}"
$ aws s3 cp s3://try-reports-kms-111122223333/q3.txt - --profile try-reader
q3 revenue: confidential

With kms:ViaService the role can decrypt through S3 only - not by calling kms decrypt directly on some other ciphertext made with the same key.

Grants, rotation, deletion

Grants delegate specific operations on a key to one principal without editing the key policy (create-grant --grantee-principal ... --operations Decrypt). AWS services use them all the time (an EBS volume attached to an instance is a grant). list-grants, revoke-grant.
Rotation: enable-key-rotation creates new key material every year (or every 90-2560 days); old material stays, so old ciphertext still decrypts and nothing needs re-encrypting.
Disable a key to stop all use at once (reversible). Schedule deletion with a waiting period of 7 to 30 days (default 30); the key is unusable meanwhile, cancel-key-deletion brings it back disabled. After that, every object, volume and snapshot encrypted under it is permanently unreadable.

$ aws kms enable-key-rotation --key-id alias/try-reports --rotation-period-in-days 180
$ aws kms get-key-rotation-status --key-id alias/try-reports --query '[KeyRotationEnabled,RotationPeriodInDays]' --output text
True	180
$ aws kms schedule-key-deletion --key-id alias/try-reports --pending-window-in-days 3
aws: [ERROR]: An error occurred (ValidationException) when calling the ScheduleKeyDeletion operation: 1 validation error detected: Value '3' at 'pendingWindowInDays' failed to satisfy constraint: Member must have value greater than or equal to 7
$ aws kms disable-key --key-id alias/try-reports
$ aws s3 cp s3://try-reports-kms-111122223333/q3.txt -
download failed: s3://try-reports-kms-111122223333/q3.txt to - An error occurred (KMS.DisabledException) when calling the GetObject operation: arn:aws:kms:eu-central-1:111122223333:key/317afb64-316b-4d0f-a549-8418f71e61c9 is disabled.
$ aws kms enable-key --key-id alias/try-reports

A disabled key took the bucket's objects offline for everyone, admins included - which is exactly what you want during a breach, and exactly the outage you get when someone disables "an unused key".

In an interview: "What is envelope encryption?" - "KMS generates a data key and returns it twice: plaintext and encrypted under the KMS key. You encrypt the data locally with the plaintext data key, discard it, and store the encrypted data key with the data. To decrypt, KMS decrypts the small data key. The master key never leaves KMS, KMS only handles 4 KB at a time, and every decrypt is authorised by the key policy and logged in CloudTrail."

You can now: create keys and aliases, encrypt and decrypt from the CLI with blobs and encryption context, read a key policy and explain the delegation to IAM, avoid the lockout, diagnose an AccessDenied on kms:Decrypt behind S3 and fix it in the right place, and choose between grants, rotation, disabling and deletion.

Why it helps

Encryption at rest is a requirement in almost every company, and KMS is how AWS does it for S3, EBS, RDS and many more. The hard part is not the cryptography but the permissions: a role that can read a bucket but not decrypt its objects is one of the most common "AccessDenied but the policy allows it" incidents.

Knowing that the key policy is the root of access, that S3 calls KMS on behalf of the caller, and how disabling or deleting a key cuts off every reader at once lets you fix those incidents quickly and avoid causing them.

Commands in this lesson

cd aws cat echo

FAQ

What is the difference between an AWS managed key and a customer managed key?

An AWS managed key (alias aws/s3, aws/ebs...) is created by a service in your account; AWS controls its key policy, rotation and lifetime, and you cannot use it across accounts. A customer managed key is yours: you write its key policy, grant other accounts, enable or disable it, rotate it and schedule it for deletion.

Why does the default key policy mention the account root?

The default statement allows arn:aws:iam::<account>:root to do kms:* on the key. That delegates access to IAM: identity policies in the account can then grant use of the key. Without such a statement, IAM policies do nothing for that key and only principals named in the key policy can use it.

What is the 4 KB limit?

kms encrypt takes at most 4096 bytes of plaintext. KMS is meant to protect keys and small secrets, not files. For data use envelope encryption: generate-data-key returns a data key in plaintext and encrypted; encrypt locally with the plaintext key, keep only the encrypted copy. S3, EBS and the SDKs do this for you.

What happens if I schedule a key for deletion?

The key becomes PendingDeletion and unusable at once, and after the waiting period (7 to 30 days, 30 by default) it and everything encrypted only under it is gone forever. cancel-key-deletion during the window brings it back (disabled). Disable a key first to see what breaks before you ever schedule deletion.

What is an encryption context?

A set of non-secret key-value pairs passed with encrypt and required again, exactly, to decrypt. It binds a ciphertext to its purpose (for example the S3 object ARN), shows up in CloudTrail, and can be used in policy conditions (kms:EncryptionContext). Decrypt with a different context fails with InvalidCiphertextException.

In an interview Mid

What is envelope encryption, and why does KMS use it?

KMS generates a data key and returns it twice: in plaintext and encrypted under the KMS key (aws kms generate-data-key). You encrypt the data locally with the plaintext data key, discard it, and store the encrypted data key next to the data. To read, KMS decrypts only the small data key. The master key never leaves KMS, KMS only handles up to 4 KB per call, large data never travels to KMS, and every kms:Decrypt is authorised by the key policy and logged in CloudTrail. S3 with SSE-KMS does exactly this for every object.

Also asked: Why can a role read a bucket but not download its objects? · What is a KMS key policy, and how does it relate to IAM policies? · What happens when you disable a KMS key?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.