OnCallReadyStart learning

AWSSecretsSRE · 5 min read

S3 AccessDenied explained: identity policy, bucket policy, KMS key or SCP?

The end of an AWS AccessDenied message names the policy type that said no. Five real S3 errors decoded, from a missing ListBucket to a KMS key and an SCP.

"AccessDenied" from S3 sounds like one problem. It is at least six, and AWS tells you which one, at the very end of the message:

terminal
$ ~/oncall-lab/labs/aws/denied/nightly.sh
aws: [ERROR]: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: User: arn:aws:iam::111122223333:user/reports-bot is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::oncall-lab-reports-111122223333" because no identity-based policy allows the s3:ListBucket action

Read it in four parts: who (user/reports-bot), what (s3:ListBucket, the IAM action, not the API name ListObjectsV2), on what (the bucket ARN, not an object), and why (because no identity-based policy allows): the part that tells you which policy to open.

How AWS decides, in one paragraph

An explicit Deny in any policy that applies wins, full stop. Otherwise the request needs an Allow from the identity's policies (or from the resource's policy, for same-account access), and it must also fit inside every ceiling on the way: the organization's service control policies (SCPs), the identity's permissions boundary, the session policy of an assumed role. And if the data is encrypted with a customer-managed KMS key, the request needs a second permission on a second resource: the key.

the message ends withthe policy to look at
because no identity-based policy allowsthe user's or role's own policies (an implicit deny)
because no permissions boundary allowsthe boundary attached to that user or role
with an explicit deny in an identity-based policya Deny statement in one of its policies
with an explicit deny in a resource-based policythe bucket policy (or the KMS key policy)
because no resource-based policy allows + kms:Decryptthe KMS key policy
with an explicit deny in a service control policyan SCP, set by the organization above your account

1. Nothing allows it: the missing ListBucket

The bot's policy allows GetObject on the objects, but listing is a bucket-level action on the bucket ARN, without /*:

terminal
$ aws iam get-user-policy --user-name reports-bot --policy-name reports-job --query PolicyDocument
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ReadDaily",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::oncall-lab-reports-111122223333/daily/2025/*"
        },
        {
            "Sid": "WriteSummaries",
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::oncall-lab-reports-111122223333/summaries/*"
        }
    ]
}

Two bugs in one policy: no s3:ListBucket on arn:aws:s3:::oncall-lab-reports-111122223333, and a daily/2025/* resource that matches none of the keys the job reads today (daily/2026-10-07.csv). A related gotcha: without s3:ListBucket, a GetObject on a key that does not exist returns 403 AccessDenied, not 404, so S3 does not leak which keys exist.

2. Allowed, but outside the boundary

After adding the List statement, the same call fails again, with a different ending:

output
aws: [ERROR]: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: User: arn:aws:iam::111122223333:user/reports-bot is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::oncall-lab-reports-111122223333" because no permissions boundary allows the s3:ListBucket action

A permissions boundary is a ceiling: the effective rights are the intersection of the policy and the boundary. Adding to the policy alone can never get past it; the boundary needs the action too.

3. An explicit deny you should not remove

Listing works now; the upload fails:

terminal
upload failed: ../../tmp/summary.txt to s3://oncall-lab-reports-111122223333/summaries/2026-10-07.txt An error occurred (AccessDenied) when calling the PutObject operation: User: arn:aws:iam::111122223333:user/reports-bot is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::oncall-lab-reports-111122223333/summaries/2026-10-07.txt" with an explicit deny in an identity-based policy, with policy ARN: arn:aws:iam::111122223333:policy/data-guardrails
$ aws iam get-policy-version --policy-arn arn:aws:iam::111122223333:policy/data-guardrails --version-id v1 --query PolicyVersion.Document.Statement
[
    {
        "Sid": "DenyUnencryptedUploads",
        "Effect": "Deny",
        "Action": "s3:PutObject",
        "Resource": "*",
        "Condition": {
            "StringNotEquals": {
                "s3:x-amz-server-side-encryption": "aws:kms"
            }
        }
    }
]

The deny has a condition: it only bites uploads that do not ask for KMS encryption. The fix is in the caller (aws s3 cp --sse aws:kms), not in the guardrail. Deleting a deny to make a job green is how guardrails disappear.

4. The KMS key said no

A different job reads an object from a bucket that was moved to a new KMS key last night. Its role policy allows s3:GetObject, and still:

terminal
$ ./export.sh
download failed: s3://oncall-lab-ledger-111122223333/exports/ledger-latest.csv to ../../../../../../tmp/ledger.csv An error occurred (AccessDenied) when calling the GetObject operation: User: arn:aws:sts::111122223333:assumed-role/ledger-export/ledger-export-job is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:eu-central-1:111122223333:key/b484357f-b585-4742-9296-adb128829332 because no resource-based policy allows the kms:Decrypt action

The S3 call failed on a KMS action. Reading an SSE-KMS object means S3 asks KMS to decrypt the data key on your behalf, as you. Unlike most resources, a KMS key's own policy must allow you, either by naming you or by delegating to IAM with a statement for the account root. This key policy did neither for ledger-export. The least-privilege fix is a statement for that role, kms:Decrypt only, with kms:ViaService set to s3.eu-central-1.amazonaws.com so the role can use the key through S3 and nothing else.

5. The organization said no

Reading works; writing the reporting copy to a bucket in another Region fails:

terminal
upload failed: ../../../../../../tmp/ledger.csv to s3://oncall-legacy-ledger-111122223333/ledger/ledger-latest.csv An error occurred (AccessDenied) when calling the PutObject operation: User: arn:aws:sts::111122223333:assumed-role/ledger-export/ledger-export-job is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::oncall-legacy-ledger-111122223333/ledger/ledger-latest.csv" with an explicit deny in a service control policy, with policy ARN: arn:aws:organizations::444455556666:policy/o-a1b2c3d4e5/service_control_policy/p-4kq7x2rguard
$ aws s3api get-bucket-location --bucket oncall-legacy-ledger-111122223333
{
    "LocationConstraint": null
}

LocationConstraint: null means us-east-1, and the policy ARN belongs to the management account 444455556666, not yours. A Region guardrail SCP denies it, and nothing in this account can override an SCP: the job moves to the approved bucket.

The checklist

  1. Read the action and the resource exactly: bucket ARN or object ARN, S3 action or KMS.
  2. Read the ending: it names the policy type.
  3. Fix the smallest thing in the right place, and never delete a Deny to make a job pass.
  4. Test with the job's own credentials (--profile), or aws iam simulate-principal-policy first.

Secret stores fail with the same "the policy looks right" feeling: compare Vault's permission denied with the right policy. A VPC endpoint policy can deny too (the message then says so), which matters once your instances reach S3 through endpoints instead of a NAT gateway. And when the error is not 403 but 502 from your load balancer, that is a different chain.

Practise it

Three AccessDenied, three different fixes walks the first three errors on one job. Incident: AccessDenied, but the policy allows it is the KMS key and the SCP together, with a change log to read. Drill: read the AccessDenied, fix the right policy deals a new combination every round, and A bucket only the finance roles can decrypt builds the key policy from the other side.

OnCallReady is free, with no ads and no tracking. RSS · All posts