Lesson 35.22 · AWS I: CLI, IAM, S3 & KMS · 16 min read

S3: buckets, keys, prefixes and the s3 commands

Q: Is S3 a filesystem?

No. S3 is an object store: a bucket holds objects under keys in a flat namespace. "Folders" are only prefixes shown with a delimiter such as /, there is no rename (a move is a copy plus a delete), no append and no partial update, so you always replace whole objects. Reads after writes and lists are strongly consistent. Bucket names are global, while the data lives in the bucket's Region. For moving data use aws s3 cp and aws s3 sync; for exact API control use aws s3api.

In plain words

S3 is like a giant warehouse of numbered boxes. Each customer rents shelves under a name that must be unique in the whole world (a bucket), and every box has a label (the key), like photos/2026/beach.jpg. The slashes in the label are just part of the text; there are no real drawers inside drawers. The warehouse can show you "all boxes whose label starts with photos/", which looks like a folder but is only a search.

You can put a box in, take a copy out, or replace it, but you cannot open a box and change one page inside.

S3: buckets, keys and the s3 commands

S3 is where AWS keeps the things that must not be lost: backups, logs, build artifacts, Terraform state, data lakes, static sites. It looks like a filesystem in the console and behaves like a key-value store underneath, and the gap between the two is where the surprises live. This lesson is the model and the everyday commands; the next two are security and data protection.

Need to know: a bucket has a name that is unique across all of AWS, and lives in one Region. It holds objects: a key (the full name, slashes included), the data (up to 5 TB) and metadata. There are no folders: logs/2026/ is a prefix, and listing with the delimiter / makes prefixes look like folders. aws s3 is the high-level file tool (cp, sync, ls, rm, mb, rb, presign); aws s3api is one command per API operation. Reads after writes are strongly consistent.

Buckets

$ cd ~/oncall-lab/labs/aws
$ aws s3 mb s3://try-status-111122223333
make_bucket: try-status-111122223333
$ aws s3api create-bucket --bucket try-status-dr-111122223333
aws: [ERROR]: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.
$ aws s3api create-bucket --bucket try-status-dr-111122223333 --create-bucket-configuration LocationConstraint=eu-central-1
{
    "Location": "http://try-status-dr-111122223333.s3.amazonaws.com/"
}
$ aws s3 mb s3://backups
make_bucket failed: s3://backups An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
$ aws s3 mb s3://Try_Status
make_bucket failed: s3://Try_Status An error occurred (InvalidBucketName) when calling the CreateBucket operation: The specified bucket is not valid.

aws s3 mb creates the bucket in the CLI's Region and fills in the LocationConstraint for you. s3api create-bucket is the raw API: outside us-east-1 you must pass the constraint, and it must match the Region you send the request to.
Names are global. backups belongs to some other AWS customer, forever. Teams put the account ID (and often the Region) in bucket names - oncall-lab-docs-111122223333 - to stay unique and to make a bucket's owner obvious.
Rules: 3-63 characters, lowercase letters, digits, hyphens and dots, starting and ending with a letter or digit, not shaped like an IP address. Avoid dots: they break HTTPS on the bucket's own hostname (a.b.s3.eu-central-1.amazonaws.com does not match the wildcard certificate *.s3.eu-central-1.amazonaws.com).
An account can have 10,000 general purpose buckets by default; deleting a bucket frees its name for anyone, which is why you rarely delete and recreate buckets that others reference.

Objects, keys and prefixes

$ aws s3 cp try/site/index.html s3://try-status-111122223333/
upload: try/site/index.html to s3://try-status-111122223333/index.html
$ aws s3 sync try/site s3://try-status-111122223333/site --exclude '.DS_Store'
upload: try/site/css/site.css to s3://try-status-111122223333/site/css/site.css
upload: try/site/img/logo.svg to s3://try-status-111122223333/site/img/logo.svg
upload: try/site/index.html to s3://try-status-111122223333/site/index.html
$ aws s3 ls s3://try-status-111122223333/
                           PRE site/
2026-09-22 20:00:04         77 index.html
$ aws s3 ls s3://try-status-111122223333/site/
                           PRE css/
                           PRE img/
2026-09-22 20:00:04         77 index.html
$ aws s3 ls s3://try-status-111122223333 --recursive --human-readable --summarize
2026-09-22 20:00:04   77 Bytes index.html
2026-09-22 20:00:04   34 Bytes site/css/site.css
2026-09-22 20:00:04   65 Bytes site/img/logo.svg
2026-09-22 20:00:04   77 Bytes site/index.html

Total Objects: 4
   Total Size: 253 Bytes

PRE site/ is not a directory: it is the common part of the keys site/index.html, site/css/site.css and site/img/logo.svg, shown because ls lists with the delimiter /. The API makes this explicit:

$ aws s3api list-objects-v2 --bucket try-status-111122223333 --prefix site/ --delimiter / --query '{files: Contents[].Key, folders: CommonPrefixes[].Prefix}'
{
    "files": [
        "site/index.html"
    ],
    "folders": [
        "site/css/",
        "site/img/"
    ]
}
$ aws s3api head-object --bucket try-status-111122223333 --key site/index.html
{
    "AcceptRanges": "bytes",
    "LastModified": "2026-09-22T20:00:04+00:00",
    "ContentLength": 77,
    "ETag": "\"816b4122092246278988f944b2861955\"",
    "ChecksumType": "FULL_OBJECT",
    "ContentType": "text/html",
    "ServerSideEncryption": "AES256",
    "Metadata": {}
}

head-object is the metadata of one object: size, ETag (a hash of the content, quoted), the content type aws s3 cp guessed from the extension, the encryption (SSE-S3, AES256, applied to every new object by default), and ChecksumType: since CLI 2.23 the CLI sends a CRC64NVME checksum with every upload and S3 verifies it. Consequences of "no folders":

"Renaming a folder" is a copy plus a delete of every object under the prefix.
An empty "folder" only exists if someone uploaded a zero-byte object whose key ends in /.
Permissions and lifecycle rules work on prefixes and tags, not folders.

sync, and what it decides

aws s3 sync copies what is new or different (the size changed, or the local file is newer) and nothing else. It is the deploy tool for static sites and the backup tool for directories:

$ aws s3 sync try/site s3://try-status-111122223333/site --exclude '.DS_Store'
$ echo '' >> try/site/index.html
$ rm try/site/img/logo.svg
$ aws s3 sync try/site s3://try-status-111122223333/site --exclude '.DS_Store' --delete --dryrun
(dryrun) upload: try/site/index.html to s3://try-status-111122223333/site/index.html
(dryrun) delete: s3://try-status-111122223333/site/img/logo.svg
$ aws s3 sync try/site s3://try-status-111122223333/site --exclude '.DS_Store' --delete
upload: try/site/index.html to s3://try-status-111122223333/site/index.html
delete: s3://try-status-111122223333/site/img/logo.svg

The first sync printed nothing - nothing had changed. --delete removes destination objects that are gone from the source; always look at --dryrun first, because a wrong source path plus --delete empties the prefix. --exclude / --include filters apply in order, the last match wins: --exclude '*' --include '*.html' uploads only HTML.

Sharing one object: presigned URLs

A presigned URL carries a signature made with your credentials in its query string. Anyone who has the URL can GET that one object until it expires - no AWS account needed, no bucket policy change:

$ aws s3 presign s3://try-status-111122223333/site/index.html --expires-in 300 > /tmp/url
$ cut -c 1-120 /tmp/url
https://try-status-111122223333.s3.eu-central-1.amazonaws.com/site/index.html?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Cre
$ curl -s "$(cat /tmp/url)"
<!doctype html>
<title>oncall-lab status</title>
<h1>All systems normal</h1>

$ curl -s -o /dev/null -w '%{http_code}\n' https://try-status-111122223333.s3.eu-central-1.amazonaws.com/site/index.html
403

presign calls nothing: it signs locally, so it "works" even for objects you may not read - the URL then fails with your AccessDenied. A URL is valid for --expires-in seconds (default 3600, at most 7 days) and no longer than the credentials that signed it: one signed with an assumed role's session dies with the session. Without a signature the same object is a 403: the bucket is private.

Quick check

$ aws s3 ls s3://oncall-lab-docs-111122223333/

The bucket has the keys README.md, runbooks/aws-access.md and runbooks/s3-public.md. What does ls print?

three lines, one per key
a PRE runbooks/ line and the README.md line
runbooks/ and README.md, both with sizes

Answer

a PRE runbooks/ line and the README.md line - ls lists one level with the delimiter /: keys under runbooks/ collapse into one common prefix.

Moving, deleting, removing buckets

$ aws s3 mv s3://try-status-111122223333/index.html s3://try-status-111122223333/old/index.html
move: s3://try-status-111122223333/index.html to s3://try-status-111122223333/old/index.html
$ aws s3 rm s3://try-status-111122223333/site/ --recursive --dryrun
(dryrun) delete: s3://try-status-111122223333/site/css/site.css
(dryrun) delete: s3://try-status-111122223333/site/index.html
$ aws s3 rb s3://try-status-111122223333
remove_bucket failed: s3://try-status-111122223333 An error occurred (BucketNotEmpty) when calling the DeleteBucket operation: The bucket you tried to delete is not empty
$ aws s3 rb s3://try-status-111122223333 --force
delete: s3://try-status-111122223333/old/index.html
delete: s3://try-status-111122223333/site/css/site.css
delete: s3://try-status-111122223333/site/index.html
remove_bucket: try-status-111122223333
$ aws s3 rb s3://try-status-dr-111122223333
remove_bucket: try-status-dr-111122223333

A bucket must be empty to be deleted. rb --force deletes the objects first - but only the current versions: on a versioned bucket (next lesson but one) old versions and delete markers remain and rb still fails.

Storage classes, briefly

Every object has a storage class: STANDARD (the default), INTELLIGENT_TIERING (moves objects between tiers by access), STANDARD_IA and ONEZONE_IA (cheaper storage, a per-GB retrieval fee, a 30-day minimum), GLACIER_IR, GLACIER (Flexible Retrieval) and DEEP_ARCHIVE (cheapest, retrieval in hours). Set it per upload (--storage-class) or move objects with lifecycle rules. All of them except the One Zone class store data across at least three AZs and are designed for 99.999999999% (eleven nines) durability.

In an interview: "Is S3 a filesystem?" - "No: it is an object store with a flat namespace of keys. 'Folders' are key prefixes shown with a delimiter, there is no rename or append, and you replace whole objects. Reads after writes are strongly consistent. Bucket names are global, the data lives in the bucket's Region."

You can now: create buckets in the right Region with valid names, explain keys versus prefixes, copy and sync with filters and --delete safely, share one object with a presigned URL, and remove buckets (and know when --force is not enough).

Why it helps

Almost every system on AWS keeps something in S3: build artifacts, backups, logs, static websites, data for analytics. Copying, syncing and listing are daily work, and mistakes are expensive: a sync with --delete in the wrong direction, a bucket created in the wrong Region, a rename that is really a copy of terabytes.

Understanding that S3 is an object store, not a filesystem, explains its behaviour: why folders appear and disappear, why there is no append, why every change is a whole new object, and why reads after writes are consistent.

Commands in this lesson

cd aws echo rm cut curl

FAQ

Why do bucket names have to be globally unique?

Because a bucket name is part of a DNS name (bucket.s3.eu-central-1.amazonaws.com) shared by every AWS customer. Names are 3-63 characters, lowercase letters, digits, hyphens and dots, starting and ending with a letter or digit. Adding the account ID and Region to the name, as this lab does, avoids collisions.

What is the difference between aws s3 and aws s3api?

aws s3 is the high-level command set (ls, cp, sync, mv, rm, mb, rb, presign) that works like a file tool and handles multipart uploads and paging for you. aws s3api is one command per S3 API operation (put-object, list-objects-v2, put-bucket-policy...) with every parameter. Use s3 for moving data, s3api for configuration and exact control.

Does aws s3 rb --force delete everything?

It deletes the current objects and then the bucket. In a versioned bucket it does not delete older versions or delete markers, so the bucket is not empty and removing it fails with BucketNotEmpty. You have to delete every version (or let a lifecycle rule expire them) first.

What is a presigned URL?

A URL that carries a signature made with your credentials, valid for a limited time (up to 7 days with long-term credentials, and never longer than the credentials themselves). Anyone with the URL can do that one operation, usually a GET of one object, without AWS credentials of their own. It is the safe way to share a file without making a bucket public.

Why did create-bucket fail with IllegalLocationConstraintException?

Outside us-east-1 the API needs --create-bucket-configuration LocationConstraint=<region> matching the endpoint's Region. aws s3 mb sets it for you from the Region you use. In us-east-1 you must not pass it at all. The bucket's Region cannot be changed later, so get it right at creation.

In an interview Junior

Is S3 a filesystem?

No. S3 is an object store: a bucket holds objects under keys in a flat namespace. "Folders" are only prefixes shown with a delimiter such as /, there is no rename (a move is a copy plus a delete), no append and no partial update, so you always replace whole objects. Reads after writes and lists are strongly consistent. Bucket names are global, while the data lives in the bucket's Region. For moving data use aws s3 cp and aws s3 sync; for exact API control use aws s3api.

Also asked: What is the difference between aws s3 and aws s3api? · How do you share one object without making the bucket public? · What are S3 storage classes used for?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.