OnCallReadyStart learning

AWS Lesson 38.13 · AWS V: Data, Resilience & the Edge · 17 min read

DynamoDB in production: capacity, throttling, hot keys, TTL and backups

In plain words

Every shelf in the DynamoDB library has its own clerk, and each clerk can fetch only so many books per second. A big table has many clerks. When everyone asks for books on the same shelf, that one clerk is swamped and starts saying "come back later" while the others stand idle. That is a hot key, and hiring more clerks for the building does not help.

What helps is spreading those books over several shelves, so several clerks share the work. The rest of the lesson is about the library's housekeeping: throwing out what has expired, and keeping copies.

DynamoDB in production: capacity, throttling, hot keys, TTL and backups

DynamoDB has no servers to patch, so its incidents look different: throttling. A request is refused because the table, an index or a single partition has used up its capacity for that second. This lesson reads capacity the way DynamoDB bills it, throttles a table on purpose, finds a hot key with Contributor Insights, and covers what keeps the data safe: TTL, point-in-time recovery and global tables.

Need to know: 1 WCU = one write per second of up to 1 KB; 1 RCU = one strongly consistent read per second of up to 4 KB (or two eventually consistent ones). Provisioned tables have a fixed (or auto scaled) capacity plus up to 5 minutes of unused capacity as burst; on-demand tables scale by themselves and bill per request. Either way, one partition handles at most 1,000 WCU and 3,000 RCU per second: a key that gets all the traffic is throttled no matter what the table has. The SDKs retry throttled requests; when they give up, the caller sees ProvisionedThroughputExceededException.

Capacity, as DynamoDB counts it

$ aws dynamodb describe-table --table-name shop-carts --query 'Table.{rcu: ProvisionedThroughput.ReadCapacityUnits, wcu: ProvisionedThroughput.WriteCapacityUnits, items: ItemCount}'
{
    "rcu": 5,
    "wcu": 2,
    "items": 0
}
$ aws dynamodb put-item --table-name shop-carts --item '{"cartId": {"S": "c-1001"}, "items": {"N": "3"}}' --return-consumed-capacity TOTAL --query ConsumedCapacity
{
    "TableName": "shop-carts",
    "CapacityUnits": 1
}
$ aws dynamodb get-item --table-name shop-carts --key '{"cartId": {"S": "c-1001"}}' --return-consumed-capacity TOTAL --query ConsumedCapacity
{
    "TableName": "shop-carts",
    "CapacityUnits": 0.5
}
$ aws dynamodb get-item --table-name shop-carts --key '{"cartId": {"S": "c-1001"}}' --consistent-read --return-consumed-capacity TOTAL --query ConsumedCapacity
{
    "TableName": "shop-carts",
    "CapacityUnits": 1
}

A small item costs 1 WCU to write, 0.5 RCU to read eventually consistently, 1 RCU strongly consistently. Units round up per item: a 1.1 KB item costs 2 WCU. Transactions cost double. Every GSI that receives the write costs its own WCUs. Capacity planning is this arithmetic times requests per second.

Throttling a provisioned table

shop-carts is provisioned at 2 WCU and has no burst left. A sale starts:

$ for i in $(seq 1 8); do aws dynamodb put-item --table-name shop-carts --item '{"cartId": {"S": "c-'$i'"}, "items": {"N": "1"}}' && echo "c-$i ok"; done 2>&1 | cut -c1-150
c-1 ok
c-2 ok
c-3 ok
aws: [ERROR]: An error occurred (ProvisionedThroughputExceededException) when calling the PutItem operation (reached max retries: 2): The level of con
aws: [ERROR]: An error occurred (ProvisionedThroughputExceededException) when calling the PutItem operation (reached max retries: 2): The level of con
aws: [ERROR]: An error occurred (ProvisionedThroughputExceededException) when calling the PutItem operation (reached max retries: 2): The level of con
aws: [ERROR]: An error occurred (ProvisionedThroughputExceededException) when calling the PutItem operation (reached max retries: 2): The level of con
aws: [ERROR]: An error occurred (ProvisionedThroughputExceededException) when calling the PutItem operation (reached max retries: 2): The level of con
$ aws cloudwatch get-metric-statistics --namespace AWS/DynamoDB --metric-name WriteThrottleEvents --dimensions Name=TableName,Value=shop-carts --start-time $(date -u -d '-5 min' +%FT%TZ) --end-time $(date -u -d '+1 min' +%FT%TZ) --period 60 --statistics Sum --query 'Datapoints[].Sum' --output text
5

The first writes used the second's capacity, then the CLI retried twice and gave up: (reached max retries: 2). Applications see the same exception after their SDK's retries, usually as errors and latency. The metrics to alarm on are ThrottledRequests and Read/WriteThrottleEvents, plus ConsumedWriteCapacityUnits against what is provisioned. Three fixes, from quick to right:

$ aws dynamodb update-table --table-name shop-carts --billing-mode PAY_PER_REQUEST --query 'TableDescription.[TableStatus,BillingModeSummary.BillingMode]' --output text
UPDATING	PAY_PER_REQUEST
$ aws dynamodb put-item --table-name shop-carts --item '{"cartId": {"S": "c-9"}, "items": {"N": "1"}}' && echo ok
ok
  • On-demand (PAY_PER_REQUEST): no capacity to plan, up to twice the previous peak at once, more per request than well-used provisioned capacity. You can switch modes, with limits on how often.
  • Provisioned with auto scaling: Application Auto Scaling changes RCU/WCU towards a target utilization (70% is typical). It reacts in minutes, so a sudden spike still throttles; burst capacity covers short ones.
  • Reserved capacity for steady, predictable tables, the cheapest per unit.

A hot key throttles an on-demand table too

shop-events is on-demand. Every event of the day is written under one partition key, the date:

$ aws dynamodb describe-table --table-name shop-events --query 'Table.[BillingModeSummary.BillingMode,KeySchema[0].AttributeName]' --output text
PAY_PER_REQUEST	day
$ for i in $(seq 1 10); do aws dynamodb put-item --table-name shop-events --item '{"day": {"S": "2026-10-16"}, "ts": {"S": "09:00:0'$i'"}, "type": {"S": "click"}}' 2>&1 && echo written; done | cut -c1-100 | sort | uniq -c
      7 aws: [ERROR]: An error occurred (ProvisionedThroughputExceededException) when calling the PutItem op
      3 written
$ aws dynamodb describe-contributor-insights --table-name shop-events --query '[ContributorInsightsStatus,ContributorInsightsRuleList]' --output json
[
    "ENABLED",
    [
        "DynamoDBContributorInsights-PKC-shop-events-1790100003000",
        "DynamoDBContributorInsights-PKT-shop-events-1790100003000",
        "DynamoDBContributorInsights-SKC-shop-events-1790100003000",
        "DynamoDBContributorInsights-SKT-shop-events-1790100003000"
    ]
]
$ aws cloudwatch get-insight-rule-report --rule-name $(aws dynamodb describe-contributor-insights --table-name shop-events --query 'ContributorInsightsRuleList[?contains(@, `-PKT-`)] | [0]' --output text) --start-time $(date -u -d '-10 min' +%FT%TZ) --end-time $(date -u -d '+1 min' +%FT%TZ) --period 60 --query 'Contributors[].[Keys[0],ApproximateAggregateValue]' --output text
2026-10-16	7

On-demand removed the table's limit but not the partition's (the lab uses lab-sized numbers; the real cap is 1,000 WCU per second per partition). CloudWatch Contributor Insights (update-contributor-insights --contributor-insights-action ENABLE) ranks keys by traffic (PKC) and by throttling (PKT), which is how you find a hot key. The fix is in the key, not the capacity: write sharding spreads one logical key over several (2026-10-16#0 ... 2026-10-16#9, a random or computed suffix) and readers query all shards. Other classic hot keys: a global counter item updated on every request, a "status = PENDING" GSI partition, a celebrity user.

TTL: let DynamoDB delete what expires

$ aws dynamodb update-time-to-live --table-name shop-carts --time-to-live-specification Enabled=true,AttributeName=expiresAt
{
    "TimeToLiveSpecification": {
        "Enabled": true,
        "AttributeName": "expiresAt"
    }
}
$ aws dynamodb put-item --table-name shop-carts --item '{"cartId": {"S": "c-old"}, "items": {"N": "2"}, "expiresAt": {"N": "'$(date -u -d '-1 hour' +%s)'"}}'
$ aws dynamodb get-item --table-name shop-carts --key '{"cartId": {"S": "c-old"}}' --query 'Item.expiresAt.N' --output text
1790103604

TTL deletes items whose attribute holds an epoch second in the past, for free (no WCU). It is not exact: AWS deletes expired items within a few days, so an expired item can still be read and queries should filter on the attribute when that matters. TTL deletions appear in the table's stream with a service identity, which lets a Lambda archive them to S3.

Backups, point-in-time recovery and global tables

$ aws dynamodb describe-continuous-backups --table-name shop-orders --query 'ContinuousBackupsDescription.PointInTimeRecoveryDescription'
{
    "PointInTimeRecoveryStatus": "ENABLED",
    "RecoveryPeriodInDays": 35,
    "EarliestRestorableDateTime": "2026-09-22T00:00:03.000000+00:00",
    "LatestRestorableDateTime": "2026-09-22T19:55:04.000000+00:00"
}
$ aws dynamodb create-backup --table-name shop-orders --backup-name shop-orders-before-migration --query 'BackupDetails.[BackupName,BackupStatus,BackupType]' --output text
shop-orders-before-migration	AVAILABLE	USER
$ aws dynamodb restore-table-to-point-in-time --source-table-name shop-orders --target-table-name try-orders-restored --use-latest-restorable-time --query 'TableDescription.[TableName,TableStatus]' --output text
try-orders-restored	CREATING
  • Point-in-time recovery (PITR): continuous backups, restore to any second in the last 35 days (configurable 1-35) up to about 5 minutes ago. Like RDS, the restore is a new table. It does not copy the table's auto scaling, IAM policies, alarms, TTL or stream settings, so you set those again.
  • On-demand backups (create-backup) are kept until deleted, for before-a-migration copies and long retention. AWS Backup schedules them, copies them to other Regions and accounts, and can lock them.
  • Global tables: update-table --replica-updates adds a replica in another Region. Every Region is writable, replication is asynchronous (usually under a second), and concurrent writes to the same item resolve as last writer wins. That gives multi-Region active-active with an RPO of seconds. It needs streams, and it multiplies the write cost by the number of Regions.

On the exam (SOA-C03 2.1 and 2.3, SAA-C03 3.3): "unpredictable traffic, no capacity planning" = on-demand; "steady traffic, lowest cost" = provisioned + auto scaling (and reserved capacity). "Throttling on one key" = partition key design / write sharding, not more capacity. "Delete old items automatically at no cost" = TTL. "Recover from an accidental delete" = PITR (new table). "Multi-Region, low-latency, active-active" = global tables. "Microsecond reads" = DAX.

In an interview: "DynamoDB is throttling. How do you investigate?" - "First which kind: table, GSI or partition. Compare consumed against provisioned capacity and look at ThrottledRequests per operation. A GSI with too little capacity throttles writes to the base table. If the table is far below its capacity, it is a hot partition: Contributor Insights shows the key. Then fix the cause: on-demand or auto scaling for capacity, key design or write sharding for hot keys, and retries with backoff in the client."

You can now: compute RCU and WCU for a request, recognise and measure throttling, choose between on-demand, provisioned and auto scaling, find a hot key with Contributor Insights and fix it with sharding, use TTL, and restore a table to a point in time. You can also say what global tables trade for multi-Region writes.

Why it helps

DynamoDB has no servers to patch, so its incidents are throttling: a request refused because the table, an index or a single partition used up its capacity for that second. Reading capacity the way DynamoDB bills it, choosing between on-demand and provisioned, and finding a hot key with Contributor Insights are the on-call skills.

TTL, point-in-time recovery and global tables are the other half: what deletes data for you, what brings it back, and what keeps it in more than one Region.

Commands in this lesson

aws

FAQ

How much capacity does one request use?

One WCU writes up to 1 KB; one RCU reads up to 4 KB strongly consistently, or two reads eventually consistently. Units round up per item, so a 1.1 KB item costs 2 WCU. Transactions cost double, and every GSI that receives the write uses its own WCUs. Capacity planning is that arithmetic times requests per second.

Should a table be on-demand or provisioned?

On-demand (PAY_PER_REQUEST) needs no planning, handles up to twice the previous peak at once and bills per request: good for unknown or spiky traffic. Provisioned with auto scaling is cheaper for steady traffic, but scaling reacts in minutes, so sudden spikes still throttle once the burst capacity is spent. Reserved capacity is the cheapest for predictable tables.

Why is my on-demand table throttled when traffic is low?

A hot partition. On-demand removes the table's limit but not the partition's: one partition serves at most 1,000 WCU and 3,000 RCU per second. If one key gets most of the writes, such as a date used as the partition key, it throttles. Contributor Insights ranks keys by traffic and throttling; write sharding with a suffix spreads the key.

Does TTL delete items exactly when they expire?

No. TTL deletes items whose attribute holds an epoch second in the past, at no WCU cost, but within a few days rather than on the second. An expired item can still be read, so queries should filter on the attribute when that matters. TTL deletions appear in the table's stream with a service identity, so they can be archived.

How does point-in-time recovery work for DynamoDB?

PITR keeps continuous backups for 1 to 35 days and restores to any second up to about five minutes ago. Like RDS, the restore is a new table, and it does not carry over auto scaling, IAM policies, alarms, TTL or stream settings, so you set those again. create-backup makes on-demand backups that stay until deleted.

In an interview Mid

DynamoDB is throttling. How do you investigate?

First which kind: the table, a GSI or a single partition. Compare consumed against provisioned capacity and look at ThrottledRequests and the read and write throttle events per operation. A GSI with too little capacity throttles writes to the base table. If the table is far below its capacity, it is a hot partition, and CloudWatch Contributor Insights shows the key. Then fix the cause: on-demand or auto scaling for capacity, key design or write sharding for hot keys, and retries with backoff in the client.

Also asked: What is a hot partition in DynamoDB, and how do you fix it? · When would you choose on-demand capacity over provisioned capacity? · How do DynamoDB global tables resolve conflicting writes?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.