OnCallReadyStart learning

AWS Lesson 39.6 · AWS VI: Architecting on AWS & the exams · 19 min read

Lambda behind a queue: event source mappings, partial batches and destinations

In plain words

Picture a mail room that collects letters from a mailbox for a clerk, ten at a time. If the clerk hands back the whole bundle saying "I could not do this", every letter goes back in the mailbox, even the nine that were fine, and they get handled twice. A better clerk returns only the one bad letter. After a few tries, that bad letter is put in a separate tray. The mail room is the event source mapping, the bundle is a batch, and the separate tray is the dead-letter queue.

Lambda behind a queue: event source mappings, partial batches and destinations

The most common serverless shape in real systems is not an API - it is a queue with a function behind it. Orders, emails, thumbnails, webhooks: the producer drops a message in SQS and Lambda works through the backlog. Lambda does the polling for you through an event source mapping, and almost every production problem with this shape comes from three settings: the visibility timeout, how a batch fails, and where failures end up.

Need to know: an event source mapping polls the queue, receives up to BatchSize messages (10 by default for SQS, up to 10,000 with a batching window) and invokes the function once per batch with event.Records. If the function returns normally, the mapping deletes the whole batch; if it throws, nothing is deleted and the whole batch comes back after the visibility timeout. With ReportBatchItemFailures the function returns the ids that failed and only those come back. The queue's redrive policy still decides when a message gives up and goes to the DLQ.

Connect the queue

try-orders has a DLQ (maxReceiveCount 2) and a visibility timeout of 10 seconds; try-worker has a timeout of 30 seconds:

$ cd ~/oncall-lab/labs/aws/try6
$ cat worker/index.mjs
export const handler = async (event) => {
  if (!event.Records) throw new Error('not an SQS event: ' + JSON.stringify(event));
  const batchItemFailures = [];
  for (const r of event.Records) {
    const order = JSON.parse(r.body);
    if (order.bad) {
      console.log('cannot process order', order.id);
      if (process.env.PARTIAL === 'on') { batchItemFailures.push({ itemIdentifier: r.messageId }); continue; }
      throw new Error('order ' + order.id + ' is malformed');
    }
    console.log('processed order', order.id);
  }
  return { batchItemFailures };
};
$ aws lambda create-event-source-mapping --function-name try-worker --event-source-arn arn:aws:sqs:eu-central-1:111122223333:try-orders --batch-size 10
aws: [ERROR]: An error occurred (InvalidParameterValueException) when calling the CreateEventSourceMapping operation: Queue visibility timeout: 10 seconds is less than Function timeout: 30 seconds

Lambda refuses: while the function may still be working on a batch, the messages must stay invisible, or a second poller would hand them to another copy of the function. AWS recommends a visibility timeout of at least six times the function timeout (room for retries of a throttled batch). Fix the queue, not the function:

$ Q=$(aws sqs get-queue-url --queue-name try-orders --query QueueUrl --output text)
$ aws sqs set-queue-attributes --queue-url $Q --attributes VisibilityTimeout=180
$ aws lambda create-event-source-mapping --function-name try-worker --event-source-arn arn:aws:sqs:eu-central-1:111122223333:try-orders --batch-size 10 --query '[UUID,State,BatchSize]' --output text
d513291c-f36f-4168-bd5e-597c624f59ef	Creating	10
$ sleep 5
$ aws lambda list-event-source-mappings --function-name try-worker --query 'EventSourceMappings[0].[State,StateTransitionReason]' --output text
Enabled	USER_INITIATED

The execution role needs sqs:ReceiveMessage, sqs:DeleteMessage and sqs:GetQueueAttributes on the queue - the managed policy AWSLambdaSQSQueueExecutionRole has them. Without them the mapping is created and then reports the error in StateTransitionReason and LastProcessingResult, not in your logs.

One bad message, a whole batch back

Three orders, the second one malformed:

$ for i in 1 2 3; do aws sqs send-message --queue-url $Q --message-body "{\"id\":$i,\"bad\":$([ $i = 2 ] && echo true || echo false)}" --query MessageId --output text; done
c0ae4440-54a0-4af4-821f-1f9763cb060f
94983b61-586f-4668-a2a5-6695341f228b
94fc4d19-5377-43b3-a607-33d378313773
$ sleep 10
$ aws logs tail /aws/lambda/try-worker --format short | grep -v 'START\|END\|REPORT'
2026-09-22T20:00:11 2026-09-22T20:00:11.157Z	52397999-4959-4861-b858-1384653817bd	INFO	processed order 1
2026-09-22T20:00:11 2026-09-22T20:00:11.157Z	52397999-4959-4861-b858-1384653817bd	INFO	cannot process order 2
2026-09-22T20:00:11 2026-09-22T20:00:11.159Z	52397999-4959-4861-b858-1384653817bd	ERROR	Invoke Error 	{"errorType":"Error","errorMessage":"order 2 is malformed","stack":["Error: order 2 is malformed","    at Runtime.handler (file:///var/task/index.mjs:9:13)","    at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1206:29)"]}
$ aws sqs get-queue-attributes --queue-url $Q --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible --query 'Attributes' --output text
0	3

Order 1 was processed, then the function threw on order 2 (order 3 never ran) - and all three are back in flight. In 180 seconds the batch is processed again: order 1 twice, a customer charged twice if the handler is not idempotent. Turn on partial batch responses: the mapping has to know about them (--function-response-types) and the code has to return them (here an environment variable switches the handler to that mode):

$ M=$(aws lambda list-event-source-mappings --function-name try-worker --query 'EventSourceMappings[0].UUID' --output text)
$ aws lambda update-event-source-mapping --uuid $M --function-response-types ReportBatchItemFailures --query FunctionResponseTypes
[
    "ReportBatchItemFailures"
]
$ aws lambda update-function-configuration --function-name try-worker --environment 'Variables={PARTIAL=on}' --query Environment
{
    "Variables": {
        "PARTIAL": "on"
    }
}
$ aws lambda wait function-updated-v2 --function-name try-worker
$ sleep 200
$ aws logs tail /aws/lambda/try-worker --since 4m --format short | grep -v 'START\|END\|REPORT'
2026-09-22T20:00:11 2026-09-22T20:00:11.157Z	52397999-4959-4861-b858-1384653817bd	INFO	processed order 1
2026-09-22T20:00:11 2026-09-22T20:00:11.157Z	52397999-4959-4861-b858-1384653817bd	INFO	cannot process order 2
2026-09-22T20:00:11 2026-09-22T20:00:11.159Z	52397999-4959-4861-b858-1384653817bd	ERROR	Invoke Error 	{"errorType":"Error","errorMessage":"order 2 is malformed","stack":["Error: order 2 is malformed","    at Runtime.handler (file:///var/task/index.mjs:9:13)","    at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1206:29)"]}
2026-09-22T20:03:11 2026-09-22T20:03:11.179Z	217636f4-06f9-4b76-95aa-0774149adbe3	INFO	processed order 1
2026-09-22T20:03:11 2026-09-22T20:03:11.179Z	217636f4-06f9-4b76-95aa-0774149adbe3	INFO	cannot process order 2
2026-09-22T20:03:11 2026-09-22T20:03:11.179Z	217636f4-06f9-4b76-95aa-0774149adbe3	INFO	processed order 3
$ aws sqs get-queue-attributes --queue-url $Q --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible --query 'Attributes' --output text
0	1

The redelivered batch ran again (order 1 a second time - the cost of the first failure); this time order 3 went through too and only order 2 was reported and stayed. It has now been received twice, so the next receive moves it to the DLQ:

$ sleep 200
$ D=$(aws sqs get-queue-url --queue-name try-orders-dlq --query QueueUrl --output text)
$ aws sqs receive-message --queue-url $D --query 'Messages[].Body' --output text
{"id":2,"bad":true}

Two rules for the handler: report failures with itemIdentifier = the record's messageId, and never both throw and report (a thrown error fails the whole batch again). An empty or invalid response with ReportBatchItemFailures on counts as complete success or complete failure, so test it.

Scaling and throttling a mapping

The mapping scales its pollers with the backlog - up to 1,000 concurrent invocations for one queue - and every batch is an invocation, so it competes for the function's concurrency. Two ways to protect what is behind the function (a database with 50 connections):

  • --scaling-config MaximumConcurrency=N on the mapping (2 to 1,000): the mapping never runs more than N batches at once. Excess messages simply wait in the queue - no throttling errors.
  • reserved concurrency on the function: a hard cap for every trigger. Batches beyond it are throttled; the messages go back to the queue and count as a receive, so a long throttle can push healthy messages into the DLQ. Prefer MaximumConcurrency for queues.

Watch ApproximateAgeOfOldestMessage (how far behind you are) and ApproximateNumberOfMessagesVisible on the queue, and Throttles and ConcurrentExecutions on the function. A backlog that grows while the function is idle at a low concurrency is a cap you set; a backlog with errors is a bad message or a broken dependency.

Asynchronous invocations: retries and destinations

When S3, SNS or EventBridge invoke a function they use the asynchronous path: Lambda queues the event and, on an error, retries twice (after about 1 and 2 minutes). Then the event is gone

  • unless you configure where it goes. An on-failure destination (SQS, SNS, Lambda,

EventBridge, S3) gets a record with the event and the error; an on-success destination gets the result. The older dead-letter queue setting on a function gets only the event.

$ aws lambda put-function-event-invoke-config --function-name try-worker --maximum-retry-attempts 1 --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:eu-central-1:111122223333:try-failed"}}' --query '[MaximumRetryAttempts,DestinationConfig.OnFailure.Destination]' --output text
1	arn:aws:sqs:eu-central-1:111122223333:try-failed
$ aws lambda invoke --function-name try-worker --invocation-type Event --payload '{"source":"manual"}' /dev/null
{
    "StatusCode": 202
}
$ sleep 90
$ aws sqs receive-message --queue-url $(aws sqs get-queue-url --queue-name try-failed --query QueueUrl --output text) --query 'Messages[0].Body' --output text | jq '{condition: .requestContext.condition, attempts: .requestContext.approximateInvokeCount, error: .responsePayload.errorMessage, event: .requestPayload}'
{
  "condition": "RetriesExhausted",
  "attempts": 2,
  "error": "not an SQS event: {\"source\":\"manual\"}",
  "event": {
    "source": "manual"
  }
}

Destinations are for asynchronous invocations (and, as on-failure destinations, for Kinesis and DynamoDB stream mappings). For SQS mappings the failure path is the queue's own DLQ - a DLQ or destination on the function is ignored for them.

On the exam (SAA-C03 2.1, 3.3; SOA-C03 1.2 and 2.1): Lambda + SQS to decouple and absorb bursts; visibility timeout vs function timeout (6x); partial batch responses; a DLQ on the source queue for poison messages; MaximumConcurrency or reserved concurrency to protect a database; async retries, destinations vs DLQ; "messages are processed more than once" = visibility timeout too short or no idempotency.

In an interview: "A message keeps getting processed again and again. Why?" - "Either it is never deleted - the function throws, or the batch fails because of another message and there is no partial batch response - or the visibility timeout is shorter than the processing, so it reappears while it is still being worked on. I'd check the function's errors and duration against the queue's visibility timeout, add ReportBatchItemFailures and a DLQ, and make the handler idempotent."

You can now: connect a queue to a function with the right visibility timeout and role, explain why a thrown error redelivers a whole batch, turn on and return partial batch responses, follow a poison message to the DLQ, cap a mapping with MaximumConcurrency instead of throttling it, and route failed asynchronous events to an on-failure destination with the error attached.

Why it helps

Lambda behind a queue is the most common serverless shape in production, and its failures are subtle: whole batches redelivered because of one bad message, duplicates when the visibility timeout is too short, backlogs caused by a concurrency cap, and failed events lost because no destination was configured. Each of these has a specific setting that fixes it, and knowing them is what keeps a queue-driven system correct.

Commands in this lesson

cd cat aws sleep

FAQ

Why does Lambda refuse a queue whose visibility timeout is shorter than the function timeout?

While the function may still be working on a batch, those messages must stay invisible; otherwise another poller would hand them to a second copy of the function. Lambda therefore requires the visibility timeout to be at least the function timeout, and AWS recommends six times as much to leave room for retries.

What happens if the function throws for one message in a batch?

Without partial batch responses nothing is deleted: the whole batch becomes visible again after the visibility timeout and every message in it is processed again. With ReportBatchItemFailures, the function returns the ids that failed and only those come back.

Where do failed messages go for an SQS trigger?

To the source queue's dead-letter queue, through its redrive policy, after maxReceiveCount receives. A DLQ or on-failure destination configured on the function is ignored for SQS event source mappings; those only apply to asynchronous invocations and stream sources.

How do I protect a database behind the function?

Set MaximumConcurrency on the event source mapping so it never runs more batches at once than the database can serve. Reserved concurrency also caps the function, but throttled batches go back to the queue and count as receives, which can push healthy messages into the DLQ during a long throttle.

What do destinations add over a dead-letter queue?

An on-failure destination for asynchronous invocations receives a record with the original event, the error message and type, and the number of attempts; a function DLQ gets only the event. Destinations can be SQS, SNS, another function, EventBridge or S3, and an on-success destination can record results too.

In an interview Mid

A message keeps getting processed again and again by a Lambda consumer. What could cause it?

Either it is never deleted or it reappears while still being processed. The function may throw, so the whole batch is redelivered - fixed with partial batch responses (ReportBatchItemFailures in the mapping and batchItemFailures in the code) and a DLQ on the queue. Or the visibility timeout is shorter than the processing time. I would compare the function's errors and duration with the queue's visibility timeout and make the handler idempotent.

Also asked: How do you stop one bad message from blocking a queue? · What is the difference between MaximumConcurrency and reserved concurrency? · What happens to a failed asynchronous invocation?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.