OnCallReadyStart learning

AWS Lesson 39.1 · AWS VI: Architecting on AWS & the exams · 19 min read

Decoupling: queues, topics and the event bus

In plain words

Imagine a restaurant where every waiter must hand each order to the cook in person and wait until the dish is ready. One slow cook and the whole room stops. Now put a ticket rail between them: waiters clip the order and go back to the tables, cooks take tickets when they can. If a cook drops a ticket, it goes back on the rail; if a ticket is unreadable after a few tries, it goes to the manager's tray. That rail is a queue, the manager's tray a dead-letter queue, and a loudspeaker that tells every station at once is a topic.

Decoupling: queues, topics and the event bus

A checkout that calls the warehouse, the invoice service and the fraud check one after the other is only as available as the slowest of them, and a burst of orders hits all four at once. Put a queue between the checkout and each worker and the checkout only has to hand the order over: the workers take it when they can, a slow or broken worker builds a backlog instead of an outage, and you scale the workers on the backlog. That is decoupling, and on AWS it has three building blocks: SQS (a queue: one consumer group takes each message), SNS (a topic: every subscriber gets a copy) and EventBridge (a bus: rules route events by their content, recapped from AWS IV).

Need to know: an SQS message is not removed when you receive it. It becomes invisible for the visibility timeout; you delete it with the receipt handle of that receive when the work is done. If you never delete it, it comes back and is received again - that is how SQS retries. A redrive policy moves a message that has been received maxReceiveCount times to a dead-letter queue (DLQ), so one bad message cannot loop forever. Standard queues are nearly unlimited in throughput but deliver at least once in roughly the order sent; FIFO queues (.fifo) keep order per message group and drop duplicates within 5 minutes.

A queue with a dead-letter queue

The lab has try-jobs (visibility timeout 5 s, maxReceiveCount 2, DLQ try-jobs-dlq). Send one message and receive it the way a worker does:

$ cd ~/oncall-lab/labs/aws/try6
$ Q=$(aws sqs get-queue-url --queue-name try-jobs --query QueueUrl --output text); echo $Q
https://sqs.eu-central-1.amazonaws.com/111122223333/try-jobs
$ aws sqs get-queue-attributes --queue-url $Q --attribute-names VisibilityTimeout RedrivePolicy ReceiveMessageWaitTimeSeconds
{
    "Attributes": {
        "VisibilityTimeout": "5",
        "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-central-1:111122223333:try-jobs-dlq\",\"maxReceiveCount\":\"2\"}",
        "ReceiveMessageWaitTimeSeconds": "0"
    }
}
$ aws sqs send-message --queue-url $Q --message-body '{"job":"resize","id":7}'
{
    "MD5OfMessageBody": "513c59a96caf8f78ebc2b8a20d7989df",
    "MessageId": "4962a78a-a572-4879-857d-9961655a9d91"
}
$ aws sqs receive-message --queue-url $Q --attribute-names ApproximateReceiveCount
{
    "Messages": [
        {
            "MessageId": "4962a78a-a572-4879-857d-9961655a9d91",
            "ReceiptHandle": "AQEBdHJ5LWpvYnN8NDk2MmE3OGEtYTU3Mi00ODc5LTg1N2QtOTk2MTY1NWE5ZDkxfDE3OTAxMDcyMDQ0MDB8MQbc243746679e4763929134529fa093d993bee824902a918a",
            "MD5OfBody": "513c59a96caf8f78ebc2b8a20d7989df",
            "Body": "{\"job\":\"resize\",\"id\":7}",
            "Attributes": {
                "ApproximateReceiveCount": "1"
            }
        }
    ]
}
$ aws sqs receive-message --queue-url $Q

The second receive printed nothing: the message is in flight. The worker "crashes" - it never deletes it - so after 5 seconds it is back, with a receive count of 2:

$ sleep 6
$ aws sqs receive-message --queue-url $Q --attribute-names ApproximateReceiveCount --query 'Messages[0].Attributes'
{
    "ApproximateReceiveCount": "2"
}
$ sleep 6
$ aws sqs receive-message --queue-url $Q --wait-time-seconds 2
$ D=$(aws sqs get-queue-url --queue-name try-jobs-dlq --query QueueUrl --output text)
$ aws sqs receive-message --queue-url $D --attribute-names DeadLetterQueueSourceArn --query 'Messages[0].[Body,Attributes.DeadLetterQueueSourceArn]' --output text
{"job":"resize","id":7}	arn:aws:sqs:eu-central-1:111122223333:try-jobs

The third receive found nothing to give: with maxReceiveCount 2 the message was moved to the DLQ instead of being handed out a third time. That is the whole contract of a worker: receive, do the work, delete with the receipt handle. The timeout has to be longer than the work takes, or a second worker gets the same message while the first is still on it.

A few knobs that matter in production:

AttributeDefaultWhat it does
VisibilityTimeout30 s (0 s - 12 h)how long a received message stays hidden; longer than the work
MessageRetentionPeriod4 days (1 min - 14 days)after this a message is gone, processed or not
ReceiveMessageWaitTimeSeconds0long polling (up to 20 s): fewer empty receives, lower cost
DelaySeconds0 (up to 15 min)a delay queue: new messages are invisible at first
RedrivePolicynonethe DLQ and maxReceiveCount
SqsManagedSseEnabled / KmsMasterKeyIdSSE-SQS onencryption at rest (SSE-SQS, or SSE-KMS with your key)

A message is up to 1 MiB; anything bigger goes to S3 with a pointer in the message. A DLQ is not a bin: alarm on ApproximateNumberOfMessagesVisible of the DLQ, find out why, fix it, then redrive the messages back with start-message-move-task.

FIFO: order and exactly-once processing

A FIFO queue needs a MessageGroupId on every message, and a deduplication id (or ContentBasedDeduplication, which hashes the body):

$ F=$(aws sqs get-queue-url --queue-name try-pay.fifo --query QueueUrl --output text)
$ aws sqs send-message --queue-url $F --message-body 'charge 42' --message-group-id acct-1
{
    "MD5OfMessageBody": "8cf687c6b184bc7d7b0d2a818c681c55",
    "MessageId": "43834c6f-456b-49a0-ad96-a86f316270e6",
    "SequenceNumber": "18900000000000000000"
}
$ aws sqs get-queue-attributes --queue-url $F --attribute-names ContentBasedDeduplication FifoQueue
{
    "Attributes": {
        "ContentBasedDeduplication": "true",
        "FifoQueue": "true"
    }
}
$ for m in 'charge 42' 'charge 43' 'charge 42' 'refund 42'; do aws sqs send-message --queue-url $F --message-body "$m" --message-group-id acct-1 --query SequenceNumber --output text; done
18900000000000000000
18900000000000004296
18900000000000000000
18900000000000008392
$ aws sqs receive-message --queue-url $F --max-number-of-messages 10 --query 'Messages[].Body'
[
    "charge 42",
    "charge 43",
    "refund 42"
]
$ aws sqs receive-message --queue-url $F --max-number-of-messages 10 --query 'Messages[].Body'

Both repeats of "charge 42" were duplicates of the first send inside the 5-minute window: they got its sequence number back and were not queued again. The receive returned the group in order, and the next receive returned nothing - while messages of a group are in flight, the whole group waits. Order per group is the price: throughput is 300 API calls per second per action (3,000 messages with batches of 10), more with high-throughput mode, and parallelism comes from many groups (one per account or per order), never from one.

In an interview: "Standard or FIFO?" - "Standard unless order or duplicates break the business logic. Even then I make the consumer idempotent, because a standard queue can deliver a message twice; FIFO gives order per message group and deduplication within 5 minutes, at a lower throughput, and I pick the group id so the work can still run in parallel."

SNS fan-out with filter policies

One order event, three consumers that each need their own copy: publish to an SNS topic and subscribe a queue per consumer (fan-out). Each queue keeps its own backlog and retries, and a filter policy on a subscription decides which messages that subscriber gets. The topic try-orders-events has two queues: try-fulfil (everything) and try-fraud (only orders with a total of 1000 or more, matched on the message body):

$ T=arn:aws:sns:eu-central-1:111122223333:try-orders-events
$ aws sns list-subscriptions-by-topic --topic-arn $T --query 'Subscriptions[].Endpoint' --output text
arn:aws:sqs:eu-central-1:111122223333:try-fulfil	arn:aws:sqs:eu-central-1:111122223333:try-fraud
$ aws sns get-subscription-attributes --subscription-arn $(aws sns list-subscriptions-by-topic --topic-arn $T --query "Subscriptions[?ends_with(Endpoint,'try-fraud')].SubscriptionArn" --output text) --query 'Attributes.[FilterPolicy,FilterPolicyScope]' --output text
{"total":[{"numeric":[">=",1000]}]}	MessageBody
$ aws sns publish --topic-arn $T --message '{"order":1,"total":40}' --query MessageId --output text
8ae56a48-078a-4318-a408-489a5157ac36
$ aws sns publish --topic-arn $T --message '{"order":2,"total":2500}' --query MessageId --output text
71484646-4993-4ad9-883c-38f831b35f94
$ for q in try-fulfil try-fraud; do echo "$q: $(aws sqs get-queue-attributes --queue-url $(aws sqs get-queue-url --queue-name $q --query QueueUrl --output text) --attribute-names ApproximateNumberOfMessages --query Attributes.ApproximateNumberOfMessages --output text)"; done
try-fulfil: 2
try-fraud: 1
$ aws sqs receive-message --queue-url $(aws sqs get-queue-url --queue-name try-fraud --query QueueUrl --output text) --query 'Messages[0].Body' --output text | jq '{Type, Message}'
{
  "Type": "Notification",
  "Message": "{\"order\":2,\"total\":2500}"
}

Without RawMessageDelivery the queue gets the SNS envelope (Type, MessageId, TopicArn, Message, ...) and the consumer reads .Message. Two traps: the queue's access policy must allow sns.amazonaws.com to sqs:SendMessage (with aws:SourceArn = the topic) or deliveries are silently dropped, and a filter policy matches message attributes unless its scope is MessageBody.

Queue, topic or bus?

You needUseWhy
a buffer between a producer and one consumer group, retries, a backlog to scale onSQS standardpull-based, at-least-once, nearly unlimited throughput
strict order per key, no duplicates (payments per account)SQS FIFOorder per message group, 5-minute dedup
the same message to several consumers, filtered per subscriberSNS + SQS fan-outpush to every subscription; a queue per consumer keeps them independent
route events from AWS services or SaaS by their content, to many target types, replayEventBridgerules with patterns, 20+ target types, archives, cross-account buses
a business workflow with steps, retries, waits and compensationStep Functionsa state machine, later in this chapter
ordered streams many consumers replay (clickstream, logs)Kinesis Data Streamsshards, retention, re-reading; later in this chapter

SQS is pull (workers ask), SNS and EventBridge push. A push target that is down loses messages unless the subscription or target has a DLQ - which is one more reason to put an SQS queue behind a topic instead of an HTTP endpoint. Amazon MQ (ActiveMQ / RabbitMQ) exists for applications that already speak JMS, AMQP or MQTT and are being moved as they are.

On the exam (SAA-C03 2.1 "scalable and loosely coupled architectures"; SOA-C03 3.2): queue vs topic vs bus; FIFO when order or exactly-once processing is stated; visibility timeout longer than the processing time; DLQ with maxReceiveCount; long polling to cut empty receives and cost; SNS fan-out to several SQS queues; "decouple" and "buffer the burst" both mean SQS.

In an interview: "How does SQS make a system more resilient?" - "It turns a synchronous call into a hand-over: the producer only needs the queue, consumers process at their own pace and retry by not deleting, a DLQ catches poison messages, and the backlog absorbs bursts and becomes the scaling signal."

You can now: send, receive and delete with receipt handles, explain visibility timeouts and redelivery, configure a DLQ and know when a message moves there, use FIFO groups and deduplication, fan out with SNS and filter per subscriber (typed matchers, body scope, the queue policy), and pick a queue, topic, bus, workflow or stream for a requirement.

Why it helps

Most outages in distributed systems spread through synchronous calls: one slow service makes every caller slow. Queues and topics break that chain. Knowing how SQS really behaves - messages stay until deleted, the visibility timeout, redrive to a DLQ, FIFO groups - prevents the classic bugs of duplicate processing and lost work, and choosing between SQS, SNS fan-out and EventBridge is one of the most common design questions on the Solutions Architect exam and in interviews.

Commands in this lesson

cd aws sleep

FAQ

Why is a received SQS message not deleted automatically?

Because the consumer might crash before it finishes. SQS only hides the message for the visibility timeout; the consumer deletes it with the receipt handle when the work is done. If it never does, the message becomes visible again and another consumer retries it, which is how SQS gives at-least-once processing without losing work.

What does maxReceiveCount do?

It is part of the queue's redrive policy. Every receive increments a message's receive count; when a receive would exceed maxReceiveCount, SQS moves the message to the dead-letter queue instead. That stops a poison message from being retried forever and keeps it for someone to inspect and redrive later with start-message-move-task.

Can a standard queue deliver a message twice?

Yes. Standard queues are at-least-once: in rare cases a message is delivered more than once, and a short visibility timeout causes redeliveries too. Consumers should therefore be idempotent - for example by recording processed message or order ids. FIFO queues add deduplication within a five-minute window for exactly-once processing.

Why do SNS deliveries to my queue silently fail?

SNS sends to the queue as the service principal sns.amazonaws.com, and the queue's access policy must allow that with sqs:SendMessage, ideally restricted with aws:SourceArn to the topic. Without it the publish succeeds but nothing arrives. With a KMS-encrypted queue, the key policy must allow SNS to use the key as well.

Is long polling always better?

Almost always. With ReceiveMessageWaitTimeSeconds up to 20, a receive waits for a message instead of returning empty immediately, which cuts empty receives, API cost and latency to the next message. Short polling only makes sense when a single thread must check several queues and cannot block on one.

In an interview Mid

How does SQS make a system more resilient, and what settings matter?

It turns a synchronous call into a hand-over: the producer only needs the queue, consumers process at their own pace, and a backlog absorbs bursts. The settings that matter are the visibility timeout - longer than the processing time, or messages are processed twice - a redrive policy with maxReceiveCount and a DLQ that someone alarms on, long polling to cut empty receives, and FIFO only when order or exactly-once processing is required.

Also asked: What is the difference between a standard and a FIFO queue? · How does SNS fan-out with SQS work? · What happens to a message that keeps failing?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.