Step Functions: workflows with retries, catches and compensation
An order is not one call: reserve the stock, charge the card, ship, email - and when the charge is declined, release the stock again. Chaining functions directly (one invokes the next) hides that logic in code, loses track of where an order is, and makes retries everyone's own problem. AWS Step Functions runs the workflow as a state machine: each step's input and output are recorded, retries and error handling are declared, and you can see exactly where execution order-1042 stopped and why.
Need to know: a state machine is JSON in the Amazon States Language: StartAt and States, each state with a Type - Task (call a function or one of 200+ AWS APIs), Choice, Wait, Parallel, Map (the same steps for each item), Pass, Succeed, Fail. A Task can have Retry (which errors, IntervalSeconds, MaxAttempts, BackoffRate) and Catch (which errors go to which state, with the error in ResultPath). Standard workflows run up to a year, exactly-once, every step in the history, billed per state transition; Express workflows run up to 5 minutes, at-least-once, billed by requests and duration - for high-volume event processing.
The order workflow
order.asl.json reserves stock, charges with retries on a busy payment gateway, and on a declined card releases the stock and fails the order. The role try-sfn-role trusts states.amazonaws.com and may invoke the functions:
$ cd ~/oncall-lab/labs/aws/try6/sfn
$ jq -r '.States | to_entries[] | "\(.key)\t\(.value.Type)"' order.asl.json
Reserve Task
Charge Task
Release Task
Declined Fail
Done Succeed
$ jq '.States.Charge | {Retry, Catch}' order.asl.json
{
"Retry": [
{
"ErrorEquals": [
"PaymentGatewayBusy"
],
"IntervalSeconds": 2,
"MaxAttempts": 3,
"BackoffRate": 2
}
],
"Catch": [
{
"ErrorEquals": [
"CardDeclined"
],
"ResultPath": "$.error",
"Next": "Release"
}
]
}
$ aws stepfunctions validate-state-machine-definition --definition file://broken.asl.json
{
"result": "FAIL",
"diagnostics": [
{
"severity": "ERROR",
"code": "MISSING_TRANSITION_TARGET",
"message": "Missing 'Next' target: ReleaseStock",
"location": "/States/Charge/Catch[0]/Next"
}
],
"truncated": false
}
$ SM=$(aws stepfunctions create-state-machine --name try-order --definition file://order.asl.json --role-arn arn:aws:iam::111122223333:role/try-sfn-role --query stateMachineArn --output text); echo $SM
arn:aws:states:eu-central-1:111122223333:stateMachine:try-order
validate-state-machine-definition (and create) reject a Next that points at nothing - the most common mistake after a rename. Run an order whose payment gateway is busy twice:
$ E=$(aws stepfunctions start-execution --state-machine-arn $SM --name order-1042 --input '{"order":1042,"amount":59,"flaky":true}' --query executionArn --output text)
$ aws stepfunctions describe-execution --execution-arn $E --query status --output text
RUNNING
$ sleep 15
$ aws stepfunctions describe-execution --execution-arn $E --query '[status,output]' --output text
SUCCEEDED {"order":1042,"amount":59,"flaky":true,"stock":{"reserved":true},"payment":{"charged":59,"attempt":3}}
$ aws stepfunctions get-execution-history --execution-arn $E --query "events[?starts_with(type,'Task')].type" --output text | tr '\t' '\n' | sort | uniq -c
2 TaskFailed
4 TaskScheduled
4 TaskStarted
2 TaskStateEntered
2 TaskStateExited
2 TaskSucceeded
Four tasks were scheduled for two Task states: Reserve once, Charge three times - TaskFailed twice, then success on the third attempt, after two seconds and then four (backoff rate 2). No retry code in the function. Now a declined card:
$ E2=$(aws stepfunctions start-execution --state-machine-arn $SM --name order-1043 --input '{"order":1043,"amount":59,"card":"stolen"}' --query executionArn --output text)
$ sleep 5
$ aws stepfunctions describe-execution --execution-arn $E2 --query '[status,error,cause]' --output text
FAILED OrderDeclined The card was declined; the stock was released
$ aws stepfunctions get-execution-history --execution-arn $E2 --query "events[?type=='TaskStateEntered'].stateEnteredEventDetails.name" --output text
Reserve Charge Release
$ aws stepfunctions list-executions --state-machine-arn $SM --query 'executions[].[name,status]' --output text
order-1043 FAILED
order-1042 SUCCEEDED
The Catch sent CardDeclined to Release - the compensating step that undoes the reservation - and the order ended in a deliberate Fail with its own error and cause. That is the saga pattern: every step that changes something has an undo, and the workflow runs the undos when a later step fails.
Retry well
- Retry the transient, catch the permanent. Throttling, timeouts and "busy" errors deserve a retry with backoff (
States.Timeout,Lambda.TooManyRequestsException, your ownPaymentGatewayBusy); a declined card or a validation error will fail every time - catch it. - Backoff and jitter:
BackoffRatemultiplies the interval each attempt,MaxDelaySecondscaps it,JitterStrategy: FULLspreads retries so a thousand executions do not retry in the same second. - Retries need idempotent steps: a charge that timed out may have succeeded. Pass an idempotency key (the execution name or order id) to the payment API.
- Timeouts: a Task without
TimeoutSecondswaits as long as the integration does; set one.
More than Lambda
A Task can call AWS services directly - arn:aws:states:::dynamodb:putItem, sqs:sendMessage, sns:publish, ecs:runTask.sync (wait for a container job), glue:startJobRun.sync, aws-sdk:<service>:<action> for almost any API - so many workflows need no glue functions at all. .waitForTaskToken pauses until something calls SendTaskSuccess with the token: a human approval, a callback from another system. Map processes a list (a distributed Map fans out over millions of S3 objects); Parallel runs branches at once and waits for all of them.
| Use | When |
|---|---|
| Step Functions Standard | business workflows, long waits, human approval, auditability |
| Step Functions Express | high-volume, short event processing (IoT, streaming transforms) |
| SQS between functions | simple hand-over, no need to see the whole flow |
| EventBridge | routing events to whoever is interested, not ordering steps |
On the exam (SAA-C03 2.1 "serverless workflows", 3.3; SOA-C03 3.2): orchestration of multiple Lambda functions or services with retries and error handling = Step Functions; Standard vs Express; human approval = a task token; "avoid writing retry and state-tracking code" points here.
In an interview: "How would you implement an order process that must undo earlier steps when a later one fails?" - "A Step Functions state machine as a saga: each step is a Task with Retry on transient errors; a Catch on the permanent ones goes to compensating steps that undo what succeeded - release the stock, refund - and the execution ends in a Fail with a clear error. Every step's input and output is in the execution history, so support can see where an order stopped."
You can now: read and validate a state machine definition, create one with a role that trusts states.amazonaws.com, start executions and read their status, output and history, write Retry with backoff and Catch with a compensating step, explain Standard vs Express, use direct service integrations and task tokens, and choose between Step Functions, queues and EventBridge.