OnCallReadyStart learning

AWS Lesson 39.20 · AWS VI: Architecting on AWS & the exams · 12 min read

Step Functions: workflows with retries, catches and compensation

In plain words

Think of a recipe card pinned in a kitchen: first reserve the ingredients, then cook, and if the oven is busy wait a little and try again - but if the customer cancels, put the ingredients back and close the ticket. Step Functions is that recipe card, followed by a tireless cook who writes down every step it took. When something goes wrong you can read exactly where the order stopped instead of asking everyone in the kitchen.

Step Functions: workflows with retries, catches and compensation

An order is not one call: reserve the stock, charge the card, ship, email - and when the charge is declined, release the stock again. Chaining functions directly (one invokes the next) hides that logic in code, loses track of where an order is, and makes retries everyone's own problem. AWS Step Functions runs the workflow as a state machine: each step's input and output are recorded, retries and error handling are declared, and you can see exactly where execution order-1042 stopped and why.

Need to know: a state machine is JSON in the Amazon States Language: StartAt and States, each state with a Type - Task (call a function or one of 200+ AWS APIs), Choice, Wait, Parallel, Map (the same steps for each item), Pass, Succeed, Fail. A Task can have Retry (which errors, IntervalSeconds, MaxAttempts, BackoffRate) and Catch (which errors go to which state, with the error in ResultPath). Standard workflows run up to a year, exactly-once, every step in the history, billed per state transition; Express workflows run up to 5 minutes, at-least-once, billed by requests and duration - for high-volume event processing.

The order workflow

order.asl.json reserves stock, charges with retries on a busy payment gateway, and on a declined card releases the stock and fails the order. The role try-sfn-role trusts states.amazonaws.com and may invoke the functions:

$ cd ~/oncall-lab/labs/aws/try6/sfn
$ jq -r '.States | to_entries[] | "\(.key)\t\(.value.Type)"' order.asl.json
Reserve	Task
Charge	Task
Release	Task
Declined	Fail
Done	Succeed
$ jq '.States.Charge | {Retry, Catch}' order.asl.json
{
  "Retry": [
    {
      "ErrorEquals": [
        "PaymentGatewayBusy"
      ],
      "IntervalSeconds": 2,
      "MaxAttempts": 3,
      "BackoffRate": 2
    }
  ],
  "Catch": [
    {
      "ErrorEquals": [
        "CardDeclined"
      ],
      "ResultPath": "$.error",
      "Next": "Release"
    }
  ]
}
$ aws stepfunctions validate-state-machine-definition --definition file://broken.asl.json
{
    "result": "FAIL",
    "diagnostics": [
        {
            "severity": "ERROR",
            "code": "MISSING_TRANSITION_TARGET",
            "message": "Missing 'Next' target: ReleaseStock",
            "location": "/States/Charge/Catch[0]/Next"
        }
    ],
    "truncated": false
}
$ SM=$(aws stepfunctions create-state-machine --name try-order --definition file://order.asl.json --role-arn arn:aws:iam::111122223333:role/try-sfn-role --query stateMachineArn --output text); echo $SM
arn:aws:states:eu-central-1:111122223333:stateMachine:try-order

validate-state-machine-definition (and create) reject a Next that points at nothing - the most common mistake after a rename. Run an order whose payment gateway is busy twice:

$ E=$(aws stepfunctions start-execution --state-machine-arn $SM --name order-1042 --input '{"order":1042,"amount":59,"flaky":true}' --query executionArn --output text)
$ aws stepfunctions describe-execution --execution-arn $E --query status --output text
RUNNING
$ sleep 15
$ aws stepfunctions describe-execution --execution-arn $E --query '[status,output]' --output text
SUCCEEDED	{"order":1042,"amount":59,"flaky":true,"stock":{"reserved":true},"payment":{"charged":59,"attempt":3}}
$ aws stepfunctions get-execution-history --execution-arn $E --query "events[?starts_with(type,'Task')].type" --output text | tr '\t' '\n' | sort | uniq -c
      2 TaskFailed
      4 TaskScheduled
      4 TaskStarted
      2 TaskStateEntered
      2 TaskStateExited
      2 TaskSucceeded

Four tasks were scheduled for two Task states: Reserve once, Charge three times - TaskFailed twice, then success on the third attempt, after two seconds and then four (backoff rate 2). No retry code in the function. Now a declined card:

$ E2=$(aws stepfunctions start-execution --state-machine-arn $SM --name order-1043 --input '{"order":1043,"amount":59,"card":"stolen"}' --query executionArn --output text)
$ sleep 5
$ aws stepfunctions describe-execution --execution-arn $E2 --query '[status,error,cause]' --output text
FAILED	OrderDeclined	The card was declined; the stock was released
$ aws stepfunctions get-execution-history --execution-arn $E2 --query "events[?type=='TaskStateEntered'].stateEnteredEventDetails.name" --output text
Reserve	Charge	Release
$ aws stepfunctions list-executions --state-machine-arn $SM --query 'executions[].[name,status]' --output text
order-1043	FAILED
order-1042	SUCCEEDED

The Catch sent CardDeclined to Release - the compensating step that undoes the reservation - and the order ended in a deliberate Fail with its own error and cause. That is the saga pattern: every step that changes something has an undo, and the workflow runs the undos when a later step fails.

Retry well

  • Retry the transient, catch the permanent. Throttling, timeouts and "busy" errors deserve a retry with backoff (States.Timeout, Lambda.TooManyRequestsException, your own PaymentGatewayBusy); a declined card or a validation error will fail every time - catch it.
  • Backoff and jitter: BackoffRate multiplies the interval each attempt, MaxDelaySeconds caps it, JitterStrategy: FULL spreads retries so a thousand executions do not retry in the same second.
  • Retries need idempotent steps: a charge that timed out may have succeeded. Pass an idempotency key (the execution name or order id) to the payment API.
  • Timeouts: a Task without TimeoutSeconds waits as long as the integration does; set one.

More than Lambda

A Task can call AWS services directly - arn:aws:states:::dynamodb:putItem, sqs:sendMessage, sns:publish, ecs:runTask.sync (wait for a container job), glue:startJobRun.sync, aws-sdk:<service>:<action> for almost any API - so many workflows need no glue functions at all. .waitForTaskToken pauses until something calls SendTaskSuccess with the token: a human approval, a callback from another system. Map processes a list (a distributed Map fans out over millions of S3 objects); Parallel runs branches at once and waits for all of them.

UseWhen
Step Functions Standardbusiness workflows, long waits, human approval, auditability
Step Functions Expresshigh-volume, short event processing (IoT, streaming transforms)
SQS between functionssimple hand-over, no need to see the whole flow
EventBridgerouting events to whoever is interested, not ordering steps

On the exam (SAA-C03 2.1 "serverless workflows", 3.3; SOA-C03 3.2): orchestration of multiple Lambda functions or services with retries and error handling = Step Functions; Standard vs Express; human approval = a task token; "avoid writing retry and state-tracking code" points here.

In an interview: "How would you implement an order process that must undo earlier steps when a later one fails?" - "A Step Functions state machine as a saga: each step is a Task with Retry on transient errors; a Catch on the permanent ones goes to compensating steps that undo what succeeded - release the stock, refund - and the execution ends in a Fail with a clear error. Every step's input and output is in the execution history, so support can see where an order stopped."

You can now: read and validate a state machine definition, create one with a role that trusts states.amazonaws.com, start executions and read their status, output and history, write Retry with backoff and Catch with a compensating step, explain Standard vs Express, use direct service integrations and task tokens, and choose between Step Functions, queues and EventBridge.

Why it helps

Business processes span several services and fail in the middle. Without orchestration, retry logic, error handling and undo steps are scattered across functions and nobody can tell where an order is. Step Functions declares them - Retry with backoff, Catch, compensating steps - and records every execution's history. It is the standard answer for workflows on the exams and a practical tool for keeping multi-step operations safe.

Commands in this lesson

cd jq aws sleep

FAQ

What is the difference between Retry and Catch?

Retry re-runs the failing state for listed errors with an interval, a maximum number of attempts and a backoff rate - for transient problems like a busy dependency. Catch routes listed errors to another state, with the error stored under ResultPath - for permanent problems that need a different path, such as compensation or a clean failure.

What happens to an error that no Retry or Catch matches?

The Task state fails and so does the whole execution, with that error and cause. Error names are matched exactly, so many workflows end their Catch list with States.ALL leading to a cleanup or alerting state, to make sure nothing fails without being handled.

When should I use Express instead of Standard workflows?

Express workflows suit high-volume, short event processing: up to five minutes, at-least-once execution, billed by requests and duration, with history in CloudWatch Logs. Standard workflows run up to a year with exactly-once execution and a full visual history, which suits business processes, long waits and human approval.

Do I need a Lambda function for every step?

No. Tasks can call more than 200 AWS services directly - DynamoDB, SQS, SNS, ECS, Glue and the general aws-sdk integration - and pass data between steps with JSON paths. Many workflows have no glue functions at all, which removes code, cold starts and cost.

What is the saga pattern?

A way to keep a multi-step process consistent without a distributed transaction: every step that changes something has a compensating step that undoes it, and when a later step fails, the workflow runs the compensations in reverse order before failing cleanly. In Step Functions, a Catch leads to those compensating states.

In an interview Mid

How would you implement an order process that must undo earlier steps when a later one fails?

As a Step Functions state machine using the saga pattern. Each step is a Task with Retry on transient errors, with backoff and jitter. A Catch on the permanent errors, such as a declined card, goes to compensating steps that undo what already succeeded - releasing reserved stock, refunding a charge - and then to a Fail state with a clear error. Steps are idempotent, and the execution history shows support exactly where each order stopped.

Also asked: When would you choose Step Functions over SQS between functions? · What is the difference between Standard and Express workflows? · How do you add a human approval step to a workflow?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.