Designing for resilience: AZs, Regions and DR strategies
Every architecture question on the Solutions Architect exam, and every design review, comes down to two numbers the business sets: how much data it may lose - the recovery point objective (RPO) - and how long it may be down - the recovery time objective (RTO). Everything else is engineering towards them at a price: more Availability Zones, a second Region, replicas, backups, rehearsals. This lesson is the design view; AWS IV and AWS V ran the tools (CloudFormation and StackSets, AWS Backup, RDS Multi-AZ, Route 53 failover, S3 replication).
Need to know: a Region is a separate geography with its own control plane; it has three or more Availability Zones - separate data centres with their own power and network, a few milliseconds apart. High availability = surviving the loss of a component or an AZ automatically (Multi-AZ). Disaster recovery = getting back after losing a Region, or after a mistake that replicates everywhere (a deleted table, ransomware, a bad deployment) - which only backups and point-in-time recovery fix, because replication copies the mistake too.
Find the single points of failure
The try-shop system: an API on ECS Fargate and an intake queue. Review it the way a design review does - component by component, "what happens when this AZ / this instance / this dependency is gone?":
$ aws ec2 describe-availability-zones --query 'AvailabilityZones[].ZoneName' --output text
eu-central-1a eu-central-1b eu-central-1c
$ aws ecs describe-services --cluster try-shop --services try-api --query 'services[0].[desiredCount,runningCount,networkConfiguration.awsvpcConfiguration.subnets]' --output json
[
1,
1,
[
"subnet-07d93f499311787f9"
]
]
$ aws ec2 describe-subnets --subnet-ids $(aws ecs describe-services --cluster try-shop --services try-api --query 'services[0].networkConfiguration.awsvpcConfiguration.subnets' --output text) --query 'Subnets[].[SubnetId,AvailabilityZone]' --output text
subnet-07d93f499311787f9 eu-central-1a
$ Q=$(aws sqs get-queue-url --queue-name try-intake --query QueueUrl --output text)
$ aws sqs get-queue-attributes --queue-url $Q --attribute-names RedrivePolicy MessageRetentionPeriod
{
"Attributes": {
"MessageRetentionPeriod": "345600"
}
}
$ aws lambda list-functions --region eu-west-1 --query 'length(Functions)'
0
Three findings: one task in one subnet = one AZ (an AZ outage, or just a task replacement, is downtime); a queue without a DLQ (a poison message loops until the 4-day retention drops it, taking good messages' capacity with it); and nothing in a second Region. The first is cheap to fix - spread over two AZs with at least two tasks:
$ cd ~/oncall-lab/labs/aws/try6
$ cat network-2az.txt
awsvpcConfiguration={subnets=[subnet-07d93f499311787f9,subnet-013c96da193a52f19],securityGroups=[sg-0469a55e27b3c238a],assignPublicIp=ENABLED}
$ aws ecs update-service --cluster try-shop --service try-api --desired-count 2 --network-configuration "$(cat network-2az.txt)" --query 'service.desiredCount'
2
$ aws ecs wait services-stable --cluster try-shop --services try-api
$ aws ecs describe-tasks --cluster try-shop --tasks $(aws ecs list-tasks --cluster try-shop --service-name try-api --query taskArns --output text) --query 'tasks[].[lastStatus,availabilityZone]' --output text
RUNNING eu-central-1a
RUNNING eu-central-1b
Multi-AZ is the default answer for anything stateful too: RDS Multi-AZ, Aurora replicas in other AZs, ElastiCache with replicas, EFS (Regional), at least two subnets for every load balancer, an Auto Scaling group across AZs. S3, DynamoDB, SQS, SNS, Lambda and Step Functions are Regional services - already spread across AZs by AWS.
The four DR strategies
| Strategy | In the recovery Region | RPO | RTO | Cost |
|---|---|---|---|---|
| Backup and restore | backups / snapshots copied there (AWS Backup cross-Region copy, S3 replication); infrastructure as code ready | hours (since the last backup) | hours (rebuild + restore) | lowest |
| Pilot light | data replicated continuously (Aurora Global Database, DynamoDB global tables, S3 CRR); core services running, app servers off or at zero | seconds to minutes | tens of minutes (start + scale + DNS) | low |
| Warm standby | a scaled-down but fully working copy, taking no (or test) traffic | seconds to minutes | minutes (scale up + switch) | medium |
| Multi-site active/active | full capacity serving users in both Regions | near zero (depends on replication) | near zero | highest |
How to pick: start from the RPO and RTO, then the budget. "RPO 24 hours, RTO 1 day, lowest cost" = backup and restore. "RTO under an hour, keep costs low" = pilot light. "RTO of minutes" = warm standby. "No downtime, global users" = multi-site. Then the details that sink real failovers:
- Data first: the strategy is only as good as the replication or backup behind it. Asynchronous cross-Region replication means the last seconds are lost (that is your RPO).
- Everything else too: AMIs, container images (ECR replication), secrets, parameters, KMS keys (multi-Region keys), DNS, certificates, and service quotas in the recovery Region - a standby that cannot launch enough instances is not a standby.
- Static stability: the recovery path should not depend on the failed Region's control plane. Pre-provision what you need; prefer data-plane actions (Route 53 health-check failover) over "create everything during the outage".
- Rehearse: a DR plan that has never been run has an unknown RTO. Game days, AWS Fault Injection Service, and Route 53 Application Recovery Controller for controlled switches.
Resilience inside one design
- Decouple (this chapter's first lessons): a queue absorbs a slow dependency; a failed consumer builds a backlog, not an outage.
- Timeouts, retries with backoff and jitter, circuit breakers - and idempotent operations so a retry is safe.
- Health checks that mean something: the load balancer's check should fail when the instance cannot serve, not only when the process is gone - but not when a shared dependency is down, or every target fails at once.
- Limit blast radius: cells, shuffle sharding, separate accounts per environment, one change at a time (canaries, the ECS circuit breaker, CloudFormation rollback).
- Backups you have restored: AWS Backup plans with vault lock against deletion; a restore test on a schedule.
On the exam (SAA-C03 2.2 "highly available and/or fault-tolerant architectures", 2.1; SOA-C03 2.2, 2.3): Multi-AZ for HA, Multi-Region for DR, the four strategies by RPO / RTO / cost, Route 53 failover routing with health checks, Aurora Global Database / DynamoDB global tables / S3 CRR for cross-Region data, AWS Backup cross-Region copies, service quotas in the standby Region, "most cost-effective that meets RTO X" questions.
In an interview: "What's the difference between high availability and disaster recovery?" - "HA keeps the service running through expected failures - an instance, an AZ - automatically, with redundancy in one Region. DR is how you recover from a disaster or a mistake that takes the whole thing down: another Region, or backups when the data itself is corrupted. HA has no RPO to speak of; DR is chosen by the RPO and RTO the business accepts and what it will pay."
You can now: define RPO and RTO and get them from the business, review a design for single points of failure, spread a service across AZs, pick backup-and-restore, pilot light, warm standby or multi-site for a requirement and budget, list what besides data must exist in a recovery Region, and explain static stability and why DR must be rehearsed.