OnCallReadyStart learning

AWS Lesson 39.27 · AWS VI: Architecting on AWS & the exams · 13 min read

Designing for resilience: AZs, Regions and DR strategies

In plain words

A sensible family keeps a spare key with a neighbour and copies of important papers at a relative's house in another town. Losing a key is a nuisance; a fire in the street is a disaster. Availability Zones are the neighbours - close, separate, quick to reach. Regions are the other town. How much you keep in the other town, and how quickly you could move there, depends on how much you can afford to lose and how long you can wait: the RPO and the RTO.

Designing for resilience: AZs, Regions and DR strategies

Every architecture question on the Solutions Architect exam, and every design review, comes down to two numbers the business sets: how much data it may lose - the recovery point objective (RPO) - and how long it may be down - the recovery time objective (RTO). Everything else is engineering towards them at a price: more Availability Zones, a second Region, replicas, backups, rehearsals. This lesson is the design view; AWS IV and AWS V ran the tools (CloudFormation and StackSets, AWS Backup, RDS Multi-AZ, Route 53 failover, S3 replication).

Need to know: a Region is a separate geography with its own control plane; it has three or more Availability Zones - separate data centres with their own power and network, a few milliseconds apart. High availability = surviving the loss of a component or an AZ automatically (Multi-AZ). Disaster recovery = getting back after losing a Region, or after a mistake that replicates everywhere (a deleted table, ransomware, a bad deployment) - which only backups and point-in-time recovery fix, because replication copies the mistake too.

Find the single points of failure

The try-shop system: an API on ECS Fargate and an intake queue. Review it the way a design review does - component by component, "what happens when this AZ / this instance / this dependency is gone?":

$ aws ec2 describe-availability-zones --query 'AvailabilityZones[].ZoneName' --output text
eu-central-1a	eu-central-1b	eu-central-1c
$ aws ecs describe-services --cluster try-shop --services try-api --query 'services[0].[desiredCount,runningCount,networkConfiguration.awsvpcConfiguration.subnets]' --output json
[
    1,
    1,
    [
        "subnet-07d93f499311787f9"
    ]
]
$ aws ec2 describe-subnets --subnet-ids $(aws ecs describe-services --cluster try-shop --services try-api --query 'services[0].networkConfiguration.awsvpcConfiguration.subnets' --output text) --query 'Subnets[].[SubnetId,AvailabilityZone]' --output text
subnet-07d93f499311787f9	eu-central-1a
$ Q=$(aws sqs get-queue-url --queue-name try-intake --query QueueUrl --output text)
$ aws sqs get-queue-attributes --queue-url $Q --attribute-names RedrivePolicy MessageRetentionPeriod
{
    "Attributes": {
        "MessageRetentionPeriod": "345600"
    }
}
$ aws lambda list-functions --region eu-west-1 --query 'length(Functions)'
0

Three findings: one task in one subnet = one AZ (an AZ outage, or just a task replacement, is downtime); a queue without a DLQ (a poison message loops until the 4-day retention drops it, taking good messages' capacity with it); and nothing in a second Region. The first is cheap to fix - spread over two AZs with at least two tasks:

$ cd ~/oncall-lab/labs/aws/try6
$ cat network-2az.txt
awsvpcConfiguration={subnets=[subnet-07d93f499311787f9,subnet-013c96da193a52f19],securityGroups=[sg-0469a55e27b3c238a],assignPublicIp=ENABLED}
$ aws ecs update-service --cluster try-shop --service try-api --desired-count 2 --network-configuration "$(cat network-2az.txt)" --query 'service.desiredCount'
2
$ aws ecs wait services-stable --cluster try-shop --services try-api
$ aws ecs describe-tasks --cluster try-shop --tasks $(aws ecs list-tasks --cluster try-shop --service-name try-api --query taskArns --output text) --query 'tasks[].[lastStatus,availabilityZone]' --output text
RUNNING	eu-central-1a
RUNNING	eu-central-1b

Multi-AZ is the default answer for anything stateful too: RDS Multi-AZ, Aurora replicas in other AZs, ElastiCache with replicas, EFS (Regional), at least two subnets for every load balancer, an Auto Scaling group across AZs. S3, DynamoDB, SQS, SNS, Lambda and Step Functions are Regional services - already spread across AZs by AWS.

The four DR strategies

StrategyIn the recovery RegionRPORTOCost
Backup and restorebackups / snapshots copied there (AWS Backup cross-Region copy, S3 replication); infrastructure as code readyhours (since the last backup)hours (rebuild + restore)lowest
Pilot lightdata replicated continuously (Aurora Global Database, DynamoDB global tables, S3 CRR); core services running, app servers off or at zeroseconds to minutestens of minutes (start + scale + DNS)low
Warm standbya scaled-down but fully working copy, taking no (or test) trafficseconds to minutesminutes (scale up + switch)medium
Multi-site active/activefull capacity serving users in both Regionsnear zero (depends on replication)near zerohighest

How to pick: start from the RPO and RTO, then the budget. "RPO 24 hours, RTO 1 day, lowest cost" = backup and restore. "RTO under an hour, keep costs low" = pilot light. "RTO of minutes" = warm standby. "No downtime, global users" = multi-site. Then the details that sink real failovers:

  • Data first: the strategy is only as good as the replication or backup behind it. Asynchronous cross-Region replication means the last seconds are lost (that is your RPO).
  • Everything else too: AMIs, container images (ECR replication), secrets, parameters, KMS keys (multi-Region keys), DNS, certificates, and service quotas in the recovery Region - a standby that cannot launch enough instances is not a standby.
  • Static stability: the recovery path should not depend on the failed Region's control plane. Pre-provision what you need; prefer data-plane actions (Route 53 health-check failover) over "create everything during the outage".
  • Rehearse: a DR plan that has never been run has an unknown RTO. Game days, AWS Fault Injection Service, and Route 53 Application Recovery Controller for controlled switches.

Resilience inside one design

  • Decouple (this chapter's first lessons): a queue absorbs a slow dependency; a failed consumer builds a backlog, not an outage.
  • Timeouts, retries with backoff and jitter, circuit breakers - and idempotent operations so a retry is safe.
  • Health checks that mean something: the load balancer's check should fail when the instance cannot serve, not only when the process is gone - but not when a shared dependency is down, or every target fails at once.
  • Limit blast radius: cells, shuffle sharding, separate accounts per environment, one change at a time (canaries, the ECS circuit breaker, CloudFormation rollback).
  • Backups you have restored: AWS Backup plans with vault lock against deletion; a restore test on a schedule.

On the exam (SAA-C03 2.2 "highly available and/or fault-tolerant architectures", 2.1; SOA-C03 2.2, 2.3): Multi-AZ for HA, Multi-Region for DR, the four strategies by RPO / RTO / cost, Route 53 failover routing with health checks, Aurora Global Database / DynamoDB global tables / S3 CRR for cross-Region data, AWS Backup cross-Region copies, service quotas in the standby Region, "most cost-effective that meets RTO X" questions.

In an interview: "What's the difference between high availability and disaster recovery?" - "HA keeps the service running through expected failures - an instance, an AZ - automatically, with redundancy in one Region. DR is how you recover from a disaster or a mistake that takes the whole thing down: another Region, or backups when the data itself is corrupted. HA has no RPO to speak of; DR is chosen by the RPO and RTO the business accepts and what it will pay."

You can now: define RPO and RTO and get them from the business, review a design for single points of failure, spread a service across AZs, pick backup-and-restore, pilot light, warm standby or multi-site for a requirement and budget, list what besides data must exist in a recovery Region, and explain static stability and why DR must be rehearsed.

Why it helps

Resilience questions make up a quarter of the Solutions Architect exam and most of the hard decisions in a design review. High availability across Availability Zones handles everyday failures; disaster recovery across Regions handles the rare catastrophic ones and the mistakes that replication copies. Choosing between backup and restore, pilot light, warm standby and multi-site by RPO, RTO and budget - and knowing what else must exist in the recovery Region - is what makes a recovery plan real.

Commands in this lesson

aws cd cat

FAQ

What is the difference between RPO and RTO?

RPO, the recovery point objective, is how much data the business can afford to lose, measured as time since the last good copy. RTO, the recovery time objective, is how long it can be down. Backups every six hours give an RPO of up to six hours; how long a restore takes determines the RTO.

Why is replication not a backup?

Replication copies every change, including mistakes: a dropped table, corrupted data or ransomware encryption arrives in the replica within seconds. Backups and point-in-time recovery keep older versions you can go back to. A resilient design needs both: replication for availability and RPO, backups for recovery from errors.

Which DR strategy is the cheapest?

Backup and restore: only backups exist in the recovery Region and everything is rebuilt from infrastructure as code during a disaster. It has RPO and RTO in hours. Pilot light, warm standby and multi-site active/active cost progressively more because more is running in the second Region all the time.

What is static stability?

A design where the recovery path does not depend on making changes during the outage, especially through a failing control plane. Capacity is already provisioned and traffic is shifted with data-plane mechanisms such as Route 53 health checks, so recovery works even when APIs to create resources are degraded.

Why must service quotas be checked in the recovery Region?

Quotas apply per Region. A warm standby or pilot light that needs to scale to full size during a disaster may hit the default limits for instances, tasks or Lambda concurrency in that Region. Raise them in advance, or the failover fails exactly when it is needed.

In an interview Mid

What is the difference between high availability and disaster recovery?

High availability keeps a service running through expected failures - an instance, an Availability Zone - automatically, with redundancy inside one Region. Disaster recovery is how you get back after losing a Region or after a mistake that replication copies, using another Region or backups. DR strategies range from backup and restore through pilot light and warm standby to multi-site, chosen by the RPO and RTO the business accepts and what it will pay - and rehearsed.

Also asked: How do you choose between pilot light and warm standby? · What must exist in the recovery Region besides the data? · How would you test a disaster recovery plan?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.