OnCallReadyStart learning

AWS Lesson 38.28 · AWS V: Data, Resilience & the Edge · 14 min read

Route 53 routing policies and health checks: DNS that answers per query

In plain words

Route 53 is the receptionist who answers "where is the shop?". A plain record gives everyone the same address. With a routing policy, the receptionist answers per caller: nine callers in ten get the old shop and one gets the new one (weighted), each caller gets the nearest branch (latency), and if the main branch is closed the callers are sent to the backup branch (failover). Health checks are the receptionist phoning each branch every few seconds to see whether it is open.

The catch: callers write the address down and keep using it for a while, so a change takes time to reach everyone.

Route 53 routing policies and health checks: DNS that answers per query

AWS II pointed a name at a load balancer with an alias record. A DNS name can do more than hold one answer: Route 53 can answer differently per query. It can split traffic by weight, send each user to the closest Region, or stop returning an endpoint whose health check fails. That makes DNS a failover mechanism between Regions, with one big caveat: clients cache answers for the TTL, so DNS failover is never instant.

Need to know: records with a routing policy come in sets with the same name and type, each with a SetIdentifier: weighted (shares), latency (lowest latency Region for the resolver), failover (PRIMARY / SECONDARY), geolocation (by continent or country, with a default), geoproximity (by distance with a bias), multivalue answer (up to 8 healthy values), IP-based (by the resolver's CIDR). Health checks probe an endpoint from many locations (HTTP, HTTPS, TCP, string match), or combine others (calculated), or follow a CloudWatch alarm (for private endpoints). A record attached to a failing health check is left out of answers.

Two web servers, one name

The zone shop.oncall-lab.example has two servers in different subnets (web-a, web-b) and a health check on each:

$ Z=$(aws route53 list-hosted-zones-by-name --dns-name shop.oncall-lab.example --query 'HostedZones[0].Id' --output text)
$ aws route53 list-resource-record-sets --hosted-zone-id $Z --query "ResourceRecordSets[?Type=='A'].[Name,SetIdentifier,Failover,TTL,ResourceRecords[0].Value,HealthCheckId]" --output text
app.shop.oncall-lab.example.	primary	PRIMARY	60	3.66.184.122	94a3cab6-745d-48a6-a843-7913c91ddd11
app.shop.oncall-lab.example.	secondary	SECONDARY	60	3.75.97.100	None
$ aws route53 list-health-checks --query 'HealthChecks[].[Id,HealthCheckConfig.Type,HealthCheckConfig.IPAddress,HealthCheckConfig.ResourcePath,HealthCheckConfig.FailureThreshold,HealthCheckConfig.RequestInterval]' --output text
94a3cab6-745d-48a6-a843-7913c91ddd11	HTTP	3.66.184.122	/healthz	3	10
65a15379-eef2-472a-8470-88e62d2c6605	HTTP	3.75.97.100	/healthz	3	10

app.shop.oncall-lab.example is a failover pair: primary (web-a, with a health check) and secondary (web-b). Route 53 answers with the primary while its health check passes, and with the secondary otherwise. The health checker probes /healthz every 10 seconds and marks it unhealthy after 3 failures in a row: about 30 seconds to detect, plus the record's TTL (60 s) before every client has asked again.

$ aws route53 get-health-check-status --health-check-id $(aws route53 list-health-checks --query "HealthChecks[?HealthCheckConfig.IPAddress=='$(aws ec2 describe-instances --filters Name=tag:Name,Values=web-a Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)'].Id" --output text) --query 'HealthCheckObservations[0].[Region,StatusReport.Status]' --output text
us-east-1	Success: HTTP Status Code 200, OK
$ aws route53 test-dns-answer --hosted-zone-id $Z --record-name app.shop.oncall-lab.example --record-type A --query '[RecordData,ResponseCode]' --output text
NOERROR
3.66.184.122
$ dig +noall +answer app.shop.oncall-lab.example
app.shop.oncall-lab.example.	60	IN	A	3.66.184.122

test-dns-answer asks Route 53 what it would answer, without any resolver cache in the way: the first thing to run when "DNS failover did not work".

A failover, end to end

$ aws ec2 stop-instances --instance-ids $(aws ec2 describe-instances --filters Name=tag:Name,Values=web-a Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].InstanceId' --output text) --query 'StoppingInstances[0].CurrentState.Name' --output text
stopping
$ sleep 45
$ aws route53 get-health-check-status --health-check-id $(aws route53 list-health-checks --query 'HealthChecks[?HealthCheckConfig.ResourcePath==`/healthz`]|[0].Id' --output text) --query 'HealthCheckObservations[0].StatusReport.Status' --output text
Failure: Connection timed out. The endpoint or the internet connection is down, or requests are being blocked by your firewall. See https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-failover-router-firewall-rules.html
$ aws route53 test-dns-answer --hosted-zone-id $Z --record-name app.shop.oncall-lab.example --record-type A --query RecordData --output text
3.75.97.100
$ dig +short app.shop.oncall-lab.example
3.75.97.100

The checker saw the failure, and the answer is now web-b's address. In production the next minute matters. Resolvers and clients that cached the old answer keep using it until the TTL runs out, and some clients (old JVMs, connection pools, mobile apps) keep it much longer. Keep TTLs on failover records short (30-60 s), and make sure the secondary can take the full load. Failover moves traffic, not capacity.

Weighted and latency: splitting and steering

$ aws route53 change-resource-record-sets --hosted-zone-id $Z --change-batch file://$HOME/oncall-lab/labs/aws/try5/canary.json --query 'ChangeInfo.Status' --output text
PENDING
$ for i in $(seq 1 50); do dig +short new.shop.oncall-lab.example; done | sort | uniq -c
     44 192.0.2.10
      6 192.0.2.20
$ aws route53 change-resource-record-sets --hosted-zone-id $Z --change-batch '{"Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"x.shop.oncall-lab.example","Type":"A","Weight":10,"TTL":60,"ResourceRecords":[{"Value":"192.0.2.1"}]}}]}'
aws: [ERROR]: An error occurred (InvalidInput) when calling the ChangeResourceRecordSets operation: Invalid request: Missing field 'SetIdentifier' in Change with [Action=UPSERT, Name=x.shop.oncall-lab.example., Type=A, SetIdentifier=null]

The canary file sends weight 90 to the current version and 10 to the new one: about one answer in ten goes to the canary. A weight of 0 takes an endpoint out without deleting it. Every record in a routing set needs a SetIdentifier (the error says so). Latency records (one per Region, "Region": "eu-central-1") answer with the Region that has the lowest measured latency from the user's resolver, the basis of active-active multi-Region designs. Geolocation answers by where the user is (continent, country, US state). It is for compliance and localisation, not for performance, and it needs a default (CountryCode: "*") or users elsewhere get no answer. Multivalue answer returns up to eight healthy values: a simple client-side load spreading, not a load balancer.

Health checks for what the internet cannot reach

Route 53's health checkers run on the internet. A private endpoint (an internal ALB, an instance in a private subnet) cannot be probed directly. The pattern is a CloudWatch alarm on a metric that proves health (the ALB's HealthyHostCount, a canary's success metric) and a health check of type CLOUDWATCH_METRIC that follows the alarm. A calculated health check combines up to 256 others ("healthy when at least 2 of 3 are"). For alias records to AWS resources (load balancers, CloudFront, S3 websites), EvaluateTargetHealth: true uses the target's own health instead of a separate check: an ALB with no healthy targets makes the record unhealthy.

Other Route 53 parts on the exams:

  • Private hosted zones answer only inside the VPCs associated with them (enableDnsSupport and enableDnsHostnames must be on), and win over public zones with the same name there.
  • Route 53 Resolver endpoints connect VPC DNS with on-premises DNS: inbound endpoints let on-premises servers resolve private zones, outbound endpoints plus forwarding rules let instances resolve corp.example against on-premises servers. Resolver DNS Firewall blocks lookups of bad domains from the VPC.
  • Query logging records the queries a public zone receives (to CloudWatch Logs in us-east-1); Resolver query logging records what the VPC asks.
  • Route 53 Application Recovery Controller adds readiness checks and routing controls for Region failover that does not depend on health checks guessing.

On the exam (SOA-C03 5.2, SAA-C03 2.2 and 3.4): "active-passive DR across Regions" = failover routing + health check on the primary; "send users to the Region with the lowest latency" = latency routing; "blue/green or canary by percentage" = weighted; "serve content by country (compliance, language)" = geolocation (with a default record); "route by distance and shift traffic with a bias" = geoproximity; "several healthy IPs, no load balancer" = multivalue answer; "health check a private resource" = CloudWatch alarm health check. Aliases are free, work at the zone apex, and can evaluate target health.

In an interview: "DNS failover did not work during a Region outage. Where do you look?" - "Was the health check actually failing (get-health-check-status from its Regions; does it check a page that really depends on the backend)? What does Route 53 answer right now (test-dns-answer)? Then the caches: the record's TTL, resolvers that ignore it, clients and connection pools that never re-resolve. And whether the secondary could take the load, or quietly failed too."

You can now: read routing sets and health checks, explain failover timing (detection plus TTL plus client caching), use test-dns-answer and dig to see what Route 53 answers, split traffic with weights, and choose between failover, latency, geolocation, geoproximity, weighted and multivalue routing, including health checks for private endpoints.

Why it helps

DNS is the usual way to move traffic between Regions, for disaster recovery, canaries and latency. It is also the mechanism people trust without measuring: failover takes detection time plus the TTL plus however long clients cache the answer.

Knowing how to ask Route 53 what it answers right now, how health checks decide, and what a record without a health check does, turns "DNS failover did not work" from a mystery into a short checklist.

Commands in this lesson

aws dig sleep

FAQ

How long does a DNS failover take?

Detection plus caching. In the lab the health checker probes /healthz every 10 seconds and marks the endpoint unhealthy after 3 failures, about 30 seconds; then clients keep the old answer for the record's TTL, 60 seconds. Some clients, such as old JVMs, connection pools and mobile apps, keep it far longer. Keep failover TTLs short, 30 to 60 seconds.

What does Route 53 answer when both endpoints of a failover pair are down?

It depends on the secondary. A record without a health check is always considered healthy, so Route 53 keeps answering with the secondary's address. That is often intended, for a static "sorry" page, but it also means a broken secondary is served anyway. Attach a health check to it if that matters.

How do I health-check an endpoint the internet cannot reach?

Route 53's checkers run on the internet. For a private endpoint, create a CloudWatch alarm on a metric that proves health, such as an internal ALB's HealthyHostCount, and a health check of type CLOUDWATCH_METRIC that follows it. For alias records to AWS resources, EvaluateTargetHealth uses the target's own health instead.

What is the difference between latency and geolocation routing?

Latency routing answers with the Region that has the lowest measured latency from the user's resolver: it is about performance. Geolocation answers by where the user is, by continent, country or US state, for compliance and localisation. Geolocation needs a default record (CountryCode "*"), or users elsewhere get no answer.

How do I check what Route 53 is answering?

test-dns-answer asks Route 53 directly what it would answer for a name and type, without any resolver cache in the way, so it is the first thing to run when failover seems broken. Compare it with dig, which shows what your resolver hands out, and get-health-check-status, which shows what each checker Region sees.

In an interview Mid

DNS failover did not work during a Region outage. Where do you look?

First, was the health check really failing: get-health-check-status from its checker Regions, and does it check a page that actually depends on the backend? Then what Route 53 answers right now with test-dns-answer, which bypasses every cache. Then the caches: the record's TTL, resolvers that ignore it, clients and connection pools that never re-resolve. Finally whether the secondary could take the full load or quietly failed too, because a failover record moves traffic, not capacity.

Also asked: What is the difference between latency-based and geolocation routing in Route 53? · How would you send ten percent of traffic to a new version with Route 53? · Why might clients keep using an old IP address after a DNS change?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.