OnCallReadyStart learning

AWS Lesson 38.3 · AWS V: Data, Resilience & the Edge · 22 min read

Multi-AZ, read replicas and the failover your clients must survive

In plain words

RDS can keep two kinds of copy. The first is an understudy in the wings: they learn every line as it is spoken, never go on stage, and step in by themselves the moment the lead falls ill. That is the Multi-AZ standby. The second is a second performance in another hall, a few moments behind: anyone can watch it, but it never replaces the lead unless you promote it. That is a read replica.

When the understudy takes over, every guest is asked to leave and come back in. Guests who remember the theatre's name find the new lead; guests who wrote down the old seat number wait forever.

Multi-AZ, read replicas and the failover your clients must survive

RDS gives you two different kinds of copy, and people mix them up in interviews and in designs. A Multi-AZ standby exists for availability. It is written synchronously in another AZ, you cannot read from it, and it takes over by itself when the primary fails. A read replica exists for read scaling. It is fed asynchronously, you can read from it (it may be behind), and it never takes over unless you promote it. This lesson fails over a real instance with real applications connected, and shows which client settings decide whether users notice.

Need to know: a failover moves the endpoint's CNAME to the standby (60-120 s on AWS, 35 s here) and cuts every open session. Clients survive it when they connect by name, do not cache DNS for long, and use a pool that validates and replaces dead connections. RDS Proxy keeps client connections open through a failover. Aurora shares one storage volume between a writer and up to 15 readers, behind a cluster endpoint and a reader endpoint.

Two kinds of copy

Multi-AZ instanceMulti-AZ DB clusterRead replicaAurora reader
purposeavailabilityavailability + readsread scaling, DRreads + availability
replicationsynchronoussemi-synchronousasynchronous (lag)shared storage
can you read it?noyes (2 standbys)yesyes
takes over by itself?yesyesno: you promote ityes
other Region?nonoyes (cross-Region replica)Aurora Global Database
$ aws rds describe-db-instances --query 'DBInstances[?Engine==`postgres`].[DBInstanceIdentifier,MultiAZ,AvailabilityZone,SecondaryAvailabilityZone,ReadReplicaSourceDBInstanceIdentifier]' --output table
----------------------------------------------------------------------
|                         DescribeDBInstances                        |
+------------+--------+-----------------+-----------------+----------+
|  shop-db   |  True  |  eu-central-1b  |  eu-central-1a  |  None    |
|  shop-db-ro|  False |  eu-central-1a  |  None           |  shop-db |
+------------+--------+-----------------+-----------------+----------+
$ aws rds describe-db-instances --db-instance-identifier shop-db --query 'DBInstances[0].ReadReplicaDBInstanceIdentifiers' --output text
shop-db-ro

shop-db is Multi-AZ: primary in one AZ, standby in the other, and you never see the standby's address. shop-db-ro is a read replica of it, with its own endpoint, in one AZ.

A failover, with applications connected

Two applications run on app-1: checkout connects straight to shop-db's endpoint, cart goes through an RDS Proxy (more on that below). Both log like Spring Boot with HikariCP. Force a failover, which is exactly how you test that clients survive one:

$ dig +short shop-db.c3a56c7217fa.eu-central-1.rds.amazonaws.com
ip-10-30-11-19.eu-central-1.compute.internal.
10.30.11.19
$ aws rds reboot-db-instance --db-instance-identifier shop-db --force-failover --query 'DBInstance.DBInstanceStatus' --output text
rebooting
$ sleep 40
$ aws rds describe-events --source-type db-instance --source-identifier shop-db --duration 10 --query 'Events[].[Date,Message]' --output text
2026-09-22T20:00:04.100+00:00	Multi-AZ instance failover started
2026-09-22T20:00:39.100+00:00	Multi-AZ instance failover completed
2026-09-22T20:00:39.100+00:00	DB instance restarted
$ aws rds describe-db-instances --db-instance-identifier shop-db --query 'DBInstances[0].[DBInstanceStatus,AvailabilityZone,SecondaryAvailabilityZone]' --output text
available	eu-central-1a	eu-central-1b
$ dig +short shop-db.c3a56c7217fa.eu-central-1.rds.amazonaws.com
ip-10-30-10-67.eu-central-1.compute.internal.
10.30.10.67

The events are the record of what happened. The AZs swapped, and the endpoint now resolves to the other address. What the applications saw:

$ APP=$(aws ec2 describe-instances --filters Name=tag:Name,Values=app-1 Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].InstanceId' --output text)
$ aws ssm start-session --target $APP

Starting session with SessionId: learner-0eb1902a958853116
$ grep -l ' WARN \| ERROR ' /var/log/checkout/app.log /var/log/cart/app.log
/var/log/checkout/app.log
$ grep -v ' INFO ' /var/log/checkout/app.log | sed -E 's/^.{23}.* : //' | awk '!seen[substr($0, 1, 45)]++' | cut -c1-140
HikariPool-1 - Cannot acquire connection from data source org.postgresql.ds.PGSimpleDataSource (org.postgresql.util.PSQLException: Connectio
SQL Error: 0, SQLState: null
HikariPool-1 - Connection is not available, request timed out after 30000ms (total=0, active=0, idle=0, waiting=20)
$ exit


Exiting session with sessionId: learner-0eb1902a958853116.

checkout lost its pool (every session was terminated), could not connect while the standby was being promoted, then reconnected, by name, to the new primary. Requests that arrived during that window failed. cart saw nothing.

What clients must do

  • Connect by the endpoint name. An application that resolved the name once and kept the IP (a config file with 10.30.11.19, a hosts entry, a JVM with networkaddress.cache.ttl=-1) keeps connecting to the old primary's address, which is now the standby: it times out forever. Java caches DNS forever when a security manager is installed and 30 s otherwise. Set networkaddress.cache.ttl to a few seconds for database clients.
  • Validate and recycle connections. HikariCP tests a connection before handing it out and replaces dead ones; maxLifetime (shorter than any server-side timeout) makes sure no connection lives forever on an old address. psycopg's pool has check and max_lifetime.
  • Retry what is safe to retry. A failover always fails the requests in flight. Idempotent reads can retry; a payment cannot blindly retry.
  • Alarm on the event, not on luck. RDS events (failover started / completed) go to EventBridge and RDS event subscriptions. A failover at 03:00 that "nobody noticed" means nobody checked whether every client came back.

RDS Proxy: the pool in front of the database

RDS Proxy is a managed connection pool with its own endpoint. Clients connect to the proxy; the proxy holds a smaller set of connections to the database and multiplexes client transactions onto them. Three reasons to use it: thousands of short-lived clients (Lambda functions) that would exhaust max_connections, failover (the proxy keeps client connections open and reattaches them to the new primary, so clients see a pause instead of an error), and IAM authentication with the password kept in Secrets Manager.

$ aws rds describe-db-proxies --db-proxy-name shop-proxy --query 'DBProxies[0].{status: Status, endpoint: Endpoint, auth: Auth[0].AuthScheme, secret: Auth[0].SecretArn, tls: RequireTLS, sgs: VpcSecurityGroupIds}'
{
    "status": "available",
    "endpoint": "shop-proxy.proxy-c3a56c7217fa.eu-central-1.rds.amazonaws.com",
    "auth": "SECRETS",
    "secret": "arn:aws:secretsmanager:eu-central-1:111122223333:secret:lab/shop-db/dbadmin-MdadYv",
    "tls": true,
    "sgs": [
        "sg-09147540b7423e757"
    ]
}
$ aws rds describe-db-proxy-targets --db-proxy-name shop-proxy --query 'Targets[].[RdsResourceId,Type,TargetHealth.State]' --output text
shop-db	RDS_INSTANCE	AVAILABLE
$ aws ec2 describe-security-groups --filters Name=group-name,Values=data-db --query 'SecurityGroups[0].IpPermissions[].UserIdGroupPairs[].GroupId' --output text
sg-0469a55e27b3c238a	sg-09147540b7423e757

The proxy reads the database password from a Secrets Manager secret (through an IAM role it assumes), and it has its own security group, which the database's security group must allow on

  1. Otherwise the target shows UNAVAILABLE with "Database host is unreachable" and clients hang.

Session state (SET, temporary tables, prepared statements) pins a client to one database connection, which reduces the multiplexing: keep session state out of pooled traffic, as with PgBouncer in transaction mode.

IAM database authentication

Instead of a password, a client can present a 15-minute token signed with its IAM credentials:

$ aws rds generate-db-auth-token --hostname shop-db.c3a56c7217fa.eu-central-1.rds.amazonaws.com --port 5432 --username app_iam --region eu-central-1 | cut -c1-120
shop-db.c3a56c7217fa.eu-central-1.rds.amazonaws.com:5432/?Action=connect&DBUser=app_iam&X-Amz-Algorithm=AWS4-HMAC-SHA256

The token is generated locally (a SigV4 presigned URL, no API call). It works when the instance has --enable-iam-database-authentication, the database user is granted the rds_iam role, and the caller's IAM policy allows rds-db:connect on arn:aws:rds-db:eu-central-1:111122223333:dbuser:<DbiResourceId>/app_iam. Note the resource ID, not the instance name. No password lives in the app's config, and access is revoked by changing IAM. TLS is required.

Read replicas: reading from a copy that may be behind

$ aws ssm start-session --target $APP

Starting session with SessionId: learner-0eb1902a958853116
$ psql "host=shop-db-ro.c3a56c7217fa.eu-central-1.rds.amazonaws.com user=dbadmin dbname=orders" -Atc 'select count(*) from orders'
psql: error: connection to server at "shop-db-ro.c3a56c7217fa.eu-central-1.rds.amazonaws.com" (10.30.10.141), port 5432 failed: FATAL:  password authentication failed for user "dbadmin"
$ psql "host=shop-db-ro.c3a56c7217fa.eu-central-1.rds.amazonaws.com user=dbadmin dbname=orders" -c "update orders set status = 'PAID' where id = 1"
psql: error: connection to server at "shop-db-ro.c3a56c7217fa.eu-central-1.rds.amazonaws.com" (10.30.10.141), port 5432 failed: FATAL:  password authentication failed for user "dbadmin"
$ exit


Exiting session with sessionId: learner-0eb1902a958853116.
$ aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name ReplicaLag --dimensions Name=DBInstanceIdentifier,Value=shop-db-ro --start-time $(date -u -d '-10 min' +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 300 --statistics Maximum --query 'Datapoints[].Maximum' --output text
0	0	0

A replica is read-only and asynchronous: a write is visible on the primary first and on the replica after ReplicaLag seconds. Reports, search indexing and analytics belong there, but never "write then read your own write" flows. Replicas need automated backups on the source (retention above 0), can live in another Region (a cheap cross-Region DR copy), and promote-read-replica turns one into a standalone, writable instance. Promotion is one-way and manual, and the app must be pointed at the new endpoint. That is why a replica is not a substitute for Multi-AZ.

Aurora: one volume, many readers

$ aws rds describe-db-clusters --db-cluster-identifier shop-aurora --query 'DBClusters[0].{engine: Engine, version: EngineVersion, writer: Endpoint, readers: ReaderEndpoint, members: DBClusterMembers[].[DBInstanceIdentifier,IsClusterWriter]}'
{
    "engine": "aurora-postgresql",
    "version": "17.7",
    "writer": "shop-aurora.cluster-c3a56c7217fa.eu-central-1.rds.amazonaws.com",
    "readers": "shop-aurora.cluster-ro-c3a56c7217fa.eu-central-1.rds.amazonaws.com",
    "members": [
        [
            "shop-aurora-1",
            true
        ],
        [
            "shop-aurora-2",
            false
        ]
    ]
}
$ aws rds failover-db-cluster --db-cluster-identifier shop-aurora --query 'DBCluster.Status' --output text
failing-over
$ sleep 40
$ aws rds describe-db-clusters --db-cluster-identifier shop-aurora --query 'DBClusters[0].DBClusterMembers[].[DBInstanceIdentifier,IsClusterWriter]' --output text
shop-aurora-1	False
shop-aurora-2	True

Aurora PostgreSQL separates compute from storage: one cluster volume, six copies across three AZs, and instances that all read the same volume. The cluster endpoint always points at the writer; the reader endpoint balances across the readers (read-only sessions). A failover promotes a reader (lowest promotion tier first) in about 30 seconds, with no data copy, because the reader already has the data. Readers show near-zero lag, and Aurora Global Database adds a secondary Region with typically under a second of lag. Aurora Serverless v2 scales capacity (ACUs) with load. The trade-off is cost and an engine that is PostgreSQL-compatible rather than PostgreSQL itself.

On the exam (SOA-C03 2.2, SAA-C03 2.2 and 3.3): "survive an AZ failure with no data loss" = Multi-AZ. "Offload reporting queries" = read replica (or Aurora readers). "DR in another Region" = cross-Region read replica or Aurora Global Database. "Lambda functions exhaust database connections" or "make failover faster for clients" = RDS Proxy. "No passwords in the app" = IAM database authentication. Multi-AZ standbys are never readable, except in a Multi-AZ DB cluster.

In an interview: "What happens to your application during an RDS Multi-AZ failover?" - "Every open connection is dropped, the endpoint's CNAME moves to the promoted standby within a minute or two, and clients that reconnect by name land on the new primary. The application survives if its pool validates and replaces dead connections and it does not cache DNS or pin an IP; requests in flight fail and should be retried if they are idempotent. RDS Proxy hides most of it by keeping client connections open."

You can now: tell a Multi-AZ standby from a read replica and an Aurora reader, force and read a failover, explain what clients must do to survive it (connect by name, short DNS cache, validating pools, retries), describe what RDS Proxy and IAM authentication add, and read a replica's lag.

Why it helps

A failover is a planned event in RDS's design, and it always cuts every open session. Whether users notice depends on the clients: do they connect by name, how long do they cache DNS, does the pool validate and replace dead connections.

Forcing a failover with the applications connected is the only way to know. The lesson also clears up the copy that people mix up in designs and interviews: the standby gives availability, a replica gives reads, and Aurora and RDS Proxy change both stories.

Commands in this lesson

aws dig sleep grep exit psql

FAQ

Can I read from the Multi-AZ standby?

No. A Multi-AZ instance's standby is written synchronously in another AZ, but it accepts no connections and you never see its address. The exception is a Multi-AZ DB cluster, which has two readable standbys. For read scaling use a read replica or Aurora readers, keeping in mind that both may be slightly behind the writer.

How long does a failover take, and what does the application see?

On AWS 60 to 120 seconds (about 35 in the lab). RDS promotes the standby and moves the endpoint's CNAME, which has a 5-second TTL. Every open session is dropped, so requests in flight fail; clients that reconnect by name land on the new primary. A client pinned to the old IP times out, because that address now belongs to the standby.

What does RDS Proxy add?

It is a managed connection pool with its own endpoint. It multiplexes many client connections onto fewer database connections, keeps client connections open through a failover so they see a pause instead of an error, and supports IAM authentication with the password in Secrets Manager. Its own security group must be allowed on 5432 by the database's group, or the target shows UNAVAILABLE.

How does IAM database authentication work?

The client generates a 15-minute token locally with generate-db-auth-token (a signed URL, no API call) and uses it as the password. It works when the instance has IAM authentication enabled, the database user has the rds_iam role, and the caller's policy allows rds-db:connect on the user's ARN, which contains the instance's resource ID, not its name. TLS is required.

How is Aurora different from RDS for PostgreSQL?

Aurora separates compute from storage: one cluster volume with six copies across three AZs, read by a writer and up to 15 readers. The cluster endpoint points at the writer, the reader endpoint balances across readers, and a failover promotes a reader in about 30 seconds without copying data. It costs more, and it is PostgreSQL-compatible rather than PostgreSQL itself.

In an interview Mid

What happens to your application during an RDS Multi-AZ failover?

Every open connection is dropped, and the endpoint's CNAME moves to the promoted standby within a minute or two. Clients that reconnect by name land on the new primary. The application survives if its pool validates and replaces dead connections, has a maxLifetime, and the runtime does not cache DNS for long (networkaddress.cache.ttl of a few seconds in Java), and if nothing pins the old IP. Requests in flight fail and should be retried only when they are idempotent. RDS Proxy hides most of it by keeping client connections open.

Also asked: What is the difference between a Multi-AZ standby and a read replica? · When would you choose Aurora over RDS for PostgreSQL? · Why should an application not read its own writes from a read replica?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.