OnCallReadyStart learning

AWS Lesson 38.15 · AWS V: Data, Resilience & the Edge · 13 min read

ElastiCache: when a cache helps, and what happens when it fails over

In plain words

A cache is a sticky note on your monitor with the answer to a question people ask you a hundred times a day. You read the note instead of looking it up again. The note has an expiry date, the TTL, because the answer might change. If it changes and nobody updates the note, you give the old answer: stale data. If the note falls off at the busiest moment, everyone runs to look it up at once: a thundering herd.

ElastiCache keeps those notes in memory for many servers, with a copy on a second node that takes over if the first one fails.

ElastiCache: when a cache helps, and what happens when it fails over

A cache trades freshness for speed and load. Put the answer to an expensive question in memory, serve the next thousand requests from there, and the database sees one query instead of a thousand. Amazon ElastiCache runs the in-memory store for you: Valkey (the open-source fork of Redis that AWS now recommends), Redis OSS, or Memcached. This lesson connects to a Valkey replication group, uses the cache-aside pattern, and fails the primary over.

Need to know: a replication group has one primary (reads and writes) and up to five replicas per shard (reads only). With cluster mode disabled there is one shard and two endpoints: the primary endpoint for writes and the reader endpoint for reads. With Multi-AZ + automatic failover, a replica in another AZ is promoted when the primary fails, and the primary endpoint's DNS moves to it. Caches are inside your VPC: no public access, security groups on port 6379.

The cache in the lab

$ aws elasticache describe-replication-groups --replication-group-id shop-sessions --query 'ReplicationGroups[0].{engine: Engine, status: Status, nodes: MemberClusters, failover: AutomaticFailover, multiAZ: MultiAZ, clusterMode: ClusterEnabled}'
{
    "engine": "valkey",
    "status": "available",
    "nodes": [
        "shop-sessions-001",
        "shop-sessions-002"
    ],
    "failover": "enabled",
    "multiAZ": "enabled",
    "clusterMode": false
}
$ aws elasticache describe-replication-groups --replication-group-id shop-sessions --query 'ReplicationGroups[0].NodeGroups[0].{primary: PrimaryEndpoint.Address, reader: ReaderEndpoint.Address, members: NodeGroupMembers[].[CacheClusterId,CurrentRole,PreferredAvailabilityZone]}'
{
    "primary": "master.shop-sessions.4ac577.euc1.cache.amazonaws.com",
    "reader": "replica.shop-sessions.4ac577.euc1.cache.amazonaws.com",
    "members": [
        [
            "shop-sessions-001",
            "primary",
            "eu-central-1a"
        ],
        [
            "shop-sessions-002",
            "replica",
            "eu-central-1b"
        ]
    ]
}

Two nodes in two AZs, one primary, one replica, automatic failover on. Applications use the endpoints, never a node's own address, for the same reason as with RDS: roles move.

Cache-aside, by hand

The shop's product page needs "top sellers of the week", a query that takes the database a second. Cache-aside (lazy loading): look in the cache; on a miss, ask the database and store the answer with a TTL:

$ APP=$(aws ec2 describe-instances --filters Name=tag:Name,Values=app-1 Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].InstanceId' --output text)
$ aws ssm start-session --target $APP

Starting session with SessionId: learner-01403a097e3540e4f
$ valkey-cli -h master.shop-sessions.4ac577.euc1.cache.amazonaws.com PING
PONG
$ valkey-cli -h master.shop-sessions.4ac577.euc1.cache.amazonaws.com GET topsellers:week
(nil)
$ valkey-cli -h master.shop-sessions.4ac577.euc1.cache.amazonaws.com SET topsellers:week '[4711,4802,5310]' EX 300
OK
$ valkey-cli -h master.shop-sessions.4ac577.euc1.cache.amazonaws.com GET topsellers:week
"[4711,4802,5310]"
$ valkey-cli -h master.shop-sessions.4ac577.euc1.cache.amazonaws.com TTL topsellers:week
(integer) 300

(nil) was the miss. The application then runs the query and SET ... EX 300: the next five minutes of requests are served from memory. The TTL is the freshness contract: a product that sells out may show for up to five minutes. When that is not acceptable, the writer also invalidates (DEL topsellers:week) or updates the cache (write-through). The usual trouble with caches:

  • stale data after a write that did not invalidate;
  • a thundering herd when a hot key expires and a thousand requests miss at once (staggered TTLs, or one request refreshes while the others serve the old value);
  • the cache becomes load-bearing: if the database cannot take the uncached load, a cache restart is an outage. Size the database for a cold cache, or warm the cache first.

Replicas are read-only

$ valkey-cli -h replica.shop-sessions.4ac577.euc1.cache.amazonaws.com GET topsellers:week
"[4711,4802,5310]"
$ valkey-cli -h replica.shop-sessions.4ac577.euc1.cache.amazonaws.com SET cart:42 x
(error) READONLY You can't write against a read only replica.
$ valkey-cli -h master.shop-sessions.4ac577.euc1.cache.amazonaws.com INFO replication | head -3
# Replication
role:master
connected_slaves:1
$ exit


Exiting session with sessionId: learner-01403a097e3540e4f.

The reader endpoint spreads reads over the replicas; a write there gets READONLY You can't write against a read only replica.. Replication is asynchronous, so a read just after a write may miss it.

A failover

$ aws elasticache test-failover --replication-group-id shop-sessions --node-group-id 0001 --query 'ReplicationGroup.Status' --output text
modifying
$ sleep 30
$ aws elasticache describe-replication-groups --replication-group-id shop-sessions --query 'ReplicationGroups[0].[Status,NodeGroups[0].NodeGroupMembers[].[CacheClusterId,CurrentRole]]' --output text
available
shop-sessions-001	replica
shop-sessions-002	primary
$ aws elasticache describe-events --source-identifier shop-sessions --source-type replication-group --duration 10 --query 'Events[].Message' --output text
Test Failover API called for node group 0001	Failover to replica node completed

test-failover does what a dead primary does: the replica is promoted, the primary endpoint's DNS name moves to it, and the old primary comes back as a replica. Clients that connect by the primary endpoint and reconnect on errors carry on. Writes in the last moment before the failover can be lost (asynchronous replication): a cache is not a system of record. Sessions and carts in a cache need a fallback, or a store that is durable.

Choosing the engine and the shape

Valkey / Redis OSSMemcached
data typesstrings, hashes, lists, sets, sorted sets, streamsstrings only
replication, failoveryes (replicas, Multi-AZ)no
persistence / snapshotsyesno
scalingcluster mode: data sharded over up to 500 shardsadd nodes, client-side hashing
typical usesessions, leaderboards, rate limits, queues, cachea simple, multi-threaded object cache
  • Cluster mode enabled splits the key space over shards (16,384 hash slots) for more memory and write throughput than one node; clients must be cluster-aware and use the configuration endpoint.
  • ElastiCache Serverless removes node sizing: it scales by itself and bills per data stored and per request.
  • Security: in-transit encryption (TLS), at-rest encryption, and AUTH (a token, or Valkey/Redis users and ACLs), all set at creation for older engine versions.
  • Memory full: the maxmemory-policy decides what is evicted (volatile-lru by default: keys with a TTL, least recently used first). Watch Evictions, CurrentConnections, EngineCPUUtilization and DatabaseMemoryUsagePercentage in CloudWatch.

Other caches you already know: CloudFront caches HTTP responses at the edge (a later lesson), DAX caches DynamoDB reads, and RDS read replicas take read load without a cache's staleness rules.

On the exam (SOA-C03 2.1, SAA-C03 3.3): "reduce database read load / latency for repeated queries" = ElastiCache (or DAX for DynamoDB). "Sessions that survive a web server failure" = sessions in ElastiCache (or DynamoDB), not on the instance. "High availability for the cache" = Multi-AZ with automatic failover (Valkey/Redis, not Memcached). "Simple, multi-threaded, no persistence" = Memcached. Lazy loading may serve stale data; write-through never misses but caches data nobody reads.

In an interview: "When would you not add a cache?" - "When the data must always be fresh (a balance, stock in a checkout), when the access pattern has no reuse, or when the real problem is a missing index. A cache adds a consistency question, a failure mode (cold cache, thundering herd) and a dependency. Fix the query first; cache what is read often, changes rarely, and can be a little stale."

You can now: read a Valkey replication group's endpoints and roles, use cache-aside with a TTL and explain invalidation, stale data and the thundering herd, run and read a failover, and choose between Valkey/Redis OSS and Memcached, cluster mode and serverless.

Why it helps

A cache can turn a one-second query into a millisecond read and spare the database a thousand identical requests. It also adds a consistency question, new failure modes and a dependency, and sometimes the database can no longer survive without it.

The lesson uses cache-aside by hand, shows why a replica refuses writes, runs a failover and reads what moved, so that "add a cache" becomes a decision with known costs rather than a reflex.

Commands in this lesson

aws valkey-cli exit sleep

FAQ

Valkey, Redis OSS or Memcached?

Valkey, the open-source fork of Redis that AWS now recommends, and Redis OSS have rich data types, replication, Multi-AZ failover and snapshots: sessions, leaderboards, rate limits and caches. Memcached stores strings only and has no replication or persistence; it suits a simple, multi-threaded object cache. For high availability you need Valkey or Redis OSS.

What is cache-aside?

Lazy loading: look in the cache first; on a miss (nil), ask the database and store the answer with a TTL, for example SET topsellers:week ... EX 300. The next requests within those five minutes come from memory. The TTL is the freshness contract; when it is not good enough, the writer also deletes the key or updates it (write-through).

Why do I get READONLY when I write?

You wrote through the reader endpoint or a replica. Replicas are read-only: writes go to the primary endpoint, and reads can be spread over replicas through the reader endpoint. Replication is asynchronous, so a read from a replica right after a write may not see it yet.

What happens during a failover?

With Multi-AZ and automatic failover, a replica in another AZ is promoted and the primary endpoint's DNS name moves to it; the old primary returns as a replica. test-failover does the same on purpose. Clients that use the endpoint and reconnect on errors carry on. Writes from the last moment can be lost, so a cache is not a system of record.

What happens when the cache's memory is full?

The maxmemory-policy decides what is evicted; the default volatile-lru removes keys with a TTL, least recently used first. Watch Evictions, CurrentConnections, EngineCPUUtilization and DatabaseMemoryUsagePercentage in CloudWatch. Many evictions mean the cache is too small for its working set or keys never expire.

In an interview Mid

When would you not add a cache?

When the data must always be fresh, like a balance or the stock in a checkout; when the access pattern has no reuse; or when the real problem is a missing index. A cache adds a consistency question (stale data after a write that did not invalidate), failure modes such as a cold cache and the thundering herd, and a dependency the database may come to rely on. Fix the query first, then use cache-aside with a TTL for data that is read often, changes rarely and can be a little stale.

Also asked: What is the difference between lazy loading and write-through caching? · How do you keep an ElastiCache replication group available through a node failure? · What is a thundering herd, and how do you prevent it?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.