OnCallReadyStart learning

AWS V: Data, Resilience & the Edge: interview questions

The question you are most likely to get for each topic, a model answer, and what else comes up.

How would you protect a shop's database on AWS against an AZ outage, a bad deploy and the loss of a Region? Mid

Each failure needs its own copy. An AZ outage: Multi-AZ, so the standby takes over and the endpoint moves, with clients that reconnect by name. A bad write or a dropped table: point-in-time restore into a new instance at a second before the mistake, then copy the rows back. A Region: snapshot or recovery point copies to another Region, or a cross-Region read replica. A compromised account: copies in a separate account behind a Vault Lock in compliance mode. Then I test the restore and the failover and record how long they took, because that is the real RTO.

Also asked: When would you choose DynamoDB over RDS for a new service? · How do you make sure backups cannot be deleted by an attacker? · What happens to an application's connections during a database failover?

What do you still own when your database runs on RDS? Mid

That is the shared responsibility model applied to a database. AWS owns the hardware, the operating system, patching mechanics, the backup machinery, the standby and the failover. I own the schema, queries and indexes, users and their privileges (the master user is only in rds_superuser), the parameter group, the security groups and subnets, backup retention and restore tests, maintenance window choices, client connection settings such as pools, timeouts and TLS, and the alarms on things like FreeStorageSpace and DatabaseConnections.

Also asked: Why should applications connect to RDS by its endpoint name rather than an IP address? · How would you stop an RDS instance from running out of storage at night? · What is the difference between static and dynamic parameters in RDS?

Learn it: 38.1 RDS: a managed PostgreSQL you reach by DNS

What happens to your application during an RDS Multi-AZ failover? Mid

Every open connection is dropped, and the endpoint's CNAME moves to the promoted standby within a minute or two. Clients that reconnect by name land on the new primary. The application survives if its pool validates and replaces dead connections, has a maxLifetime, and the runtime does not cache DNS for long (networkaddress.cache.ttl of a few seconds in Java), and if nothing pins the old IP. Requests in flight fail and should be retried only when they are idempotent. RDS Proxy hides most of it by keeping client connections open.

Also asked: What is the difference between a Multi-AZ standby and a read replica? · When would you choose Aurora over RDS for PostgreSQL? · Why should an application not read its own writes from a read replica?

Learn it: 38.3 Multi-AZ, read replicas and the failover your clients must survive

Someone ran DELETE without a WHERE on a production RDS table. How do you recover? Mid

Note the time first. Then restore-db-instance-to-point-in-time into a new instance at a second before the DELETE, with the production security group and parameter group so it is reachable and behaves the same. Copy the deleted rows back from the restored instance (or switch the application over if the damage is wide), verify, and delete the temporary instance. Multi-AZ does not help, because the standby applied the DELETE too. Afterwards, fix why it was possible: who has write access to production, and reviewed scripts. The RPO here is close to zero; the RTO is the restore time.

Also asked: Why does Multi-AZ not protect you from a bad UPDATE? · How would you keep an RDS backup for longer than 35 days? · How do you find out how long a database restore really takes?

Learn it: 38.8 RDS backups: snapshots, point-in-time restore and what RPO really costs

How do you design a DynamoDB table? Mid

List the access patterns first, before any keys. Then pick a partition key with many distinct values that spreads the load, and a sort key that serves the range and order questions, often a type prefix and a date such as ORDER#<date>#<id>, so "orders of a customer, newest first" is one Query with begins_with. Patterns the keys cannot serve get a GSI, sometimes a sparse one. Use condition expressions for uniqueness and optimistic locking, and avoid scans and hot keys.

Also asked: What is the difference between a Query and a Scan in DynamoDB? · What is single-table design, and why do teams use it? · How would you implement optimistic locking in DynamoDB?

Learn it: 38.10 DynamoDB: design for the questions you will ask

DynamoDB is throttling. How do you investigate? Mid

First which kind: the table, a GSI or a single partition. Compare consumed against provisioned capacity and look at ThrottledRequests and the read and write throttle events per operation. A GSI with too little capacity throttles writes to the base table. If the table is far below its capacity, it is a hot partition, and CloudWatch Contributor Insights shows the key. Then fix the cause: on-demand or auto scaling for capacity, key design or write sharding for hot keys, and retries with backoff in the client.

Also asked: What is a hot partition in DynamoDB, and how do you fix it? · When would you choose on-demand capacity over provisioned capacity? · How do DynamoDB global tables resolve conflicting writes?

Learn it: 38.13 DynamoDB in production: capacity, throttling, hot keys, TTL and backups

When would you not add a cache? Mid

When the data must always be fresh, like a balance or the stock in a checkout; when the access pattern has no reuse; or when the real problem is a missing index. A cache adds a consistency question (stale data after a write that did not invalidate), failure modes such as a cold cache and the thundering herd, and a dependency the database may come to rely on. Fix the query first, then use cache-aside with a TTL for data that is read often, changes rarely and can be a little stale.

Also asked: What is the difference between lazy loading and write-through caching? · How do you keep an ElastiCache replication group available through a node failure? · What is a thundering herd, and how do you prevent it?

Learn it: 38.15 ElastiCache: when a cache helps, and what happens when it fails over

Would you put a database on EFS or EBS? Mid

EBS. A database wants a block device with low, predictable latency on one host, and it replicates at the database level, or RDS does it for you. An EBS volume lives in one AZ and attaches to one instance, which is exactly that shape. EFS fits files that many hosts share at once: uploads, shared configuration, home directories, build caches, mounted through a mount target in each AZ. NFS latency and locking semantics are a poor fit for a database's data files.

Also asked: What happens to data on an instance store volume when the instance stops? · How would you share files between instances in different Availability Zones? · When would you use FSx instead of EFS?

Learn it: 38.17 Block, file and object: EBS, instance store, EFS and when each fits

How would you cut an S3 bill of 200 TB? Mid

Find out what is in it first: S3 Storage Lens or an inventory report, grouped by prefix, age and last access. Then expire what nobody needs: old noncurrent versions, incomplete multipart uploads (AbortIncompleteMultipartUpload), logs past their retention. Move cold data with lifecycle rules to IA or Glacier where object sizes and minimum durations make it cheaper, and use Intelligent-Tiering where nobody knows the pattern. Finally, check that the restore times of the new classes still meet the recovery requirements.

Also asked: What is the difference between S3 Standard-IA and One Zone-IA? · How do you read an object that lifecycle moved to Glacier Deep Archive? · Why can moving small objects to Standard-IA cost more than leaving them?

Learn it: 38.19 S3 storage classes, lifecycle and Glacier restores

How do you make sure S3 data survives a compromised admin account? Mid

Replicate to a bucket in a separate account whose admins are different people, with DeleteMarkerReplication off, so deletes in the source do not reach the copy. Lock the copies with Object Lock in compliance mode for the retention you need, so nobody, not even root, can delete them early. Keep versioning with noncurrent-version retention against overwrites, which replication would copy. Then alarm on replication failures and on changes to bucket policies, because an attacker usually starts there.

Also asked: What is the difference between S3 cross-Region and same-Region replication? · Is S3 replication a backup? Why or why not? · What does Object Lock compliance mode protect against?

Learn it: 38.22 S3 replication and Object Lock: copies elsewhere, copies nobody can delete

How do you know your backups work? Mid

Because restores are tested on a schedule. AWS Backup restore testing, or our own job with start-restore-job, restores the latest recovery points into a sandbox, runs data checks such as item counts and the application's own checks, records how long it took, and alarms if it fails. On top of that: alarms on failed or expired backup jobs, a report of resources without a backup plan, and copies in a separate account behind a Vault Lock that the production admins cannot delete.

Also asked: What is the difference between a backup plan and a backup vault in AWS Backup? · How do you make backups ransomware-proof on AWS? · Your plan backs up daily, but the business needs an RPO of one hour. What changes?

Learn it: 38.24 AWS Backup: plans, vaults, copies, locks, and restores you have actually tested

DNS failover did not work during a Region outage. Where do you look? Mid

First, was the health check really failing: get-health-check-status from its checker Regions, and does it check a page that actually depends on the backend? Then what Route 53 answers right now with test-dns-answer, which bypasses every cache. Then the caches: the record's TTL, resolvers that ignore it, clients and connection pools that never re-resolve. Finally whether the secondary could take the full load or quietly failed too, because a failover record moves traffic, not capacity.

Also asked: What is the difference between latency-based and geolocation routing in Route 53? · How would you send ten percent of traffic to a new version with Route 53? · Why might clients keep using an old IP address after a DNS change?

Learn it: 38.28 Route 53 routing policies and health checks: DNS that answers per query

After moving a bucket behind CloudFront with OAC, everything returns 403. Where do you look? Mid

First what returns the 403: x-cache: Error from cloudfront with an S3 AccessDenied body means S3 refused. Then the bucket policy: is there a statement for cloudfront.amazonaws.com with the right distribution ARN in AWS:SourceArn? Is the origin configured with the OAC rather than an old OAI? Is the object encrypted with a KMS key whose key policy does not let CloudFront decrypt? And does the default root object exist, since a missing key is also 403 when the caller cannot list the bucket?

Also asked: How do you stop users from bypassing CloudFront and reading the S3 bucket directly? · How would you improve a CloudFront cache hit ratio? · What is the difference between CloudFront and Global Accelerator?

Learn it: 38.31 CloudFront: caching at the edge, OAC, invalidations and TLS

Your login page gets a credential-stuffing attack. What do you put in place? Mid

At the edge, a WAF rate-based rule scoped to the login path and keyed by IP, or IP plus a header, together with the IP reputation and anonymous-IP managed rule groups, and a CAPTCHA or challenge action for suspicious traffic. I start the new rules in Count mode to check for false positives, then enforce. In the application: account lockouts with backoff, MFA, and alarms on the failed-login rate. Then I read the WAF logs and get-sampled-requests to find the sources and tune the rules.

Also asked: What is the difference between a security group and AWS WAF? · How would you roll out a new managed rule group without breaking customers? · What does Shield Advanced give you over Shield Standard?

Learn it: 38.35 AWS WAF and Shield: filtering requests before they reach you

Two services in peered VPCs time out talking to each other. How do you debug it? Mid

Run the Reachability Analyzer from the client's ENI to the server's on the port: it names the missing piece without sending traffic. By hand, in order: is the peering active, does each side's route table send the other's CIDR to the peering, do the security groups allow the peer's CIDR or group on the port, do the NACLs allow it both ways including ephemeral ports back, and do the CIDRs overlap? VPC Flow Logs on the server's ENI show whether packets arrive and are rejected.

Also asked: When would you choose a Transit Gateway over VPC peering? · Why can two VPCs with overlapping CIDRs not be peered? · How do you separate production and development VPCs on a Transit Gateway?

Learn it: 38.37 Many VPCs: peering, Transit Gateway and the Reachability Analyzer

Office users lose access to an internal app in AWS for a few minutes every few weeks. The VPN shows both tunnels configured. What would you look at? Mid

Whether the customer router really uses both tunnels: many configurations bring up only one, so AWS's tunnel maintenance becomes an outage. I would read TunnelState per tunnel in CloudWatch around the incidents, the VPN connection's tunnel status history, and the BGP sessions. With static routing, failover between tunnels depends on the router noticing a dead tunnel through dead peer detection. Then alarms on each tunnel, sent to the router's owners, and a runbook for Site-to-Site VPN maintenance.

Also asked: What is the difference between Site-to-Site VPN and Direct Connect? · How would you make a Direct Connect link resilient without doubling the cost? · When would you use PrivateLink instead of VPC peering?

Learn it: 38.41 Hybrid networks: Site-to-Site VPN, Direct Connect and PrivateLink

How do you know a system is resilient? Mid

Because it has failed in a test and come back within its RTO and RPO. A forced RDS failover with the clients watching, to prove the pools reconnect. A restore from the newest backup into a sandbox with the time recorded. A DNS failover with the primary stopped, read with test-dns-answer. A DynamoDB or S3 restore that the application's own checks accept. And the alarms that told us about each one. Until then it is a diagram: copies matched to the failure they cover, but unproven.

Also asked: How would you choose between RDS and DynamoDB for a new service? · Which AWS services keep billing while nobody uses them? · What would you test before calling a system resilient?

Learn it: 38.43 Choosing the data and edge services, and what keeps them alive

Practise these answers with flashcards and labs Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.