OnCallReadyStart learning

AWS IV: Operations Automation: interview questions

The question you are most likely to get for each topic, a model answer, and what else comes up.

How would you make an AWS account fix common problems by itself, without causing outages? Mid

Detect, decide, act - with guardrails. AWS Config rules and EventBridge rules detect the problem (a public bucket, SSH opened to the world, a GuardDuty finding). The action is a Systems Manager Automation runbook run with a narrow AutomationAssumeRole: automatic only where one state is acceptable, manual or with an aws:approve step otherwise, and rate-controlled with max-concurrency and max-errors when it touches a fleet. Changes to the infrastructure itself go through CloudFormation change sets or terraform plan that someone reads. Every remediation is visible in CloudTrail, and a remediation that keeps running on the same resource is an alert - it usually means it is fighting another tool.

Also asked: What is the difference between AWS Config and CloudTrail? · How do you patch a fleet of EC2 instances without downtime? · How do you store and rotate database passwords on AWS?

What is a CloudFormation stack, and how do you find out why one failed? Mid

A stack is the set of resources one template made, managed as a unit: created in dependency order, updated together and rolled back together when a step fails. To find the cause I read describe-stack-events oldest first - reverse(StackEvents) - and look for the first CREATE_FAILED or UPDATE_FAILED whose reason is not "cancelled": its ResourceStatusReason is the service's own error. Everything after it is the rollback. A template format error or a missing capability fails earlier, at create-stack.

Also asked: What is the difference between Ref and GetAtt in a template? · How do parameters, mappings and conditions differ in a template? · When would you choose CloudFormation over Terraform?

Learn it: 37.1 CloudFormation: templates, stacks and events

How do you stop a CloudFormation update from replacing a production database or bucket? Mid

Several independent layers. Every update goes through a change set that someone reads, and the pipeline stops on rows with Replacement True for stateful types. A stack policy denies Update:Replace and Update:Delete on that logical ID, so even an approved change needs a deliberate override. DeletionPolicy and UpdateReplacePolicy set to Retain or Snapshot keep the data if the resource is removed or replaced anyway, and termination protection guards the whole stack. IAM and SCPs cover the console.

Also asked: Which kinds of property changes cause a resource replacement? · What does a stack policy protect against, and what not? · How do you review CloudFormation changes in a pipeline?

Learn it: 37.4 Change sets, replacement, and protecting what matters

A stack is in UPDATE_ROLLBACK_FAILED and every deployment fails. What do you do? Mid

First find out why the rollback failed: read the events oldest first - the first UPDATE_FAILED is the original cause, and the failure during the rollback is why it is stuck, usually a resource changed or deleted out of band or a lost permission. If the cause can be fixed, continue-update-rollback retries. If the resource cannot come back, continue-update-rollback --resources-to-skip finishes the rollback, and then I repair the stack - a new logical ID for the lost resource - and run detect-stack-drift to confirm it is IN_SYNC.

Also asked: What is the difference between ROLLBACK_COMPLETE and UPDATE_ROLLBACK_COMPLETE? · How do you detect and fix drift in CloudFormation? · Why can an out-of-band change break a stack update?

Learn it: 37.7 When a stack fails: rollbacks, UPDATE_ROLLBACK_FAILED and drift

How would you roll out the same baseline to every account and Region in an organization? Mid

A service-managed StackSet from the delegated administrator account, targeting the OUs, with automatic deployment so new accounts get the baseline when they join. Operation preferences limit the blast radius - one Region at a time, a small MaxConcurrentCount, zero failure tolerance at first - and I watch list-stack-instances for OUTDATED or INOPERABLE instances, whose stack events say why. Without Organizations, a self-managed StackSet needs the administration and execution roles in place first.

Also asked: What are CloudFormation exports and imports, and what is their downside? · What is the difference between self-managed and service-managed StackSets? · How do you limit the blast radius of a StackSet update?

Learn it: 37.10 Exports, pipelines and StackSets: CloudFormation at scale

How would you automate a fix you keep doing by hand on EC2 instances? Mid

As a Systems Manager Automation runbook: the steps as aws:assertAwsResourceProperty to check the instance is in the right state, aws:runCommand for the work on the machine, and aws:executeAwsApi for the AWS side, run with an AutomationAssumeRole that trusts ssm.amazonaws.com and has only the permissions the steps need. On a fleet I run it against tags with --max-concurrency and --max-errors, add an aws:approve step if it is disruptive, and let EventBridge or an alarm start it so nobody has to.

Also asked: What makes an EC2 instance a managed instance in Systems Manager? · What is the difference between a Command document and an Automation runbook? · When would you choose Parameter Store over Secrets Manager?

Learn it: 37.12 Systems Manager: managed instances, Run Command and Automation runbooks

How do you patch a fleet of EC2 instances? Mid

With Patch Manager: a patch baseline per tier that defines what is approved, patch groups via the PatchGroup tag so each tier gets the right baseline, a daily Scan association so compliance is always current, and Install with RebootIfNeeded in a maintenance window in the team's time zone (ScheduleTimezone), with max-concurrency so only part of each tier reboots at once - staging first, production a few days later. Then I check describe-instance-patch-states, including InstalledPendingReboot, which is not patched yet.

Also asked: What is the difference between a Scan and an Install patch operation? · Why would a maintenance window run at the wrong time? · How do you find every instance that still has a vulnerable package installed?

Learn it: 37.15 Patch Manager, maintenance windows and State Manager

Your EventBridge rule matches, but the SNS topic never receives anything. Why? Mid

The topic's access policy does not allow events.amazonaws.com to sns:Publish. EventBridge delivers as a service principal, so resource-policy targets - SNS, SQS, Lambda - need a statement for it, ideally with an aws:SourceArn condition for the rule. Nothing errors where you would look: the rule's FailedInvocations metric is the signal. I'd add the statement and then test with a real event, not just test-event-pattern.

Also asked: How would you automatically close SSH opened to the world on a security group? · What is an InputTransformer used for? · When would you use EventBridge Scheduler instead of a cron rule?

Learn it: 37.19 EventBridge: rules, patterns, targets and the auto-remediation loop

How would you make sure no S3 bucket in the company is ever left without versioning? Mid

An AWS Config managed rule, S3_BUCKET_VERSIONING_ENABLED, deployed to every account through a conformance pack from the delegated administrator, with an automatic remediation that runs AWS-ConfigureS3BucketVersioning through a role that can only do that. An aggregator in the security account shows compliance across the organization. And the Terraform or CloudFormation modules enable versioning too, so the remediation and the IaC never fight over the same setting.

Also asked: What is the difference between AWS Config and CloudTrail? · When would you choose manual over automatic remediation? · What does a Config aggregator do?

Learn it: 37.24 AWS Config: recording, rules, remediation and conformance packs

How do you handle database credentials for an application on AWS? Mid

Secrets Manager, encrypted with a customer managed KMS key, with automatic rotation - alternating users if the app cannot tolerate a failed login. The app gets the secret through its instance or task role at runtime, with a caching client that refreshes and re-fetches on an authentication error, so a rotation never needs a restart. The role may read that one secret ARN and decrypt with that one key. Nothing in files, user data or environment variables, and an alarm on RotationFailed.

Also asked: What are the four steps of a Secrets Manager rotation? · How does ACM DNS validation keep a certificate renewing? · When would you choose Parameter Store SecureString over Secrets Manager?

Learn it: 37.29 Secrets and certificates: Parameter Store, Secrets Manager rotation, KMS and ACM

GuardDuty fires a High finding at 3 am. What do you do? Mid

Read it: the type tells me the threat and the resource type, the action tells me from where, count and first/last seen tell me if it is ongoing. Then contain before investigating - isolate an instance with an empty security group, revoke a role's older sessions if credentials are involved - preserve evidence with an EBS snapshot, and use CloudTrail to see what the actor did. Afterwards fix the cause, archive the finding with a note in Security Hub, and write it up.

Also asked: What is the difference between GuardDuty and Inspector? · What does IAM Access Analyzer report? · What does Security Hub need to evaluate its controls?

Learn it: 37.33 Security findings: GuardDuty, Security Hub, Inspector, Access Analyzer and the advisors

Practise these answers with flashcards and labs Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.