incident - (simulator) the incident room: channel, pager, roles, updates, status page, on an incident clock

Examples

$ incident status

the incident card: severity, roles, impact now, next update due

$ incident declare --sev 2 "checkout failing for about a third of customers since 19:33"

declare: you become incident commander

$ incident page checkout "checkout failing ~31%, SEV2"

page the checkout on-call (the team rota, with a reason)

$ incident role ops @dana

one pair of hands on production

$ incident assign @dana "roll back checkout to 2.9.0"

the mitigation, as a task for the ops lead

$ incident update status --state identified --component partial "Some customers cannot pay at checkout. We have found the cause and are fixing it. Next update at 20:15 UTC."

a status page post

$ incident wait 10

let ten minutes of incident time pass

$ incident handoff @casey "Impact now: ... Next update due 16:55 UTC. Open decision: ..."

hand incident command on

$ incident timeline --md

the timeline as markdown, for the postmortem

Gotchas

• Incident time is not wall time: every action takes the minutes it would in a real incident (a status update about 4, a page 1) and incident wait N lets N minutes pass. Responders reply, acknowledge pages late or never, and ask for news when a promised update is late.
• Severity policy: incident matrix (core journey down / degraded / non-core / no impact yet, against most / a subset / a few; data loss, security or money moved wrongly is SEV1). Cadence: SEV1 and SEV2 every 30 min, SEV3 every 60.
• A good update says who is affected and how, what is being done, and when the next update is - with a zone (UTC). incident check AUDIENCE "draft" runs the reviewer's checklist without posting.

Taught in

• 31.1 Why incident command: from wildfires to the IC who does not debug
• 31.2 Severity levels: the impact matrix, and declaring early
• 31.4 Roles: the incident commander coordinates, does not debug
• 31.8 Communication: cadence, audiences and templates
• 31.15 Paging, escalation and handoffs
• 31.21 Deciding under uncertainty: mitigate first, rollback bias, OODA
• 31.23 Ending the incident, and the postmortem that changes something