🚨 Incident Response

Core incident vocabulary: incident (unplanned service disruption), degradation (partial performance reduction), outage (complete service unavailability), SEV1/P0 (most critical severity), detection time (when incident was first noticed), resolution time (when service was restored), MTTR (Mean Time to Recovery), incident commander (person leading the response), war room / bridge call (emergency synchronous communication), runbook (step-by-step response guide), rollback (revert to previous stable state).

Incident communication patterns: "I'm declaring an incident — SEV1 — payment service is down", "I'm taking incident commander role", "Current status: we're investigating the root cause — detection was at 14:32 UTC", "Confirmed: the deployment at 14:28 is correlated with the outage — rolling back now", "Update: rollback complete, monitoring for recovery", "Resolved: service restored at 15:02 UTC, total impact 30 minutes". Keep updates frequent (every 10-15 min for SEV1), factual, and concise.

A blameless postmortem analyses what went wrong without assigning personal blame — focusing on system and process failures rather than individual errors. Pioneered by Google/SRE culture. Blameless language: "The monitoring alert threshold was set too high, allowing the issue to go undetected for 20 minutes" (not "Alex missed the alert"). Blameless postmortems create psychological safety to report incidents honestly and learn from them, rather than concealing errors to avoid punishment.

Severity levels define response urgency: SEV1/P0 (critical — complete outage, all-hands response, immediate action), SEV2/P1 (major — significant degradation, urgent response needed), SEV3/P2 (moderate — partial impact, response during business hours), SEV4/P3 (minor — minimal impact, planned fix in next sprint). Each level has defined response SLAs (e.g., "P0: respond within 5 minutes, update every 15 minutes"). Using correct severity vocabulary sets appropriate expectations for all stakeholders.

Status page writing: (1) Be factual, not technical — "Payment processing is currently unavailable" not "TCP connection pool exhausted"; (2) Use present continuous for ongoing — "We are investigating"; (3) Use past tense for resolved — "The issue has been resolved"; (4) Include affected services and estimated impact; (5) Commit to update frequency — "We will post an update in 30 minutes"; (6) Avoid speculation — "We are investigating" not "We think it might be X". Customers value transparency and frequency over technical detail.

A runbook is a step-by-step guide for responding to a specific incident type: "If alert X fires: (1) Check Y dashboard; (2) Run command Z; (3) If output shows A, do B; (4) If unresolved, escalate to [team]." During an incident, follow the runbook — don't improvise unless the runbook is clearly wrong. Runbooks are updated after incidents when new failure modes are discovered. Good runbooks reduce MTTR dramatically because responders don't have to think from scratch under pressure.

Escalation communication: "I've been investigating this for 20 minutes — I need to escalate to [team lead]", "I've exhausted the runbook steps — the issue persists — escalating to [service owner]", "This is beyond my area — looping in [expert] who owns [component]". Include in escalation: current status, what you've already tried, what you need from the escalation target, and the urgency level. Never escalate with "something's broken" — escalate with specific, actionable context.

In ITIL: Incident = any unplanned interruption or quality reduction (immediate focus: restore service). Problem = underlying cause of incidents (focus: root cause analysis, permanent fix). An incident is reactive (service is down now); a problem is proactive (what prevents this from happening again?). In practice: during the incident, you restore service (incident management); the post-incident review creates a problem record to track the permanent fix. Many organisations use these terms loosely, but understanding the distinction is useful in formal IT service environments.

PIR vocabulary: timeline (chronological sequence of events), detection gap (time between incident start and detection), contributing factors (conditions that made the incident possible), corrective actions (fixes to prevent recurrence), mitigations (temporary measures while permanent fix is implemented), action items (specific tasks with owners and deadlines), regression (previously fixed issue returning), human error (usually means system design failure in blameless cultures). Write in neutral, factual, past-tense language.

Customer incident notification: (1) Notify early even without full details — "We are investigating reports of [issue]"; (2) Use plain language — no technical jargon; (3) State impact clearly — "Users may be unable to [action]"; (4) Give a timeline — "We expect to resolve this by [time]" or "We will update in 30 minutes"; (5) Apologise without over-apologising — "We apologise for this disruption and are working urgently to restore service"; (6) Final resolution notice with RCA summary. Transparency builds trust even when things go wrong.