Incident Communication

Option B — a structured P1 opener with all required components.

P1 initial alert message structure:

[SEVERITY] INCIDENT — [TIMESTAMP UTC]
[WHAT IS BROKEN]: [specific service/endpoint]
[OBSERVED BEHAVIOR]: [exact error, status code, behavior]
Affected: [who is affected + scale]
Impact: [revenue, data, business consequence]
IC: [Incident Commander name/handle]
Bridge: [channel link or call link]
ETA for next update: [time]

Each component explained:
• Timestamp (UTC): establishes the incident timeline; always use UTC for global teams
• What is broken: specific: "Payment service / EU checkout" not "the site"
• Observed behavior: exact: "HTTP 500 for all EU checkout requests" — prevents teams from investigating different symptoms
• Affected users + scale: "~2,000 users/min" — quantified; enables triage prioritisation
• Impact: "Revenue impact: ongoing" — business consequence, not technical description
• IC: who owns the incident; single point of coordination
• Bridge: where to convene — prevents fractured communication across multiple channels
• ETA for update: sets expectations; prevents the channel from flooding with "any update?" messages

Option B — a timestamped update with progress report, current hypothesis, and a firm next update time.

Incident status update structure:

[TIMESTAMP UTC] UPDATE — [Service] [Severity]
[Investigation progress]: [what has been ruled out / what is being pursued]
[Current hypothesis]: [specific — names the component + mechanism if known]
[Next update]: [specific time — not "soon"]

Key elements:
• Timestamp on every message: the incident channel is a log; timestamps enable the postmortem timeline
• "UPDATE" prefix: clearly distinguishes updates from discussion
• Ruling out: "ruling out the 08:55 deployment" — shows progress even before finding the cause
• Current hypothesis: "connection pool exhaustion in EU-WEST-1 gateway" — gives the team a focus; others can validate or challenge
• Next update time: always a specific time, never "soon"; typically every 15–30 minutes for active P1

Why regular cadence updates matter:
• Stakeholders stop pinging ICs when updates are on a schedule
• Legal and finance teams need timestamps for impact reporting
• The postmortem timeline is built from these updates
• Even "no progress" updates signal that the incident is being actively managed

Option C — a complete all-clear with resolved time, root cause, fix description, duration, impact numbers, and a postmortem link.

All-clear message structure:

[TIMESTAMP UTC] RESOLVED — [Service] [Severity]
Services restored at: [timestamp UTC]
Root cause: [specific mechanism — what was wrong and why]
Fix: [what was changed + deployment time]
Duration: [total incident duration]
Affected [users/transactions]: [quantified]
Postmortem: [link/channel] by [deadline]

Critical components:
• "RESOLVED" prefix: explicit signal that the incident is closed
• Root cause in one sentence: "connection pool max size was 5 in EU-WEST-1 config (misconfigured during last week's parameter tuning)" — specific, causal
• Fix in one sentence: "max pool size increased to 50; config redeployed at 10:44"
• Quantified impact: "~5,800 affected transactions" — needed for customer SLA review
• Postmortem deadline: creates accountability; "by EOD Friday" not "eventually"

Why "Fixed, should be working now" is insufficient:
• No root cause means the next engineer doesn't know what to look for
• No impact number means no SLA reporting
• No postmortem link means lessons are lost
• "Should be working" expresses uncertainty; after confirmation, use "is working"

Root cause — the underlying systemic condition that enabled the incident to occur. It is never a person, and almost never a single event.

Structure of a postmortem root cause statement:

[The systemic condition] [that allowed/enabled] [the proximate cause] [to cause the incident] [without being caught by safeguards].

Good root cause examples:
• "The connection pool limit was not validated in CI/CD, allowing a misconfigured value to reach production." → Points to the missing validation, not the person who set the value
• "The alerting threshold for memory usage was set above the container limit, so the alert never fired before OOM." → Points to the monitoring gap
• "There was no canary deployment process for config changes, so the misconfiguration was applied to all instances simultaneously." → Points to the deployment gap

Common root cause anti-patterns:
• "Human error" — not a root cause; someone can always make an error; the question is why safeguards didn't catch it
• "Developer X changed the config" — this is the proximate trigger, not the systemic cause
• "The HTTP 500 error" — this is the symptom, not the cause
• "The Tuesday deployment" — this is the event; the root cause is why the deployment could reach production in a broken state

Good postmortems always ask: "What process, check, or automation was absent that allowed this to happen?"

Root cause — the primary necessary condition. Contributing factors — conditions that amplified the severity or duration without being independently sufficient.

Example distinction:

Root cause:
The connection pool limit was not validated in the infra config deployment pipeline, 
allowing a misconfigured value (max: 5) to reach production.

Contributing factors:
1. Monitoring alert threshold was set to fire at >80% pool utilization, 
   but traffic hit 100% in under 2 minutes — alert fired too late.
2. The EU-WEST-1 pod auto-scaling was disabled during last week's 
   maintenance window and not re-enabled.
3. The on-call runbook for connection pool errors had not been updated 
   since the gateway migration in Q3, delaying diagnosis by ~20 minutes.

Why the distinction matters:
• Root cause gets the primary remediation action (add config validation to pipeline)
• Contributing factors each get their own separate action items in the postmortem
• Without distinguishing them, teams may focus remediation on the wrong problem

Postmortem vocabulary:
• Root cause: "the incident was caused by…" "the primary failure was…"
• Contributing factors: "which was compounded by…" "this was made worse by…" "delayed detection because…"
• Detection gap: why it took N minutes to notice
• Resolution gap: why it took N minutes to fix after detection

Option B — a complete, specific, measurable, time-bound action item with an owner and a ticket.

Postmortem action item requirements:
• Specific: "Add a PagerDuty alert for connection pool utilization > 60% on all EU gateway pods" — not "fix monitoring"
• Owner: @devops — a named person or team; "team" is not an owner
• Due date: "2024-03-22" — specific; "ASAP" or "soon" are not acceptable
• Ticket: #4521 — tracked in the backlog; without this, action items disappear

Action item anti-patterns:
• "Fix the monitoring" — what exactly? which metric? what threshold? what alert channel?
• "Improve the deployment process" — too broad; split into specific, testable actions
• "Make sure this doesn't happen again" — not actionable; cannot be verified as done
• No owner → no accountability; assigned to "the team" → no one owns it

SMART action item template:

- [Specific action verb]: [what exactly will be done]
  - Addresses: [which contributing factor or root cause gap]
  - Owner: [@name or team]
  - Due: [specific date]
  - Ticket: [#id]

Postmortems that produce vague action items are postmortems that produce repeat incidents. The test: can someone verify this action item is complete in 30 days without asking for clarification?

Sarah's initial message is a good starting point, but incident communication needs to be precise. Adding details like specific error codes (e.g., HTTP status codes) and impacted API endpoints allows the team to quickly narrow down the problem area and begin troubleshooting. Simply stating 'the API is timing out' isn't actionable enough; the team needs concrete data to start investigating effectively. Option A highlights a key omission, while options C & D are too vague – a clear request for technical details is most effective.

Sarah's initial message is a good starting point, but incident communication needs to be precise. Adding details like specific error codes (e.g., HTTP status codes) and impacted API endpoints allows the team to quickly narrow down the problem area and begin troubleshooting. Simply stating 'the API is timing out' isn't actionable enough; the team needs concrete data to start investigating effectively. Option A highlights a key omission, while options C & D are too vague – a clear request for technical details is most effective.

Sarah's initial message is a good starting point, but incident communication needs to be precise. Adding details like specific error codes (e.g., HTTP status codes) and impacted API endpoints allows the team to quickly narrow down the problem area and begin troubleshooting. Simply stating 'the API is timing out' isn't actionable enough; the team needs concrete data to start investigating effectively. Option A highlights a key omission, while options C & D are too vague – a clear request for technical details is most effective.

Sarah's initial message is a good starting point, but incident communication needs to be precise. Adding details like specific error codes (e.g., HTTP status codes) and impacted API endpoints allows the team to quickly narrow down the problem area and begin troubleshooting. Simply stating 'the API is timing out' isn't actionable enough; the team needs concrete data to start investigating effectively. Option A highlights a key omission, while options C & D are too vague – a clear request for technical details is most effective.

Option 1 directly addresses the on-call team and clearly states the issue and action being taken. Options 2 uses a direct mention of the team, providing urgency and context. Options 3 is too technical for an initial alert, and option 4 is not informative at all.

Option 2 is the most appropriate response - providing a concise update to stakeholders outlining the situation without getting bogged down in technical details. Options 1 & 4 are reactive and potentially harmful. Option 3 needs more detail than is necessary at this stage.

Option 1 provides a clear and concise statement that the issue is resolved without unnecessary technical jargon. Options 2 & 3 are too brief, while option 4 implies ongoing monitoring which may not be required.

Option 1 is an oversimplification and fails to recognize contributing factors. Option 2 correctly identifies that multiple elements played a role in the incident. Options 3 & 4 are incorrect assumptions about the relationship between load balancers and incidents.

Option 0 accurately identifies the problem: 'Improve code coverage' is too broad. It lacks specifics and doesn't provide guidance for the team. A better version would be more targeted (e.g., 'Increase code coverage in the payment processing module to 80%').