Incident Log Analysis

Reading an incident timeline from logs:

Always read logs chronologically when reconstructing an incident. The first anomaly in time is typically the root cause or closest to it.

The causal chain in this incident:

1. 03:30 — Normal state: 1,240 orders/min, normal latency
2. 03:41 — First anomaly: p99 latency spikes to 2.1s. Something is slow. The query is identified in the next entry.
3. 03:43 — Connection pool at 97%. Why? The slow query (2+ seconds each) is holding connections. With 1,240 req/min, slow queries accumulate and exhaust the pool.
4. 03:44 — Query timeouts. Pool exhausted, requests queuing, some hitting 30s timeout.
5. 03:47 — Health check fails. The service can't reach the database at all — connections are completely exhausted or database is overwhelmed.

The likely root cause:
SELECT * FROM orders WHERE status='pending' — without an index on status, this is a full-table scan. As the orders table grows, or as the count of "pending" orders grows, this query gets slower. A slow query with high request volume = connection pool saturation.

Incident timeline vocabulary:
• root cause — the initiating event that triggered the failure chain
• proximate cause — the immediate cause of the failure (health check failure) vs the root cause (slow query)
• cascading failure — one component's failure causes downstream failures
• contributing factor — condition that made the failure worse (high request volume)

This log sequence reveals a classic distributed systems design problem: treating non-critical operations as critical-path dependencies.

What happened (reading the log):
1. Authentication ✅
2. Order received ✅
3. Inventory reserved ✅
4. Payment charged ✅ — money was taken from the user
5. Email notification ❌ — SMTP timeout
6. Order marked FAILED ❌ — because of step 5

The design problem:
The order processing flow is synchronous and treats all steps as equally critical. But email notification is NOT critical to the business transaction — the payment and inventory steps are. A failed email should not roll back or fail a completed payment.

The correct design:
• Mark the order as COMPLETED after payment and inventory steps
• Send notifications asynchronously (via a message queue)
• If notification fails, retry it — but it should never fail the order
• Consider: what does the user see? A paid order showing as "FAILED" is a billing dispute waiting to happen

Key vocabulary for this scenario:
• critical path — operations that must succeed for the transaction to complete
• non-critical / fire-and-forget — operations that should not block the happy path (notifications, analytics, audit logs)
• saga pattern — a way to manage distributed transactions with compensating actions
• compensating transaction — undoing a previous step (e.g., refund if payment succeeds but fulfilment fails)

Reading the evidence:

• evicted: 14,823 out of cache_size_before: 15,000 — 98.8% of all sessions were removed
• cache_size_after: 177 — only 177 sessions remained
• Immediately after: thousands of "session not found" → mass logouts
• Recurring every ~6 hours — NOT random, it is periodic

The "thundering herd" TTL problem:
If all sessions are created with the same TTL (e.g., 6-hour expiry) and a large number of users logged in around the same time (e.g., Monday morning login rush), they all expire at the same time. This causes a bulk eviction event that logs out everyone simultaneously.

Solutions to this problem:
1. TTL jitter — add random variation to TTL: instead of exactly 6 hours, use 5.5–6.5 hours. Sessions expire spread out over time.
2. Sliding TTL — reset TTL on each request. Active users never expire.
3. Persistent sessions — store sessions in a database, not just cache

Vocabulary for this scenario:
• cache eviction — removing entries from cache (scheduled, capacity-based, or TTL-based)
• TTL (Time To Live) — how long a cache entry remains valid
• thundering herd — many clients simultaneously triggering the same event (expiry, retry, reconnect)
• TTL jitter — randomising expiry times to avoid synchronised eviction
• sliding TTL / rolling TTL — TTL resets on each access

A post-mortem timeline must be:

1. In UTC — never local times; participants in different time zones need a common reference
2. System-oriented, not person-oriented — what the system did, not what engineers did (the actions section handles engineer responses)
3. Specific with data — exact values from logs (p99=2.1s), not vague descriptions ("things got slow")
4. Chronological — ordered by time, not by discovery order
5. Blameless — the timeline is factual, not judgmental

Why option A is wrong (person-focused):
"Alex noticed…" — this puts a person's name and actions in the timeline. In a blameless post-mortem, the timeline traces system events. Engineer actions belong in a separate "Response" section. Person-focused timelines create blame, even when unintentional.

Why option C is wrong (blame language):
"someone didn't add a database index" — this is blame, not fact. The correct framing: "the orders.status column lacked an index, resulting in full-table scans under load."

Why option D is wrong (imprecise):
"~03:40" and vague descriptions ("pretty bad") are not defensible. Post-mortem timelines must be reconstructable from logs, not from memory.

Post-mortem timeline vocabulary:
"[TIME] UTC — [system component]: [event with specific data]"
"[TIME] UTC — Incident declared / acknowledged / mitigated / resolved"
"Duration of impact: X minutes (from T to T+X)"

Incident status update anatomy:

A good incident update during a live incident includes:

1. Timestamp in UTC — when this update was written
2. What is affected — specific service, specific user-facing behaviour
3. Impact scope — which users/flows are affected
4. Evidence gathered so far — what the logs show
5. Current hypothesis — likely cause (marked as hypothesis, not fact)
6. Actions being taken — what responders are doing right now
7. Next update time — external timeline expectation management

Why option B is best:
• Uses UTC
• Specific: "order-service", "checkout flows"
• Evidence-based: "logs indicate upstream timeout"
• Hypothesis qualified: "is a likely contributing factor and is being investigated" — not stated as fact
• Action stated: "rollback being evaluated"
• Next update promised: "14:45 UTC"

Why option C is dangerous:
"The deployment caused the outage" — stated as fact. If rollback doesn't fix it and the real cause is something else, this update misled stakeholders. Use "likely contributing factor" until confirmed.

Incident communication vocabulary:
"We are investigating elevated error rates affecting [service/feature]."
"Logs indicate [evidence]. The [deployment/change] is a likely factor under investigation."
"Mitigation in progress: [action]. ETA: [time or 'unknown']."
"We will provide the next update by [time] UTC."

This log sequence demonstrates a direct dependency on the database. The WARN message indicates a cache miss, followed by an ERROR of connection refusal – this strongly suggests that the `db-primary` server was unavailable at the time the profile service attempted to access it. Options A and C are less likely given the specific log details; option B is possible but doesn't fully explain the sequence.

This exchange highlights a crucial link between code changes and system behavior. User B's observation about increased database connection pool latency directly correlates with the subsequent increase in 503 errors reported by User A. This illustrates how deployments can introduce new bottlenecks or exacerbate existing problems. Option A is possible, but the conversation focuses on the immediate impact of the deployment.

This log entry clearly shows a rate limit being triggered by Client A. The `ERROR` message explicitly states that the request was blocked due to exceeding the defined rate limit. This indicates an external application (Client A) is causing the problem, not an issue within the catalog service itself. Option B correctly identifies *why* the block occurred but doesn't explain the root cause.

This log sequence demonstrates a progressive issue. The initial `WARN` messages indicate slow response times from different users, escalating into a full `ERROR` when service-x was unable to process requests. This sequential build-up of latency is the key timeline element. Options B, C and D misrepresent the order or nature of events.

The most effective initial update provides specific details about the identified problem and its impact. Option 2 clearly states the failing order ID and the nature of the database issue, allowing for focused troubleshooting. Options A is too vague, B doesn't provide enough detail, and C suggests a broader outage than has been confirmed.