Data Incident Communication Exercises

1 / 5

An alert fires: "The data pipeline SLA was breached — the reporting dataset was delayed by 4 hours."

Which stakeholders must be notified first in a data SLA breach?

SLA breach communication — who needs to know and when:

Notification priority	Audience	Why urgent
Immediate	Dependent business consumers + eng leadership	They may be acting on stale data right now
During incident	Regular updates to affected teams	Manage expectations, allow workaround
Post-resolution	All-clear notification + RCA summary	Data is safe to use again

Message template: what data is affected, issue time, estimated recovery, business impact, any available workaround. Key vocabulary: SLA breach notification, business impact assessment, interim workaround, recovery ETA, impacted stakeholders.

2 / 5

During an incident, the team announces: "We are investigating an ingestion failure affecting the events table — data from the last 6 hours may be incomplete. Please treat any reports from that window as preliminary."

What communication principle does this statement follow?

Data incident communication best practices:

Principle	Action in the example
Acknowledge early	"We are investigating" — before root cause known
Define affected scope	Specific table + 6-hour time window named
Warn consumers	"Treat reports as preliminary"

Never wait for root cause before communicating impact to consumers. Key vocabulary: data incident, impacted time window, preliminary data warning, update cadence, data incident runbook.

3 / 5

A post-incident RCA states: "A schema change caused a silent data drop — the pipeline succeeded but records were missing."

What does "silent data drop" mean in a data pipeline context?

Silent data drop — one of the hardest data incidents to detect:

Type	What happened	Detection method
Silent drop	Pipeline exit code 0, but rows are missing	Row count monitoring, reconciliation
Explicit failure	Error raised, pipeline stopped	Error alerts, monitoring dashboards

Common causes: new NOT NULL column rejected rows silently, schema mismatch filtering, API pagination bug. Key vocabulary: silent failure, data loss without error, quality check gate, expected vs. actual row count check.

4 / 5

An incident resolution message reads: "The data team will perform a backfill for the affected date range."

What does a backfill involve in data engineering?

Backfill = run pipeline for past time periods to fill gaps or correct errors:

Backfill concept	Explanation
Idempotent backfill	Re-running produces the same result (uses WRITE_TRUNCATE or UPSERT)
Backfill window	The date range being re-processed
Incremental backfill	Re-process day by day to control cost/load

Key vocabulary: backfill window, idempotent backfill, backfill schedule, incremental backfill, cost of full historical backfill.

5 / 5

A post-mortem action item reads: "Add a row count check gate that fails the pipeline if ingested rows differ from expected by more than 5%."

What type of data reliability improvement does this represent?

Data observability = monitoring data health in production:

Observability pillar	What it monitors
Freshness	Is data up-to-date?
Volume	Is row count within expected range? (this action)
Schema	Did column names/types change unexpectedly?
Distribution	Are value distributions normal?

This action adds a volume anomaly detection gate. Key vocabulary: data observability, quality gate, row count SLO, anomaly detection threshold, shift-left data quality.