Data Pipeline Vocabulary Exercises

1 / 5

A colleague says: "The team migrated from ETL to ELT after moving to Snowflake."

What is the key architectural difference between ETL and ELT?

ETL vs. ELT — the critical architectural distinction:

Pattern	Flow	Where transformation happens
ETL	Extract → Transform → Load	In a separate pipeline/tool before writing to warehouse
ELT	Extract → Load → Transform	Inside the data warehouse (BigQuery, Snowflake) using its native compute

ELT became practical with scalable cloud DW compute — raw data lands in a "raw zone" first, then SQL/dbt transforms it into a "refined zone". Key vocabulary: transformation layer, raw zone vs. refined zone, dbt for ELT transformations, warehouse-native compute.

2 / 5

A data engineer explains: "The data pipeline ingests events in micro-batches every 30 seconds."

How does this differ from true "real-time streaming"?

Streaming vocabulary — understanding latency tiers:

Approach	Latency	Example tools
Micro-batch	1–30 second windows	Spark Structured Streaming
True streaming	Per-event, sub-second	Apache Flink, Kafka Streams

Key vocabulary: event-driven processing, windowing, event time vs. processing time, watermark, late data handling.

3 / 5

The pipeline orchestrator shows a failed DAG run.

What is a DAG in data pipeline orchestration?

DAG = Directed Acyclic Graph. In orchestration tools (Airflow, Prefect, Dagster):

DAG concept	Meaning
Nodes	Tasks (e.g., extract, transform, load steps)
Edges	Dependencies between tasks
Acyclic	No cycles — execution flows forward only
Failed DAG run	At least one task in the pipeline failed

Key vocabulary: task dependency, upstream/downstream task, DAG run, trigger rule, SLA miss, orchestrator.

4 / 5

The data team says: "We discovered a lineage gap — we cannot trace where this field originated."

What is data lineage?

Data lineage = "where did this number come from?" It answers the question of how data has flowed and been transformed from source to report.

Lineage use case	Benefit
Impact analysis	If a source column changes, which downstream models break?
Data quality debugging	Trace anomalies back to the root cause transformation
Compliance (GDPR)	Prove data provenance and demonstrate audit trail

Tools: OpenLineage, Atlan, DataHub, dbt lineage graph. Key vocabulary: data provenance, lineage graph, impact analysis, upstream source dependency.

5 / 5

A data engineer says: "The SLA requires the reporting dataset to be fresh by 6am."

What does "data freshness" mean in a pipeline context?

Data freshness SLO = the maximum allowed lag before data is considered stale.

Concept	Definition
Freshness check	Compare `max(updated_at)` vs. current time
Freshness SLO	e.g., data must be no older than 2 hours
Staleness	Data has exceeded the freshness threshold

Tools: dbt source freshness, Great Expectations freshness check. Key vocabulary: freshness SLO, staleness, freshness alert, data availability SLA, pipeline latency.