English for Apache Spark Developers
Vocabulary for developers working with Apache Spark — RDDs, DataFrames, executors, shuffles, and lazy evaluation — for teams discussing distributed data processing in English.
Apache Spark distributes data processing across a cluster of machines, and most of the confusion in Spark conversations comes from mixing up the logical plan (what you asked for) with the physical execution (what actually runs, and how slowly). Getting the vocabulary right lets you diagnose “why is this job slow” instead of just restarting it and hoping.
Core Abstractions
RDD (Resilient Distributed Dataset) — Spark’s original low-level abstraction: an immutable, partitioned collection of data spread across a cluster, with a lineage graph that lets Spark recompute lost partitions after a failure.
“We dropped down to the RDD API here because the DataFrame optimizer kept flattening a transformation we needed to control manually.”
DataFrame — a higher-level, schema-aware abstraction built on top of RDDs, similar to a table, that lets Spark’s optimizer reason about columns and types instead of opaque objects.
“Switch this from RDDs to DataFrames — the Catalyst optimizer can actually push the filter down before the join once it knows the schema.”
Partition — a chunk of a dataset that lives on a single executor and is processed independently; the unit of parallelism in Spark.
“We’ve got 200 partitions but only 20 cores, so most of the parallelism is wasted waiting in a queue.”
Execution Model
Lazy evaluation — Spark builds up a chain of transformations without running anything until an action (like collect or write) triggers execution, which lets it optimize the whole plan before running it.
“Nothing actually executes until you call
.write()— everything before that is just building the logical plan.”
Shuffle — the expensive operation of redistributing data across partitions (and across the network) so that records with the same key end up on the same executor, required for joins, group-bys, and repartitions.
“That join is triggering a full shuffle — 40GB moving across the network is why this stage takes ten minutes.”
Driver vs. executor — the driver is the single process coordinating the job and holding the logical plan; executors are the worker processes that actually run tasks on partitions of data.
“The OOM is on the driver, not the executors — we’re calling
.collect()and pulling the entire dataset back into a single process.”
Stage — a set of tasks that can run without a shuffle boundary; Spark breaks a job into stages wherever a shuffle is required.
“This job has three stages — the shuffle after the group-by is what’s splitting it into stage two and stage three.”
Performance and Debugging
Skew (data skew) — an uneven distribution of data across partitions, usually caused by a key with disproportionately many records, which leaves one executor doing far more work than the rest.
“One customer ID has ten million rows and everyone else has thousands — that’s the skew that’s making this join take forever.”
Broadcast join — a join strategy where a small table is copied in full to every executor, avoiding a shuffle of the (much larger) other table.
“This lookup table is only 50MB — force a broadcast join instead of letting Spark shuffle both sides.”
Spill (to disk) — when an executor doesn’t have enough memory to hold intermediate data and writes it to disk instead, which is correct but much slower.
“The stage isn’t failing, it’s just spilling — bump the executor memory and this should come back under a minute.”
Common Mistakes
- Saying “it’s slow” without specifying whether the bottleneck is a shuffle, a skewed partition, or a driver-side collect — each needs a completely different fix.
- Calling every wide transformation a “join issue” when group-bys, distincts, and repartitions also trigger shuffles.
- Forgetting that lazy evaluation means a stack trace often points to the action (
.write(),.count()), not the actual line where the bad transformation was defined.
Practice Exercise
- Explain, in two sentences, why a job with 200 partitions can still be slow on a 20-core cluster.
- Write a short Slack message diagnosing a slow join as data skew rather than a general “the join is slow” complaint.
- Draft a code review comment recommending a broadcast join for a small lookup table.