English for PyTorch Developers
Vocabulary for developers training models with PyTorch — tensors, autograd, the training loop, and checkpoints — for teams discussing deep learning code in English.
PyTorch conversations mix two vocabularies: the math of gradients and tensors, and the plain engineering of loops, memory, and devices. Most confusion in reviews happens when someone says “it’s not learning” without specifying which of those two layers the problem is actually in.
Tensors and Devices
Tensor — PyTorch’s core data structure, a multi-dimensional array similar to a NumPy array but with GPU support and the ability to track gradients.
“Move that tensor to the GPU before the forward pass, or you’ll get a device mismatch error the moment it meets a CUDA tensor.”
Device (CPU/CUDA/MPS) — the physical location a tensor’s data lives and computation runs, which must match between a model and its inputs.
“This crash isn’t a logic bug — the model’s on CUDA and the input batch is still on CPU.”
In-place operation — an operation that modifies a tensor’s underlying memory directly (denoted by a trailing underscore, like add_) instead of returning a new tensor, which is memory-efficient but can break gradient tracking if used carelessly.
“Don’t use an in-place op on a tensor that’s part of the autograd graph — that’s exactly the kind of thing that throws a cryptic ‘variable modified by an inplace operation’ error.”
Autograd and Training
Autograd — PyTorch’s automatic differentiation engine, which records every operation on a tensor with requires_grad=True so it can compute gradients automatically during the backward pass.
“You don’t write the derivative by hand — autograd traces every operation and computes it for you when you call
.backward().”
Backward pass — the step where gradients are computed by walking the autograd graph in reverse, from the loss back to every parameter that contributed to it.
“The forward pass looks fine — it’s the backward pass that’s producing NaN gradients, which usually means an exploding value somewhere upstream.”
Optimizer step — the point where the optimizer (SGD, Adam, etc.) actually updates model parameters using the gradients that autograd just computed.
“We’re computing gradients but never calling
optimizer.step()— that’s why the loss isn’t moving at all.”
zero_grad() — clearing accumulated gradients before the next backward pass, since PyTorch accumulates gradients by default rather than overwriting them.
“Forgetting
zero_grad()is the classic bug here — gradients from the last three batches are all stacking on top of each other.”
Data and Checkpoints
DataLoader — the utility that batches, shuffles, and (optionally) parallelizes loading from a Dataset, feeding the training loop.
“Bump num_workers on the DataLoader — the GPU is sitting idle waiting on data loading, which is the actual bottleneck here.”
Checkpoint — a saved snapshot of a model’s (and often optimizer’s) state, used to resume training or to deploy a specific trained version.
“Always save the optimizer state in the checkpoint, not just the model weights — otherwise resuming training restarts momentum from zero.”
Common Mistakes
- Saying “the model isn’t learning” without checking whether gradients are even being computed, zeroed, or applied — three separate places the pipeline can silently break.
- Confusing a device mismatch error with a logic bug, when it’s usually just a tensor that was never moved to the GPU.
- Treating in-place operations as a free performance win without checking whether the tensor is part of an active autograd graph.
Practice Exercise
- Explain, in two sentences, the difference between the forward pass and the backward pass to someone new to PyTorch.
- Write a short PR comment explaining why a missing
zero_grad()call caused gradients to accumulate across batches. - Draft a debugging message diagnosing an idle GPU as a DataLoader bottleneck rather than a model problem.