Fine-tuning: starting from a pre-trained checkpoint, you train for additional steps on curated examples. The model retains broad knowledge while specializing its outputs — style, domain vocabulary, instruction-following format.
2 / 5
What is LoRA (Low-Rank Adaptation)?
LoRA: instead of updating billions of weights, LoRA injects trainable rank-decomposition matrices into attention layers. The base model is frozen; only the adapters are trained, drastically reducing GPU memory and compute.
3 / 5
What is catastrophic forgetting in fine-tuning?
Catastrophic forgetting: fine-tuning on too little diverse data or for too many epochs can overwrite general knowledge. Techniques like LoRA, regularization, or mixing task data with general data mitigate this.
4 / 5
What is RLHF (Reinforcement Learning from Human Feedback)?
RLHF: humans rank pairs of model outputs; a reward model learns their preferences; PPO then fine-tunes the LLM to maximize reward. Used by InstructGPT and ChatGPT to align outputs with human values.
5 / 5
What is the purpose of a validation set during fine-tuning?
Validation set: held-out examples not used in gradient updates. Tracking validation loss/metrics reveals overfitting early. Early stopping based on validation performance prevents wasting compute and degrading quality.