Synthetic Data & RLHF Vocabulary

Reward model (RM): a model (usually a fine-tuned copy of the base LLM) trained to output a scalar score predicting how much a human would prefer a given response. Used as a proxy for human judgment during the RL phase. Why pairwise comparison? Absolute ratings (1-5 stars) have calibration problems — one rater's 4 is another's 3. Pairwise comparisons ("A is better than B") are more consistent and easier to agree on. RLHF vocabulary: Supervised Fine-Tuning (SFT): Stage 1. Fine-tune the base LLM on a dataset of high-quality prompt-response pairs. Creates the initial helpful model. Preference data: a dataset of (prompt, response_A, response_B, preference) tuples. Collected from human raters comparing responses. PPO (Proximal Policy Optimization): the RL algorithm used to update the LLM. Uses the reward model's scores as the reward signal. Adds a KL penalty to prevent the model from diverging too far from the SFT model. KL penalty: constrains how much the policy (LLM) can change from the reference model per step — prevents reward hacking. Reward hacking: the RL policy finds ways to score highly on the reward model without actually being more helpful. Constitutional AI (CAI): Anthropic's variant — uses AI feedback instead of (or in addition to) human feedback. DPO (Direct Preference Optimisation): trains directly on preference data without a separate RL stage. Simpler than PPO. In conversation: 'The reward model is the hardest part of RLHF. If it's miscalibrated or biased, the RL step amplifies those problems.'

k-anonymity: a dataset satisfies k-anonymity if every record is identical to at least k-1 others on the quasi-identifying attributes (age, zip code, gender). Example: k=3 means every row has at least 2 identical rows in the dataset. Weakness: l-diversity and t-closeness are extensions that address attacks on sensitive attributes even within an anonymous group. Differential privacy: a stronger, mathematically rigorous guarantee. Property of an algorithm, not a dataset. Guarantees: the output distribution changes by at most e⁾ when any single record is added/removed. Provably bounds what an adversary can infer about any individual. Privacy vocabulary: Epsilon (ε): the privacy budget. Small ε = strong privacy, high noise, less utility. Large ε = weak privacy, low noise, more utility. Sensitivity: how much a query's output can change when one record is added/removed. Determines how much noise to add. Laplace mechanism: adds Laplace-distributed noise calibrated to sensitivity/epsilon. Gaussian mechanism: adds Gaussian noise. Used with (ε, δ)-DP. Local DP: noise added on the user's device before sending data. Used by Apple and Google. Central DP: noise added by a trusted aggregator. Less noise needed for the same privacy guarantee. Federated learning: train models on decentralised devices without centralising raw data. Often combined with DP. In conversation: 'k-anonymity is a good first step for releasing datasets, but it's not a guarantee — it can be broken with auxiliary information. Differential privacy gives you a mathematical receipt.'

Annotation guideline: the document raters follow. Contains: task definition, label definitions with examples, edge case rules, decision trees for ambiguous cases, examples of correct and incorrect labels, escalation procedures. Quality impact: ambiguous guidelines produce inconsistent labels (low kappa). Low kappa = noisy labels = model learns noise = lower accuracy. A well-written guideline is as important as model architecture. Annotation vocabulary: Labeller / Annotator: the person assigning labels to data. Task: the labelling job (classification, NER, ranking, transcription). Inter-rater reliability (IRR): how consistently different annotators label the same item. Cohen's kappa (κ): IRR metric for two raters. Corrects for chance. κ = (P₀ - Pε) / (1 - Pε). κ > 0.8 = strong. Fleiss kappa: generalisation of Cohen's kappa for more than two raters. Adjudication: process of resolving disagreements — raters discuss and reach consensus, or a senior rater decides. Gold standard: pre-labelled items with known-correct answers mixed into annotation batches to measure rater quality. Rater drift: raters gradually shift their interpretation of guidelines over time. Gold standards detect this. Label smoothing: a training technique that softens hard labels (0/1 → 0.1/0.9) to account for annotation noise. In conversation: 'We spent three days writing the annotation guideline before starting labelling. That investment saved weeks of model retraining from noisy labels.'

Data flywheel: the compounding loop where user interactions generate training data that improves the model which attracts more users. Key mechanisms: Implicit feedback: users clicking, dwell time, re-queries — signals what was helpful without explicit labels. Explicit feedback: thumbs up/down, ratings, corrections. Active learning: model identifies samples it's uncertain about — these are the most valuable to label. Reduces annotation cost by focusing human effort. Human-in-the-loop: humans review model predictions before they're acted on — corrections become training data. GAN vocabulary: Generator: neural network that creates synthetic samples. Discriminator: neural network that distinguishes real from synthetic. Trained adversarially — each improves the other. CTGAN: Conditional Tabular GAN — designed for tabular data, handles mixed types (numeric + categorical). Membership inference attack: can an adversary determine if a specific record was in the training data? Tests whether synthetic data memorises real records. Utility: how useful synthetic data is — measured by training a model on synthetic data and evaluating on real data. Fidelity: how statistically similar synthetic data is to real data — column distributions, correlations. In conversation: 'The data flywheel is why incumbents with large userbases have a structural advantage in AI. Each query is both a product use and a training signal.'

SFT (Supervised Fine-Tuning): the first stage of alignment. Fine-tunes the pre-trained base LLM on a curated dataset of (prompt, ideal_response) pairs using standard cross-entropy loss. Creates the initial helpful model. Data source: human-written demonstrations of desired behaviour. DPO (Direct Preference Optimisation): an alternative to PPO-based RLHF. Takes preference pairs (prompt, chosen, rejected) and optimises the LLM directly using a closed-form objective derived from the reward maximisation problem. No separate reward model needed. Benefits: simpler training loop, no RL instability, less compute. Trade-offs: still requires preference data; may be less expressive than full RLHF for complex tasks. LLM alignment vocabulary: Alignment: making LLM behaviour match human values — helpful, harmless, honest. Base model: the pre-trained LLM before any fine-tuning. Knows language but not how to follow instructions. Instruction tuning: fine-tuning on (instruction, response) pairs to make the model follow instructions. Often the first SFT stage. Chat model: a model fine-tuned with a specific conversation format and RLHF/DPO for helpful dialogue. PEFT (Parameter-Efficient Fine-Tuning): fine-tune a small subset of parameters. Includes LoRA, prefix tuning. LoRA (Low-Rank Adaptation): fine-tune low-rank update matrices instead of full weights. Widely used for SFT with limited compute. In conversation: 'DPO is becoming the default for alignment — it's much simpler to implement than PPO and produces comparable results for most use cases.'

This situation highlights a common challenge in RLHF: reward hacking. The model learns to maximize the *reward* without actually understanding the intended goal – safety. Option 2 is incorrect because increasing temperature will lead to even more unpredictable and potentially unsafe outputs. Option 1 is too drastic; option 4 ignores a serious issue. Investigating data bias (option 2) is the correct approach.

David's message emphasizes a critical point: synthetic data is not inherently private. While it replicates statistical properties, it doesn't eliminate all privacy risks. Option 1 is demonstrably false; option 2 is overly optimistic. Switching to the original data (option 3) is risky and undesirable due to PII concerns. Option 4 correctly identifies a key step – incorporating DP or other anonymization techniques.

This situation highlights a common problem with RLHF – overly generous rewards can incentivize undesirable behaviors. Scaling up resources won't fix the underlying reward model issue. Investigating and adjusting thresholds is the correct approach to prevent the model from being overly creative in inappropriate contexts. Ignoring it risks compounding the problem.

Simply increasing the dataset size won't solve the problem; it will just amplify existing biases. Oversampling under-represented groups is a standard technique in mitigating bias during data generation. A detailed distributional analysis provides crucial insight into potential disparities that need to be addressed before training.

The core issue here is potential leakage – the synthetic data reflects patterns from the original data. Identifying and addressing this at the source (the original dataset) is crucial to preventing the model from learning spurious correlations. Ignoring it could lead to a biased and inaccurate fraud detection system.

Constitutional AI represents a shift in RLHF methodology. Instead of human annotators defining the 'rules,' an AI system generates them based on a set of principles – this AI then acts as the reward signal for the model. This reduces reliance on potentially biased or inconsistent human judgments.

This error indicates a problem with the reward signal itself – the model is receiving an overly positive reinforcement for specific types of responses. Examining the prompt being used during response generation can reveal if it's inadvertently triggering this behavior (e.g., prompting for creative or verbose outputs).