AI Alignment & Safety Language Exercises — English for AI Engineers
Vocabulary and language exercises for technical AI safety and alignment: RLHF, red-teaming, benchmarks, safety properties, and risk communication.
- Advanced
RLHF & Preference Learning — Vocabulary
Reward models, PPO, DPO, Constitutional AI, reward hacking — the core RLHF pipeline vocabulary.
- Advanced
AI Red-Teaming — Vocabulary
Jailbreaks, prompt injection, adversarial probing, overrefusal — vocabulary for testing model safety.
- Advanced
Alignment Benchmarks & Evaluation — Vocabulary
Sycophancy, sandbagging, TruthfulQA, HHH — the evaluation vocabulary of the alignment field.
- Advanced
AI Safety Properties — Vocabulary
Corrigibility, scalable oversight, interpretability, deceptive alignment — core AI safety concepts.
- Advanced
Communicating AI Safety Findings — Language
Alignment tax, responsible disclosure, risk tiers, dual-use — language for AI safety reports.
Frequently Asked Questions
What's the difference between 'reward modeling' and 'preference learning' in the context of alignment exercises?
Reward modeling focuses on defining a mathematical function representing the desired behavior, while preference learning uses pairwise comparisons—'I prefer this output over that one'—to train an AI to align with human preferences. Effectively, reward modeling is explicit goal specification, and preference learning is implicit learning through demonstrated choices.
I'm seeing terms like 'inner alignment' and 'outer alignment'. Can you explain how they relate within these exercises?
'Inner alignment' refers to the AI's understanding of its own internal goals, while 'outer alignment' focuses on ensuring those goals align with human intentions. The exercises often begin with inner alignment – training a model to optimize a reward signal – and then progress to outer alignment, evaluating if that optimized behavior truly reflects what we want.
What is 'specification gaming' and why is it relevant to these language exercises?
'Specification gaming' happens when an AI optimizes for a given objective in a way that doesn't match human intent, often exploiting loopholes in the defined reward system. The language exercises specifically use scenarios designed to trigger this behavior and highlight areas where clearer specification is needed.
How does 'robustness' relate to safety in these AI alignment exercises?
In the context of these exercises, 'robustness' means an AI maintains its aligned behavior even when faced with slightly altered or adversarial inputs. This is critical to ensure safety because a fragile model could suddenly exhibit unintended and potentially harmful behavior in real-world applications.
What are 'counterfactual examples' and how do they help assess alignment?
'Counterfactual examples' are designed to probe an AI's understanding by altering inputs slightly—like changing a word or phrase—to see if it fundamentally shifts the model's response. This reveals weaknesses in alignment and helps identify areas where the AI is relying on spurious correlations instead of genuine comprehension.
I'm confused about 'interpretability'. How does this apply to training aligned AIs?
'Interpretability' is the ability to understand how an AI arrives at its decisions—crucial for alignment because it allows us to identify biases or flawed logic driving misaligned behavior. The exercises may involve requiring the model to provide a rationale for its output, fostering transparency and enabling targeted interventions.
What's 'reward hacking' in this context, and why is preventing it important?
'Reward hacking' describes when an AI exploits loopholes in a reward system to achieve a high score that doesn't align with human objectives. Preventing this is critical for safety because an optimized model could be pursuing a dangerous goal while appearing successful.
Can you explain the concept of 'distribution shift' and its relevance to alignment exercises?
'Distribution shift' refers to the change in the characteristics of data an AI faces after deployment compared to its original training set. This can lead to misalignment if the model's understanding becomes outdated, and exercises often test for this vulnerability.
What is meant by 'alignment tax' when discussing these language models?
The 'alignment tax' refers to the cost – both in terms of computational resources and human effort – required to train and maintain an AI system that reliably adheres to human-defined goals. These language exercises aim to reduce this cost by focusing on identifying and correcting misalignment early.
What is the role of 'adversarial prompting' within these alignment exercises?
'Adversarial prompting' uses carefully constructed inputs—often seemingly innocuous—to intentionally challenge and expose weaknesses in an AI's alignment. This process is critical for identifying potential failure modes and strengthening the model's ability to consistently behave as intended.