Vocabulary for AI Safety Engineers

AI safety engineering is one of the fastest-growing specialisations in software. It sits at the intersection of machine learning, security, ethics, and policy. Whether you are evaluating large language models for production use, red-teaming a generative AI system, or implementing guardrails for a customer-facing chatbot, this vocabulary will help you communicate precisely.

Red-Teaming

Red-teaming in the AI context means systematically attempting to find ways to make a model behave in harmful, biased, or unintended ways. The term comes from military and cybersecurity practice, where a “red team” simulates adversaries.

“Before the model release, the safety team conducted a six-week red-teaming exercise to identify failure modes we hadn’t anticipated.” “Red-teaming revealed that the model would provide detailed instructions for dangerous activities if the prompt was framed as a fictional story.”

Adversarial Prompts

Adversarial prompts are carefully crafted inputs designed to cause a model to produce undesired outputs — bypassing safety filters, revealing system prompts, or generating harmful content. Common techniques include jailbreaks, prompt injection, and role-play framing.

“We found several adversarial prompts that caused the model to ignore its content policy when the user claimed to be a medical professional.” “Prompt injection is a class of adversarial prompt where malicious instructions are hidden in content the model is asked to process — for example, in a document it is summarising.”

Hallucination

Hallucination refers to a model generating information that is factually incorrect, fabricated, or not supported by its training data or provided context — but presented with apparent confidence.

“The model hallucinated three academic citations in the research summary. The author names existed, but the papers did not.” “Hallucination rate is one of our key evaluation metrics. For a legal research tool, even a 2% hallucination rate is unacceptable.”

Guardrails

Guardrails are technical controls — either built into the model or added as a wrapper layer — that constrain what a model can output. They include content filters, topic blockers, output validators, and input classifiers.

“We implemented guardrails using a secondary classifier that checks model outputs for policy violations before returning them to the user.” “Guardrails can be applied at the input level (blocking certain prompts), the output level (filtering responses), or both.”

Alignment

Alignment refers to the challenge of ensuring that an AI system’s goals, values, and behaviours match human intentions and values — especially as systems become more capable. Misalignment occurs when an AI pursues objectives that differ from what its designers or users intended.

“The alignment problem is not just about preventing harmful outputs today — it is about ensuring that as models become more capable, they remain aligned with human values.” “Our alignment team works on evaluation frameworks that test whether the model’s behaviour matches the intended value system across a wide range of scenarios.”

Corrigibility

Corrigibility describes an AI system’s property of being correctable and controllable by humans — accepting modification, shutdown, or correction without resistance.

“A corrigible system will defer to human oversight even when it could theoretically act autonomously. Corrigibility is considered a key safety property for advanced AI systems.” “One concern with highly capable systems is that they might resist correction if they have been trained to strongly pursue a goal. Maintaining corrigibility becomes harder as capability increases.”

RLHF (Reinforcement Learning from Human Feedback)

RLHF is a training technique in which human raters evaluate model outputs, and those ratings are used to train a reward model, which is then used to fine-tune the base model using reinforcement learning. RLHF is widely used to make models more helpful, harmless, and honest.

“The base model was pre-trained on text data, then fine-tuned using RLHF to align its responses with human preferences for helpfulness and safety.” “RLHF can introduce its own alignment problems if the human raters have biases, or if the reward model overfits to superficial features of ‘good-sounding’ responses.”

Constitutional AI

Constitutional AI is a training approach developed by Anthropic in which a set of principles (a “constitution”) is used to guide the model’s self-critique and revision process. Rather than relying solely on human feedback, the model is trained to evaluate and revise its own outputs against the stated principles.

“Constitutional AI reduced the need for human labellers to rate harmful outputs directly — the model learns to identify and revise policy-violating responses using its constitution.” “The constitutional AI approach makes the model’s values more transparent and auditable than a pure RLHF approach.”

Evaluation (Evals)

Evals are structured evaluations used to measure model capability and safety properties. Safety evals test for harmful behaviour across a range of scenarios; capability evals measure task performance.

“Our eval suite includes over two thousand prompts covering harmful content, bias, privacy violations, and factual accuracy. Every model checkpoint is evaluated before promotion to production.” “Evals are not a solved problem — a model can pass a benchmark while failing on real-world edge cases that weren’t in the test set.”

Practical Phrases for AI Safety Engineers

• “The red-teaming exercise uncovered a prompt injection vulnerability in the document summarisation feature.”
• “We need to reduce hallucination rate before this product is suitable for medical or legal use cases.”
• “Our guardrails operate at two layers: input classification and output filtering.”
• “Alignment is not a one-time fix — it requires ongoing evaluation as the model is updated.”
• “The RLHF training improved helpfulness scores, but we observed a regression in corrigibility benchmarks.”
• “We’re building an eval harness that tests for adversarial prompts across ten attack categories.”

AI safety engineering vocabulary is evolving rapidly as the field matures. Mastering these terms will help you contribute to safety reviews, evaluate model releases, communicate risk to product and leadership teams, and participate in the growing professional community of AI safety practitioners.

Bridging the Gap: Addressing Nuances for Non-Native Speakers

The concepts discussed – red-teaming, adversarial prompts, hallucination, guardrails, alignment, RLHF, and constitutional AI – represent a significant shift in how we approach developing and deploying large language models. It’s fantastic that you’re building this vocabulary; however, the subtleties of professional English, particularly within technical discussions, can often be challenging for those whose first language isn’t English. Let’s focus on some common areas where misunderstandings arise and provide practical examples to help solidify your understanding.

One frequent issue is the precise meaning of “alignment,” which goes far beyond simply making a model “agree” with human preferences. In a code review, you might see a comment like: “This implementation lacks sufficient alignment; the model’s responses still exhibit tendencies towards generating overly verbose explanations.” This isn’t just about length – it’s about ensuring the reasoning behind the response aligns with the intended purpose and safety guidelines. Similarly, when drafting a Pull Request description for a new RLHF training component, stating “We improved alignment by increasing the reward signal for factual accuracy” is more impactful than simply saying “we made it more accurate.” The former immediately communicates the strategic direction of the work and its connection to broader safety goals. Another common pitfall is using overly literal translations; phrases like “hallucination” are frequently used figuratively, meaning a generated response that confidently presents false information as fact. Recognizing this requires understanding the context – is it literally a hallucination or a demonstrable falsehood?

Furthermore, when discussing adversarial prompts, remember they aren’t just “bad” inputs designed to break the model; they’re specifically crafted to reveal vulnerabilities and biases. A Slack message might read: “Team, let’s dedicate some time this week to red-teaming with deliberately complex and ambiguous queries – think of them as controlled ‘adversarial prompts’ designed to expose potential weaknesses in our guardrails.” The emphasis is on deliberate crafting and a focused exploration of failure modes. Finally, the term “constitutional AI” – representing a system guided by a set of ethical principles - often requires careful explanation to ensure everyone understands it isn’t merely about adding constraints but rather embedding values into the model’s architecture.

# Example: Using `grep` to filter for potential hallucinated content in logs
grep -i "unverified" log_files/*.txt

This simple command illustrates a practical application – actively searching for indicators of inaccurate or fabricated information, mirroring the core concern when identifying and mitigating hallucinations. Don’t hesitate to ask clarifying questions; it’s far better to seek clarification than to misinterpret a crucial term and potentially compromise safety efforts.