Prompt regression testing, prompt versioning, A/B prompt comparison, golden datasets, eval harnesses, PromptFoo, and Braintrust vocabulary.
Key vocabulary
Prompt regression testing — running a fixed set of test cases against a prompt after every change, to verify that new edits do not break previously working behaviour.
Prompt versioning — tracking prompt changes with version identifiers (v1, v2…) so you can reproduce results and roll back if quality degrades.
Golden dataset — a curated set of inputs with known correct outputs used as the ground truth for evaluating prompts.
Eval harness — the infrastructure (code + datasets + metrics) that runs evaluations automatically and reports results.
A/B prompt comparison — running two prompt variants on the same inputs and comparing outputs to determine which performs better.
0 / 14 completed
1 / 14
A team runs their full prompt test suite after every PR that modifies a system prompt. This practice is called:
Prompt regression testing treats prompts like code: every change is verified against a known-good test suite. This catches accidental quality regressions — for example, adding a new instruction that fixes one case but breaks another. Tools like PromptFoo and Braintrust make this easy to integrate into CI/CD pipelines.
2 / 14
What is a golden dataset in the context of prompt evaluation?
A golden dataset contains carefully curated (input, expected output) pairs. When you run your prompt against the golden set, the eval harness compares actual outputs to expected ones. Building and maintaining a golden dataset is one of the most important investments in prompt engineering — without it, you are making changes blind.
3 / 14
Your team stores prompt changes as v1.0, v1.1, v2.0 in a prompt registry. This practice is:
Prompt versioning applies software version control concepts to prompts. It allows you to: reproduce results from a specific prompt version, roll back to a previous version if a new one degrades performance, and run A/B comparisons between versions. Tools like LangSmith, PromptLayer, and Braintrust provide prompt registries with built-in versioning.
4 / 14
PromptFoo is described as an eval harness for prompts. What does this mean?
An eval harness is the automation layer for prompt evaluation. PromptFoo is an open-source CLI tool that lets you define test cases in YAML, run them against multiple models or prompt variants, and score results with built-in or custom metrics (exact match, LLM-as-judge, regex, etc.). Braintrust is a similar but hosted platform with dataset management and tracing.
5 / 14
In an A/B prompt comparison, what are you trying to determine?
An A/B prompt comparison runs prompt variant A and variant B against the same input set, then compares results using a metric (win rate, human preference, task accuracy, etc.). This is how prompt engineers make evidence-based decisions: "Prompt B outperforms Prompt A on 73% of test cases, so we will deploy B." It prevents intuition-only prompt changes from going to production.
6 / 14
During a code review of a new prompt for our chatbot assistant, Sarah comments: 'I'm seeing inconsistent responses when the user asks about 'shipping costs'. It seems like the prompt isn't handling plural nouns well. Can we add some examples to the negative test cases?' What does Sarah likely mean by 'negative test cases'?
Sarah is referring to negative test cases, which are a crucial part of prompt testing. These tests intentionally try to *break* the prompt by providing inputs designed to trigger errors or unexpected behavior. This contrasts with positive test cases, which verify the prompt works as intended under normal conditions. Understanding this distinction is key to robust prompt development.
7 / 14
You're part of a team evaluating different prompts for a summarization API. The team has built a dataset containing thousands of documents and their corresponding human-written summaries. This dataset is used to train and evaluate the prompt's performance. What is this dataset primarily used for?
A golden dataset is a carefully curated collection of data used as a standard for evaluating model performance. In this case, it's specifically designed to represent ideal output – the human-written summaries – allowing developers to quantitatively assess how well the prompt's generated summaries align with these high-quality references. It's a critical component in A/B testing and overall prompt refinement.
8 / 14
The team uses a versioning system for prompts: `prompt_v1.0`, `prompt_v1.1`, `prompt_v2.0`. What is this practice primarily designed to achieve?
This versioning system is a standard practice in software development, particularly when dealing with evolving AI models. By tracking changes as distinct versions (v1.0, v1.1, v2.0), the team can easily revert to previous iterations if a new update introduces issues or experiment with different prompt configurations without disrupting the main codebase. It's analogous to version control for code.
9 / 14
During an A/B test of two prompts designed to generate product descriptions, you're trying to determine which prompt produces the most compelling and accurate descriptions. What are you primarily trying to determine in this comparison?
An A/B test is fundamentally about comparing two versions (A and B) to determine which performs *better* based on a specific metric. In this case, you're assessing the 'compellingness and accuracy' through engagement metrics – quantifiable measures like click-through rates or conversion rates that indicate how well each prompt resonates with users and achieves its intended goal.
10 / 14
Mark from Product is asking for feedback on a new prompt designed to generate marketing copy. He says, 'The output feels too formal and lacks personality. I need it to sound more like our brand voice.' Which of the following best describes what Mark is requesting when he asks for prompt testing?
Mark is asking for a subjective evaluation of the prompt's output. Prompt testing, in this context, involves assessing whether the generated text aligns with desired qualities like brand voice and tone – something that cannot be measured by simple metrics alone. The incorrect options focus on technical aspects (code debugging, optimization) or quantifiable outcomes (fine-tuning parameters).
11 / 14
During a sprint review, the team discusses the results of testing a prompt designed to extract key information from customer support tickets. The system flagged 30% of responses as 'low confidence' due to ambiguous phrasing in the original tickets. What does this likely indicate about the prompt's performance?
A 'low confidence' score suggests the prompt struggled with the inherent ambiguity or variability in the customer support tickets. This indicates the prompt wasn't robust enough to handle diverse phrasing and requires adjustments to improve its understanding and accuracy – a core goal of prompt testing.
12 / 14
As a lead engineer, you're reviewing a PR that introduces a new prompt for summarizing long-form documents. The PR includes a detailed description of the prompt's intended behavior and a set of test cases. What is the primary purpose of this documentation alongside the code?
The documentation serves as a crucial artifact for understanding *why* the prompt was designed and what it's supposed to do. This context is invaluable during code reviews, testing, and future maintenance – ensuring consistency and facilitating collaboration.
13 / 14
Sarah, a prompt engineer, notices that the chatbot consistently generates responses about 'shipping times' when the user asks about delivery estimates. She suspects an issue with the prompt's handling of related terminology. What is Sarah most likely doing when she investigates this problem?
Sarah is employing an A/B testing approach – comparing different versions of the prompt to isolate the specific issue causing the inconsistent output. This systematic evaluation allows her to determine which changes effectively resolve the problem and provides data-driven insights for prompt improvement.
14 / 14
During a standup meeting, Alex says, 'I'm working on refining the prompt for our sentiment analysis tool. I've created three variations and am running an A/B test to see which one yields the most accurate results.' What is Alex primarily focusing on when he describes this process?
Alex is employing a data-driven approach to prompt optimization. By tracking key performance metrics (precision, recall) during an A/B test, he can objectively assess which prompt variation delivers superior results – a core principle of prompt testing and iterative improvement.
What will I practice in "Prompt Testing Vocabulary | Coders Lingo"?
This is a Prompt Engineering Language exercise set. It walks through 14 scenario-based multiple-choice questions built around real usage of prompt engineering language terminology that IT professionals encounter on the job.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to complete with no account, sign-up, or paywall.
How many questions are in this exercise?
This set contains 14 questions. Each one shows immediate feedback and a detailed explanation after you answer, so you learn the correct usage right away rather than waiting for a final score.
Do I need prior experience to complete this exercise?
No prior experience is required. Each question includes a full explanation covering the reasoning behind the correct answer, so the exercise itself teaches the prompt engineering language vocabulary as you go.
Can I retry the exercise if I get questions wrong?
Yes — use the "Try again" button on the results screen to reset your answers and go through all the questions again. There is no limit on attempts.
Is my progress saved?
Your answers and score for the current session are tracked in the browser as you go. No account or login is needed, and there is nothing to install.
What if I don't understand a term used in a question?
Read the explanation shown after you answer each question — it breaks down the correct term in plain English with a real-world example. You can also check the site Glossary for quick definitions.
How is this different from reading a blog article on the topic?
Exercises like this one are interactive drills that test and reinforce specific vocabulary through multiple-choice questions, while blog articles explain concepts in prose. Practising here after reading builds active recall, not just passive recognition.
Where can I find more Prompt Engineering Language exercises?
See the Prompt Engineering Language exercises hub for the full set of related pages, or browse all exercise categories from the main Exercises index.
Can I use this exercise to prepare for a technical interview?
Yes — prompt engineering language vocabulary comes up often in technical discussions and interviews. Pair this exercise with our dedicated Interview Preparation section for role-specific practice.