MMLU, HumanEval, GPQA, MT-Bench, BIG-bench, HellaSwag — how benchmarks work and why saturation matters.
Key vocabulary
MMLU (Massive Multitask Language Understanding) — tests knowledge across 57 academic subjects.
HumanEval — measures code generation ability via functional correctness of Python solutions.
GPQA (Graduate-Level Google-Proof Q&A) — expert-level science questions designed to resist web search.
Benchmark saturation — when top models cluster near ceiling performance, making differentiation difficult.
Teaching to the test — concern that models are trained on or fine-tuned specifically for benchmark data.
0 / 22 completed
1 / 22
A colleague says “MMLU is saturating.” What does this mean?
Benchmark saturation means top models cluster near the ceiling, making it hard to distinguish which is truly better. MMLU saw this after GPT-4 and competitors all reached the high 80s–90s range. Saturated benchmarks lose their utility as differentiators.
2 / 22
HumanEval primarily measures a model’s ability to:
HumanEval presents Python function stubs with docstrings. A model’s solution passes if it produces correct output on all hidden test cases. The key metric is pass@k — whether at least one of k generated solutions passes. It focuses on functional correctness, not style.
3 / 22
GPQA questions are described as “Google-proof.” This means:
GPQA (Graduate-Level Google-Proof Q&A) contains expert-level biology, chemistry, and physics questions where even PhD specialists struggle. “Google-proof” means web search does not easily reveal the answer — testing genuine reasoning, not retrieval.
4 / 22
A researcher raises concern about a model “teaching to the test” on BIG-bench. What is the concern?
“Teaching to the test” (also called benchmark overfitting or contamination) is the worry that training data includes benchmark questions, or that fine-tuning specifically targets benchmark performance. A model that scores high this way may not generalise to real-world tasks.
5 / 22
MT-Bench is used to evaluate models on:
MT-Bench (Multi-Turn Benchmark) evaluates chat models on two-turn conversations across eight categories including reasoning, coding, math, and writing. It uses GPT-4 as an automated judge, scoring responses 1–10. It was introduced alongside LMSYS Chatbot Arena to benchmark instruction-following chat models.
6 / 22
PR Description
Subject: Performance improvements in the recommendation engine.
Body:
"We've integrated a new AI benchmark – MMLU – to assess and refine the model's performance. Initial results show a significant lift in accuracy, particularly on the humanities questions. We're monitoring drift closely and plan to retrain if scores degrade."
This PR highlights the use of an AI benchmark – MMLU – to evaluate model performance. The key takeaway here is the need to monitor 'drift' (a common concern when retraining models) and proactively plan for retraining if accuracy declines. Ignoring this aspect would be a missed opportunity; significant lift in accuracy, especially on humanities questions, warrants careful observation and potential action. Option A suggests focusing only on technical debt which isn't directly linked to the benchmark results.
7 / 22
Sarah: "Okay team, I've run some initial tests using the MMLU benchmark on the new recommendation engine. The scores are up by about 15% across the board – that's fantastic! But I'm seeing a concerning trend with questions related to historical figures; they seem to be consistently performing worse than others."
This question tests understanding of MMLU's limitations. While the overall score increase is positive, Sarah's observation about specific question categories (historical figures) highlights a potential issue – the model isn't generalizing well across all domains assessed by the benchmark. The incorrect options either misinterpret the purpose of MMLU or suggest solutions that don't address the core problem of domain-specific performance degradation. It's important to note that benchmarks, like MMLU, are designed to highlight areas needing improvement, not necessarily provide a holistic measure of success.
8 / 22
PR Description
During code review, consider this PR description:
"Implemented MMLU benchmark integration. Initial tests show a 15% accuracy increase across the board, particularly on humanities questions. Investigating performance degradation observed with historical figures – further analysis required."
This option best reflects a realistic code review scenario. The description clearly outlines the positive outcome (accuracy increase) while acknowledging the concerning trend with historical figures—a vital point for further investigation. Options A and B are too optimistic or lacking detail, and option C is overly critical without providing constructive feedback. Option D suggests a broader scope than the provided PR description focuses on.
9 / 22
PR Description
Subject: Performance improvements in the recommendation engine.
Body:
"We've integrated a new AI benchmark – MMLU – to assess and refine the model's performance. Initial results show a significant lift in accuracy, particularly on the humanities questions. We're monitoring drift closely and plan to retrain if scores degrade."
This PR highlights the use of an AI benchmark – MMLU – to evaluate model performance. The key takeaway here is the need to monitor 'drift' (a common concern when retraining models) and proactively plan for retraining if accuracy declines. Ignoring this aspect would be a missed opportunity; significant lift in accuracy, especially on humanities questions, warrants careful observation and potential action. Option A suggests focusing only on technical debt which isn't directly linked to the benchmark results.
10 / 22
Sarah: "Okay team, I've run some initial tests using the MMLU benchmark on the new recommendation engine. The scores are up by about 15% across the board – that's fantastic! But I'm seeing a concerning trend with questions related to historical figures; they seem to be consistently performing worse than others."
This question tests understanding of MMLU's limitations. While the overall score increase is positive, Sarah's observation about specific question categories (historical figures) highlights a potential issue – the model isn't generalizing well across all domains assessed by the benchmark. The incorrect options either misinterpret the purpose of MMLU or suggest solutions that don't address the core problem of domain-specific performance degradation. It's important to note that benchmarks, like MMLU, are designed to highlight areas needing improvement, not necessarily provide a holistic measure of success.
11 / 22
PR Description
During code review, consider this PR description:
"Implemented MMLU benchmark integration. Initial tests show a 15% accuracy increase across the board, particularly on humanities questions. Investigating performance degradation observed with historical figures – further analysis required."
This option best reflects a realistic code review scenario. The description clearly outlines the positive outcome (accuracy increase) while acknowledging the concerning trend with historical figures—a vital point for further investigation. Options A and B are too optimistic or lacking detail, and option C is overly critical without providing constructive feedback. Option D suggests a broader scope than the provided PR description focuses on.
12 / 22
PR Description
Subject: Performance improvements in the recommendation engine.
Body:
"We've integrated a new AI benchmark – MMLU – to assess and refine the model's performance. Initial results show a significant lift in accuracy, particularly on the humanities questions. We're monitoring drift closely and plan to retrain if scores degrade."
This PR highlights the use of an AI benchmark – MMLU – to evaluate model performance. The key takeaway here is the need to monitor 'drift' (a common concern when retraining models) and proactively plan for retraining if accuracy declines. Ignoring this aspect would be a missed opportunity; significant lift in accuracy, especially on humanities questions, warrants careful observation and potential action. Option A suggests focusing only on technical debt which isn't directly linked to the benchmark results.
13 / 22
Sarah: "Okay team, I've run some initial tests using the MMLU benchmark on the new recommendation engine. The scores are up by about 15% across the board – that's fantastic! But I'm seeing a concerning trend with questions related to historical figures; they seem to be consistently performing worse than others."
This question tests understanding of MMLU's limitations. While the overall score increase is positive, Sarah's observation about specific question categories (historical figures) highlights a potential issue – the model isn't generalizing well across all domains assessed by the benchmark. The incorrect options either misinterpret the purpose of MMLU or suggest solutions that don't address the core problem of domain-specific performance degradation. It's important to note that benchmarks, like MMLU, are designed to highlight areas needing improvement, not necessarily provide a holistic measure of success.
14 / 22
PR Description
During code review, consider this PR description:
"Implemented MMLU benchmark integration. Initial tests show a 15% accuracy increase across the board, particularly on humanities questions. Investigating performance degradation observed with historical figures – further analysis required."
This option best reflects a realistic code review scenario. The description clearly outlines the positive outcome (accuracy increase) while acknowledging the concerning trend with historical figures—a vital point for further investigation. Options A and B are too optimistic or lacking detail, and option C is overly critical without providing constructive feedback. Option D suggests a broader scope than the provided PR description focuses on.
15 / 22
PR Description
Subject: Performance improvements in the recommendation engine.
Body:
"We've integrated a new AI benchmark – MMLU – to assess and refine the model's performance. Initial results show a significant lift in accuracy, particularly on the humanities questions. We're monitoring drift closely and plan to retrain if scores degrade."
This PR highlights the use of an AI benchmark – MMLU – to evaluate model performance. The key takeaway here is the need to monitor 'drift' (a common concern when retraining models) and proactively plan for retraining if accuracy declines. Ignoring this aspect would be a missed opportunity; significant lift in accuracy, especially on humanities questions, warrants careful observation and potential action. Option A suggests focusing only on technical debt which isn't directly linked to the benchmark results.
16 / 22
Sarah: "Okay team, I've run some initial tests using the MMLU benchmark on the new recommendation engine. The scores are up by about 15% across the board – that's fantastic! But I'm seeing a concerning trend with questions related to historical figures; they seem to be consistently performing worse than others."
This question tests understanding of MMLU's limitations. While the overall score increase is positive, Sarah's observation about specific question categories (historical figures) highlights a potential issue – the model isn't generalizing well across all domains assessed by the benchmark. The incorrect options either misinterpret the purpose of MMLU or suggest solutions that don't address the core problem of domain-specific performance degradation. It's important to note that benchmarks, like MMLU, are designed to highlight areas needing improvement, not necessarily provide a holistic measure of success.
17 / 22
PR Description
During code review, consider this PR description:
"Implemented MMLU benchmark integration. Initial tests show a 15% accuracy increase across the board, particularly on humanities questions. Investigating performance degradation observed with historical figures – further analysis required."
This option best reflects a realistic code review scenario. The description clearly outlines the positive outcome (accuracy increase) while acknowledging the concerning trend with historical figures—a vital point for further investigation. Options A and B are too optimistic or lacking detail, and option C is overly critical without providing constructive feedback. Option D suggests a broader scope than the provided PR description focuses on.
18 / 22
Sarah is investigating a drop in MMLU scores after integrating a new model. She notices the model performs significantly better on questions related to philosophy and ethics compared to more technical domains like computer science. What's the most likely explanation for this observation?
#include int main() { // ... code ...return 0;}
This scenario highlights a common issue with benchmarks - they can reflect biases present in the training data. The model likely learned to 'teach to the test' by focusing on areas where it was already strong, as reflected in MMLU's composition. Option A is also plausible; option B reflects a known characteristic of many benchmarks and option C is possible but less directly related to the observed bias.
19 / 22
You're reviewing a PR that adds MMLU benchmark integration. The developer has used a command-line tool called `mlbench_eval` to run the evaluation. The output shows an error: 'Error: No such file or directory'. What's the most likely cause?
print(os.system('mlbench_eval'))
This is a classic path-related issue. The `mlbench_eval` executable must be accessible from the current directory or added to the system's PATH environment variable for the command-line tool to run correctly. Options B and C are possible but less likely without further context; option D isn't directly related to the error message.
20 / 22
During a standup meeting, Alex reports that the MMLU benchmark results for the new recommendation engine are 'surprisingly low'. Which of the following is the MOST important initial step to take?
print('Investigate MMLU scores')
The most crucial step is to investigate *why* the scores are low. Simply retraining without understanding the root cause could exacerbate the problem. The MMLU benchmark itself might contain biases or have limitations that need addressing before any further changes are made. Options A and C are premature steps; option D is a last resort.
21 / 22
You receive the following Slack message from a teammate:
@john_doe 'The MMLU scores are dramatically higher after we tweaked the prompt engineering. But I'm seeing inconsistencies across different runs.'
What is John Doe likely referring to?
print('Prompt engineering')
John Doe is highlighting the importance of reproducibility. Random seeds are frequently used in machine learning experiments to introduce controlled variation. Without fixing the seed, differences observed in the benchmark results could be due to this inherent randomness. Options A, B and C are less likely explanations for inconsistent benchmark scores.
22 / 22
A developer is creating a PR description for integrating the MMLU benchmark. Which of the following statements best describes the primary purpose of using this benchmark?
print('Assess model performance')
The MMLU benchmark is specifically designed as a standardized tool for evaluating AI models' capabilities. It assesses a model's ability to understand and reason about complex tasks across a range of domains – primarily humanities, which are often underrepresented in more traditional benchmarks. Options A, B and C relate to other optimization or data-related processes.
What will I practice in "AI Benchmark Vocabulary | Coders Lingo"?
This is an AI Model Evaluation Language exercise set. It walks through 22 scenario-based multiple-choice questions built around real usage of AI Model Evaluation Language terminology that IT professionals encounter on the job.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to complete with no account, sign-up, or paywall.
How many questions are in this exercise?
This set contains 22 questions. Each one shows immediate feedback and a detailed explanation after you answer, so you learn the correct usage right away rather than waiting for a final score.
Do I need prior experience to complete this exercise?
No prior experience is required. Each question includes a full explanation covering the reasoning behind the correct answer, so the exercise itself teaches the AI Model Evaluation Language vocabulary as you go.
Can I retry the exercise if I get questions wrong?
Yes — use the "Try again" button on the results screen to reset your answers and go through all the questions again. There is no limit on attempts.
Is my progress saved?
Your answers and score for the current session are tracked in the browser as you go. No account or login is needed, and there is nothing to install.
What if I don't understand a term used in a question?
Read the explanation shown after you answer each question — it breaks down the correct term in plain English with a real-world example. You can also check the site Glossary for quick definitions.
How is this different from reading a blog article on the topic?
Exercises like this one are interactive drills that test and reinforce specific vocabulary through multiple-choice questions, while blog articles explain concepts in prose. Practising here after reading builds active recall, not just passive recognition.
Where can I find more AI Model Evaluation Language exercises?
See the AI Model Evaluation Language exercises hub for the full set of related pages, or browse all exercise categories from the main Exercises index.
Can I use this exercise to prepare for a technical interview?
Yes — AI Model Evaluation Language vocabulary comes up often in technical discussions and interviews. Pair this exercise with our dedicated Interview Preparation section for role-specific practice.