Practice vocabulary for evaluating models in production: online evaluation, A/B testing models, shadow mode, comparing model versions, and interpreting live metrics.
0 / 23 completed
1 / 23
The MLOps team uses ___ evaluation to measure model performance on live traffic.
Online evaluation measures a model's performance using real production traffic — tracking metrics like click-through rate, conversion, or user satisfaction as users actually interact with the system.
2 / 23
To compare two recommendation models the team runs an A/___ test in production.
A/B testing in ML means splitting live traffic between two model versions (A and B) and measuring a business or quality metric for each group. The version with better outcomes is selected for full rollout.
3 / 23
Before full rollout the new model runs in ___ mode: it receives live requests and makes predictions, but results are not served to users.
Shadow mode evaluation runs a new model on real production requests in parallel with the live model, without serving its responses. This lets you compare behaviour and catch issues before the new model affects users.
4 / 23
The engineering report reads: 'We're comparing ___ and ___ in production.' What are v1 and v2?
'Comparing v1 and v2 in production' means running controlled traffic experiments between the current model (v1) and a new candidate (v2) to determine whether the new version improves the target metric.
5 / 23
The weekly report says: 'The new model shows 3% better ___ in shadow mode.' What metric is being measured?
CTR (click-through rate) in shadow mode means the shadow model's predictions, if they had been served, would have resulted in 3% more clicks than the live model — a positive signal before promotion to production.
6 / 23
Sarah: 'I've reviewed the PR for the new fraud detection model. The team is saying it's performing well in shadow mode, but I'm not sure I fully understand what they mean by 'drift'. It seems like a technical term with no clear business implications.' What does Sarah likely want to discuss regarding production model evaluation?
Sarah is encountering 'drift,' which refers to changes in the statistical properties of the input data or target variable. This means the relationships between features and the outcome are no longer accurately represented by the model, leading to decreased performance. The core issue isn't just about raw numbers of fraud flags; it's about the *change* in how the model behaves over time – a critical concept for ongoing production monitoring. Choosing option A would be misinterpreting 'drift' as simply needing statistical significance.
7 / 23
git checkout production_model_v3
During a code review of the new payment processing model deployment, Liam asks David: 'So, you're saying this model is running in 'Canary' mode? What does that *really* mean for our users – are we actively routing traffic to it, or just monitoring its output?'
'Canary' mode is a crucial part of production model evaluation. It describes a phased rollout where a small percentage of users receive traffic to a new model—allowing the team to monitor performance and identify issues in a real-world setting without impacting the majority of their user base. The key here is that it's an active experiment, not just monitoring; the goal is to detect 'drift' or unexpected behavior before full deployment.
8 / 23
David: 'The model is running in shadow mode. It's passively collecting data and making predictions without impacting live users.'
During a Slack conversation about the new inventory optimization model, Maria asks Ben: 'Just to be clear, when you say it's in shadow mode, are we actively comparing its performance against the existing production model, or just observing its behavior?'
Shadow mode is primarily about observing a new model's behavior without affecting live users. The key point is that it's an *observational* phase. While comparison might eventually be needed, the initial definition doesn't inherently involve direct parallel execution or comparison metrics; it simply captures predictions and data for analysis. Options A, C, and D misinterpret shadow mode as requiring active comparison or isolation from production.
9 / 23
Rahul: 'The team's running the new recommendation engine in a 'dark launch' configuration. They're monitoring its output, but it hasn't been exposed to any real users yet.'
In a PR description for a change to the user engagement model, Chloe writes: 'We deployed the updated model to production using a staged rollout, initially focusing on a small percentage of traffic and continuously monitoring key metrics before expanding.' What does 'dark launch' most closely represent in this context?
'Dark launch' is a common term for running a new model in production with limited traffic. It's essentially a controlled release where the impact on real users is minimized and performance can be carefully monitored before wider deployment. The key characteristic is close monitoring – unlike a full deployment, there are no restrictions or automated scaling initially. Options A and D misrepresent the level of control and user exposure involved.
10 / 23
Sarah: 'I've reviewed the PR for the new fraud detection model. The team is saying it's performing well in shadow mode, but I'm not sure I fully understand what they mean by 'drift'. It seems like a technical term with no clear business implications.' What does Sarah likely want to discuss regarding production model evaluation?
Sarah is encountering 'drift,' which refers to changes in the statistical properties of the input data or target variable. This means the relationships between features and the outcome are no longer accurately represented by the model, leading to decreased performance. The core issue isn't just about raw numbers of fraud flags; it's about the *change* in how the model behaves over time – a critical concept for ongoing production monitoring. Choosing option A would be misinterpreting 'drift' as simply needing statistical significance.
11 / 23
git checkout production_model_v3
During a code review of the new payment processing model deployment, Liam asks David: 'So, you're saying this model is running in 'Canary' mode? What does that *really* mean for our users – are we actively routing traffic to it, or just monitoring its output?'
'Canary' mode is a crucial part of production model evaluation. It describes a phased rollout where a small percentage of users receive traffic to a new model—allowing the team to monitor performance and identify issues in a real-world setting without impacting the majority of their user base. The key here is that it's an active experiment, not just monitoring; the goal is to detect 'drift' or unexpected behavior before full deployment.
12 / 23
David: 'The model is running in shadow mode. It's passively collecting data and making predictions without impacting live users.'
During a Slack conversation about the new inventory optimization model, Maria asks Ben: 'Just to be clear, when you say it's in shadow mode, are we actively comparing its performance against the existing production model, or just observing its behavior?'
Shadow mode is primarily about observing a new model's behavior without affecting live users. The key point is that it's an *observational* phase. While comparison might eventually be needed, the initial definition doesn't inherently involve direct parallel execution or comparison metrics; it simply captures predictions and data for analysis. Options A, C, and D misinterpret shadow mode as requiring active comparison or isolation from production.
13 / 23
Rahul: 'The team's running the new recommendation engine in a 'dark launch' configuration. They're monitoring its output, but it hasn't been exposed to any real users yet.'
In a PR description for a change to the user engagement model, Chloe writes: 'We deployed the updated model to production using a staged rollout, initially focusing on a small percentage of traffic and continuously monitoring key metrics before expanding.' What does 'dark launch' most closely represent in this context?
'Dark launch' is a common term for running a new model in production with limited traffic. It's essentially a controlled release where the impact on real users is minimized and performance can be carefully monitored before wider deployment. The key characteristic is close monitoring – unlike a full deployment, there are no restrictions or automated scaling initially. Options A and D misrepresent the level of control and user exposure involved.
14 / 23
Sarah: 'I've reviewed the PR for the new fraud detection model. The team is saying it's performing well in shadow mode, but I'm not sure I fully understand what they mean by 'drift'. It seems like a technical term with no clear business implications.' What does Sarah likely want to discuss regarding production model evaluation?
Sarah is encountering 'drift,' which refers to changes in the statistical properties of the input data or target variable. This means the relationships between features and the outcome are no longer accurately represented by the model, leading to decreased performance. The core issue isn't just about raw numbers of fraud flags; it's about the *change* in how the model behaves over time – a critical concept for ongoing production monitoring. Choosing option A would be misinterpreting 'drift' as simply needing statistical significance.
15 / 23
git checkout production_model_v3
During a code review of the new payment processing model deployment, Liam asks David: 'So, you're saying this model is running in 'Canary' mode? What does that *really* mean for our users – are we actively routing traffic to it, or just monitoring its output?'
'Canary' mode is a crucial part of production model evaluation. It describes a phased rollout where a small percentage of users receive traffic to a new model—allowing the team to monitor performance and identify issues in a real-world setting without impacting the majority of their user base. The key here is that it's an active experiment, not just monitoring; the goal is to detect 'drift' or unexpected behavior before full deployment.
16 / 23
David: 'The model is running in shadow mode. It's passively collecting data and making predictions without impacting live users.'
During a Slack conversation about the new inventory optimization model, Maria asks Ben: 'Just to be clear, when you say it's in shadow mode, are we actively comparing its performance against the existing production model, or just observing its behavior?'
Shadow mode is primarily about observing a new model's behavior without affecting live users. The key point is that it's an *observational* phase. While comparison might eventually be needed, the initial definition doesn't inherently involve direct parallel execution or comparison metrics; it simply captures predictions and data for analysis. Options A, C, and D misinterpret shadow mode as requiring active comparison or isolation from production.
17 / 23
Rahul: 'The team's running the new recommendation engine in a 'dark launch' configuration. They're monitoring its output, but it hasn't been exposed to any real users yet.'
In a PR description for a change to the user engagement model, Chloe writes: 'We deployed the updated model to production using a staged rollout, initially focusing on a small percentage of traffic and continuously monitoring key metrics before expanding.' What does 'dark launch' most closely represent in this context?
'Dark launch' is a common term for running a new model in production with limited traffic. It's essentially a controlled release where the impact on real users is minimized and performance can be carefully monitored before wider deployment. The key characteristic is close monitoring – unlike a full deployment, there are no restrictions or automated scaling initially. Options A and D misrepresent the level of control and user exposure involved.
18 / 23
Sarah: 'I've reviewed the PR for the new fraud detection model. The team is saying it's performing well in shadow mode, but I'm not sure I fully understand what they mean by 'drift'. It seems like a technical term with no clear business implications.' What does Sarah likely want to discuss regarding production model evaluation?
Sarah is encountering 'drift,' which refers to changes in the statistical properties of the input data or target variable. This means the relationships between features and the outcome are no longer accurately represented by the model, leading to decreased performance. The core issue isn't just about raw numbers of fraud flags; it's about the *change* in how the model behaves over time – a critical concept for ongoing production monitoring. Choosing option A would be misinterpreting 'drift' as simply needing statistical significance.
19 / 23
git checkout production_model_v3
During a code review of the new payment processing model deployment, Liam asks David: 'So, you're saying this model is running in 'Canary' mode? What does that *really* mean for our users – are we actively routing traffic to it, or just monitoring its output?'
'Canary' mode is a crucial part of production model evaluation. It describes a phased rollout where a small percentage of users receive traffic to a new model—allowing the team to monitor performance and identify issues in a real-world setting without impacting the majority of their user base. The key here is that it's an active experiment, not just monitoring; the goal is to detect 'drift' or unexpected behavior before full deployment.
20 / 23
David: 'The model is running in shadow mode. It's passively collecting data and making predictions without impacting live users.'
During a Slack conversation about the new inventory optimization model, Maria asks Ben: 'Just to be clear, when you say it's in shadow mode, are we actively comparing its performance against the existing production model, or just observing its behavior?'
Shadow mode is primarily about observing a new model's behavior without affecting live users. The key point is that it's an *observational* phase. While comparison might eventually be needed, the initial definition doesn't inherently involve direct parallel execution or comparison metrics; it simply captures predictions and data for analysis. Options A, C, and D misinterpret shadow mode as requiring active comparison or isolation from production.
21 / 23
Rahul: 'The team's running the new recommendation engine in a 'dark launch' configuration. They're monitoring its output, but it hasn't been exposed to any real users yet.'
In a PR description for a change to the user engagement model, Chloe writes: 'We deployed the updated model to production using a staged rollout, initially focusing on a small percentage of traffic and continuously monitoring key metrics before expanding.' What does 'dark launch' most closely represent in this context?
'Dark launch' is a common term for running a new model in production with limited traffic. It's essentially a controlled release where the impact on real users is minimized and performance can be carefully monitored before wider deployment. The key characteristic is close monitoring – unlike a full deployment, there are no restrictions or automated scaling initially. Options A and D misrepresent the level of control and user exposure involved.
22 / 23
Maria is discussing performance monitoring for the new inventory optimization model with Ben in a Slack channel. Maria types: 'I'm seeing a significant increase in 'drift' scores since deploying the shadow launch. What does 'drift' typically refer to in this context?' Which of the following is the most accurate interpretation?
In the context of shadow launches and production model evaluation, 'drift' refers to a divergence between the model's predictions and the real-world data it's observing. This often happens when the model is deployed in an environment that differs significantly from the production environment, leading to inaccurate outputs. The other options describe different types of issues but do not represent 'drift'.
23 / 23
git checkout production_model_v3
During a standup meeting, Sarah reports: 'We've been running the fraud detection model in shadow mode. We're passively collecting transaction data and comparing it to the model's predictions, but haven't yet triggered any alerts.' What is the *main* purpose of this process?
The primary goal of running a model in shadow mode is to proactively identify issues like biases or limitations *before* they impact real users. By monitoring the model's predictions against actual transactions, you can catch unexpected behavior and correct it during the testing phase, ensuring greater accuracy before full deployment.
What will I practice in "Production Model Evaluation Vocabulary"?
This is an AI Model Evaluation Language exercise set. It walks through 23 scenario-based multiple-choice questions built around real usage of AI Model Evaluation Language terminology that IT professionals encounter on the job.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to complete with no account, sign-up, or paywall.
How many questions are in this exercise?
This set contains 23 questions. Each one shows immediate feedback and a detailed explanation after you answer, so you learn the correct usage right away rather than waiting for a final score.
Do I need prior experience to complete this exercise?
No prior experience is required. Each question includes a full explanation covering the reasoning behind the correct answer, so the exercise itself teaches the AI Model Evaluation Language vocabulary as you go.
Can I retry the exercise if I get questions wrong?
Yes — use the "Try again" button on the results screen to reset your answers and go through all the questions again. There is no limit on attempts.
Is my progress saved?
Your answers and score for the current session are tracked in the browser as you go. No account or login is needed, and there is nothing to install.
What if I don't understand a term used in a question?
Read the explanation shown after you answer each question — it breaks down the correct term in plain English with a real-world example. You can also check the site Glossary for quick definitions.
How is this different from reading a blog article on the topic?
Exercises like this one are interactive drills that test and reinforce specific vocabulary through multiple-choice questions, while blog articles explain concepts in prose. Practising here after reading builds active recall, not just passive recognition.
Where can I find more AI Model Evaluation Language exercises?
See the AI Model Evaluation Language exercises hub for the full set of related pages, or browse all exercise categories from the main Exercises index.
Can I use this exercise to prepare for a technical interview?
Yes — AI Model Evaluation Language vocabulary comes up often in technical discussions and interviews. Pair this exercise with our dedicated Interview Preparation section for role-specific practice.