Learn vocabulary for designing experiments: treatment vs control, randomization unit, novelty effect, minimum detectable effect, and early stopping.
0 / 12 completed
1 / 12
What is the difference between the 'treatment group' and the 'control group' in an experiment?
The treatment group (also called the variant or experiment group) receives the new feature or change being evaluated. The control group experiences the unchanged baseline. Comparing outcomes between the two isolates the effect of the change.
2 / 12
What is the 'randomization unit' in an A/B test?
The randomization unit defines what gets assigned to each group. User-level randomization is most common (each user sees one variant consistently), but experiments may randomize at the session, device, or geo level depending on the context and risk of cross-contamination.
3 / 12
What is the 'novelty effect' in experiment analysis?
The novelty effect can make a treatment look more effective than it really is if the experiment is measured too early. Running experiments for a sufficient duration (usually at least one to two weeks) helps the novelty effect dissipate and reveals the true steady-state impact.
4 / 12
What is the 'minimum detectable effect' (MDE) in experiment planning?
The MDE is defined upfront: if you expect a treatment to improve conversion by at least 2%, you design the experiment to detect a 2% lift with sufficient power. Setting a smaller MDE requires a larger sample size and a longer experiment runtime.
5 / 12
What does 'we're stopping early due to harm' mean in an experiment context?
Early stopping due to harm is a pre-specified protocol: if a critical metric — such as error rate, user complaints, or revenue — deteriorates beyond an acceptable threshold during the experiment, the team stops the test and rolls back to protect users. This is distinct from stopping early because results look positive.
6 / 12
Alice: 'I'm running an A/B test on the new checkout flow. We've seen a small increase in conversions, but I'm worried about potential bias. What's the primary concern when interpreting this initial data?', Consider the following options
The core concern with a small increase in conversions is the potential for confounding variables. Insufficient - A large sample size doesn't negate bias; it just makes it harder to detect. The key here is recognizing that other factors (marketing campaigns, seasonal trends) could be driving the change alongside the new checkout flow.
7 / 12
Alice: 'I'm running an A/B test on the new checkout flow. We've seen a small increase in conversions, but I'm worried about potential bias. What's the primary concern when interpreting this initial data?', Consider the following options
The core concern with a small increase in conversions is the potential for confounding variables. Insufficient - A large sample size doesn't negate bias; it just makes it harder to detect. The key here is recognizing that other factors (marketing campaigns, seasonal trends) could be driving the change alongside the new checkout flow.
8 / 12
David: 'I'm using the factorial design to test three variables – button color, font size, and placement. I want to ensure that each combination of these factors is tested equally. What does 'blocking' refer to in this context?
Blocking in experimental design refers to strategically grouping subjects or treatments based on known confounding variables. This helps isolate the effect of the independent variable(s) you're testing by reducing variability caused by those other factors (e.g., user demographics). It's a fundamental technique for creating robust and interpretable results.
9 / 12
Sarah (in a Slack channel): 'The preliminary results from the new search algorithm's A/B test are showing significantly higher click-through rates on version B. However, we're seeing a noticeable increase in users bouncing immediately after clicking. What is a key concern when interpreting this result?'
While an increased click-through rate is positive, a high bounce rate indicates that users aren't finding what they were looking for after clicking. This suggests the algorithm might be attracting clicks but failing to meet user expectations – a critical flaw in experimental design. Focusing solely on the CTR without considering the bounce rate could lead to incorrect conclusions.
10 / 12
Mark (in a PR description for a feature release): 'Implemented a randomized controlled experiment to evaluate the impact of a redesigned landing page on conversion rates. The control group received the standard layout, while the treatment group saw the new design. We monitored key metrics like form submissions and time spent on page.' What is the most important metric Mark should *specifically* track to determine if the experiment was successful?
Mark needs to measure the *conversion rate* – the percentage of visitors who completed the desired action (submitting a form). This directly reflects the impact of the redesign on its intended goal. Tracking other metrics like session duration provides additional context but isn't as directly tied to the experiment's primary objective.
11 / 12
Ben (during a standup meeting): 'I'm running an experiment to test whether adding a dark mode option to the app increases user engagement. I've set up two groups: one with the standard light theme, and one with the new dark theme. We're tracking daily active users.' What is Ben primarily trying to assess?
Ben's experiment is focused on assessing *user engagement*, which in this context is defined as the number of daily active users. The core goal is to determine if the new dark theme leads to a measurable increase in user activity, rather than simply whether people like the look and feel – that's a separate consideration.
12 / 12
Chloe: 'We ran an experiment to see if offering users a discount code would increase purchases. We found a small but statistically significant uplift in sales. However, we're concerned about potential confounding factors. What is the MOST relevant consideration when interpreting this result?',
A statistically significant result doesn't automatically mean the discount is *causing* the increase in sales. External factors (seasonality, marketing campaigns, competitor actions) can also contribute to a rise in purchases. Identifying and accounting for these confounding variables is crucial for drawing accurate conclusions about the impact of the discount.
What will I practise in "Experiment Design — Vocabulary"?
Learn vocabulary for designing experiments: treatment vs control, randomization unit, novelty effect, minimum detectable effect, and early stopping.
How many exercises are in this module?
This module has 12 multiple-choice exercises, each with instant feedback and a full explanation of the correct answer.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to use with no account, sign-up, or paywall.
Do I need to create an account to do these exercises?
No account is required. Just click an option to answer — your score for this session is tracked automatically in the progress bar above.
What happens if I choose the wrong answer?
You'll immediately see which answer was correct, plus a full explanation covering the vocabulary and reasoning behind it — mistakes are where most of the learning happens.
Can I retry the exercises if I want a higher score?
Yes — use the "Try again" button on the results screen to reset and go through all the questions again.
Is my progress saved if I close the page?
No. Progress is tracked only for your current visit; reloading or leaving the page resets the counter. This keeps the exercise simple and account-free.
Where can I find more Growth & Experimentation Language exercises?
Browse the full Growth & Experimentation Language hub for related drills, or check the "Next up" link below to continue with a connected topic.
How is this different from reading an article on the same topic?
Articles explain vocabulary and concepts in prose; this exercise tests and reinforces that vocabulary through active recall with immediate feedback — the two work best together.
Who writes these exercises?
Every exercise is written by the CoderSlingo team, drawing on real workplace English used in IT roles, then reviewed for accuracy and clarity.