Practice English vocabulary for experimentation platforms: randomization, assignment consistency, exposure events, and feature flag configuration.
0 / 14 completed
1 / 14
What does 'the exp platform handles randomization' mean?
Proper random assignment is foundational to A/B testing. The experimentation platform uses a hash function (typically on user ID + experiment ID) to deterministically but pseudo-randomly assign each user to a variant, ensuring unbiased, balanced groups.
2 / 14
What does 'the assignment is consistent (same user always gets same variant)' mean?
Consistent assignment (also called sticky assignment) is achieved by hashing the user ID. The same user ID always produces the same variant, so users have a consistent experience and assignment doesn't change between sessions.
3 / 14
What is 'the exposure event' that 'fires when the user sees the change'?
The exposure event (or assignment event) is logged when a user is actually exposed to the variant (e.g., sees the new button). Only users who were exposed are included in the analysis. This prevents the dilution effect from users who were assigned but never reached the experiment.
4 / 14
What does 'the experiment configuration is in the feature flags' mean?
Many teams implement experiments through their feature flag system. The flag's targeting rules (% of users, specific segments) define variant assignment. This allows changing experiment exposure without deploying new code and enables instant kill switches.
5 / 14
What is a 'guardrail metric' in experimentation platform vocabulary?
Guardrail metrics (like page load time, error rate, or revenue per user) protect against experiments that improve one metric at the cost of another. If the treatment improves conversion but crashes error rates past a threshold, the guardrail triggers a stop.
6 / 14
Alice from the Growth team just commented on a PR draft: 'The experiment is running with a high variance. Should we investigate?' What does 'high variance' typically indicate within an experimentation platform context?
'High variance' in an experimentation platform almost invariably points to issues with randomization. It means the variants aren't being assigned to users evenly, leading to skewed results and making it difficult to confidently determine if any observed differences are truly due to the experiment itself. This often stems from a poorly configured or insufficient number of variants.
7 / 14
Ben sent this Slack message: 'I'm seeing a sudden drop in conversion rates on the new checkout flow experiment. The dashboard shows a large standard deviation for the treatment group.' What is Ben most likely concerned about?
Ben's concern centers around 'standard deviation,' a key metric in experimentation. A large standard deviation indicates high variability within the treatment group – meaning the results aren't clustered tightly around the mean. This can be caused by many factors (like uneven user distribution) and makes it hard to draw reliable conclusions about whether the experiment is truly impacting conversion rates.
8 / 14
Alice from the Growth team just commented on a PR draft: 'The experiment is running with a high variance. Should we investigate?' What does 'high variance' typically indicate within an experimentation platform context?
'High variance' in an experimentation platform almost invariably points to issues with randomization. It means the variants aren't being assigned to users evenly, leading to skewed results and making it difficult to confidently determine if any observed differences are truly due to the experiment itself. This often stems from a poorly configured or insufficient number of variants.
9 / 14
Ben sent this Slack message: 'I'm seeing a sudden drop in conversion rates on the new checkout flow experiment. The dashboard shows a large standard deviation for the treatment group.' What is Ben most likely concerned about?
Ben's concern centers around 'standard deviation,' a key metric in experimentation. A large standard deviation indicates high variability within the treatment group – meaning the results aren't clustered tightly around the mean. This can be caused by many factors (like uneven user distribution) and makes it hard to draw reliable conclusions about whether the experiment is truly impacting conversion rates.
10 / 14
David from Engineering is reviewing a PR that sets up an A/B test for a new button color. He comments: 'The experiment uses a 20% sample size. Is this sufficient to detect a meaningful difference?' What does David likely mean by 'sufficient' in the context of this experiment?
David is questioning whether the sample size (20%) is large enough to accurately detect a real change in behavior. A larger sample size increases statistical power—the ability to find a statistically significant difference when one actually exists. A smaller sample size might lead to false negatives (failing to detect a true effect).
11 / 14
Sarah from Product sent this message in Slack: 'The experiment metrics for the new onboarding flow are showing a significant drop in user activation. The control group has a standard deviation of 15%, while the treatment group is at 28%. What should I do?' What does Sarah's observation primarily suggest about the experiment?
Sarah's comment highlights a large difference in standard deviations (15% vs. 28%) between the control and treatment groups. A high standard deviation indicates that the results are widely dispersed around the mean, meaning there's significant variability in user behavior within each group – suggesting a potential problem with the new onboarding flow.
12 / 14
During a standup meeting, Mark from Data Science says: 'We're seeing a high degree of non-stationarity in the experiment data. The baseline conversion rates are shifting significantly over time.' What does 'non-stationarity' likely mean within the context of an experimentation platform?
'Non-stationarity' refers to a change in the underlying distribution of the data over time. If the baseline conversion rates shift significantly between the control and treatment groups, it becomes difficult to attribute any observed differences solely to the experiment itself – potentially due to external factors influencing user behavior.
13 / 14
You're reviewing a PR description for an experiment focused on a new website headline. The description states: 'This experiment utilizes a power calculation to ensure sufficient statistical significance with a 95% confidence level and a Type I error rate of 0.05.' What is the primary purpose of these values?
The power calculation and specified confidence level (95%) and Type I error rate (0.05) are fundamental concepts in experimental design. A 95% confidence level means there's a 95% probability of detecting a true effect if one exists. A Type I error (alpha = 0.05) sets the threshold for rejecting the null hypothesis – minimizing the risk of concluding that an effect exists when it doesn't.
14 / 14
Elena from Research is discussing a new experiment with the team. She says: 'We're using a variation ratio of 0.8 for this experiment. What does that mean?' What is Elena referring to?
A variation ratio of 0.8 means that 80% of users will be assigned to the treatment group (the variant) and 20% will be in the control group. This is a common approach when you want to test a relatively small change against a larger baseline population, minimizing disruption to the majority of users.
What will I practise in "Experimentation Platform Vocabulary"?
Practice English vocabulary for experimentation platforms: randomization, assignment consistency, exposure events, and feature flag configuration.
How many exercises are in this module?
This module has 14 multiple-choice exercises, each with instant feedback and a full explanation of the correct answer.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to use with no account, sign-up, or paywall.
Do I need to create an account to do these exercises?
No account is required. Just click an option to answer — your score for this session is tracked automatically in the progress bar above.
What happens if I choose the wrong answer?
You'll immediately see which answer was correct, plus a full explanation covering the vocabulary and reasoning behind it — mistakes are where most of the learning happens.
Can I retry the exercises if I want a higher score?
Yes — use the "Try again" button on the results screen to reset and go through all the questions again.
Is my progress saved if I close the page?
No. Progress is tracked only for your current visit; reloading or leaving the page resets the counter. This keeps the exercise simple and account-free.
Where can I find more Growth & Experimentation Language exercises?
Browse the full Growth & Experimentation Language hub for related drills, or check the "Next up" link below to continue with a connected topic.
How is this different from reading an article on the same topic?
Articles explain vocabulary and concepts in prose; this exercise tests and reinforces that vocabulary through active recall with immediate feedback — the two work best together.
Who writes these exercises?
Every exercise is written by the CoderSlingo team, drawing on real workplace English used in IT roles, then reviewed for accuracy and clarity.