Practice English vocabulary for experiment statistical power: sample size, underpowered experiments, minimum detectable effect, power analysis, and early stopping.
0 / 15 completed
1 / 15
What does '10K users per variant for 80% power' mean?
Statistical power (80% is the conventional minimum) is the probability of detecting a real effect when it exists. The required sample size is calculated from the desired power, significance level, and the minimum detectable effect — more power or smaller effects need more users.
2 / 15
What is an 'underpowered experiment' and what is its risk?
Underpowered experiments frequently produce false negatives — concluding 'no effect' when there actually is one. Teams sometimes make decisions based on these inconclusive results, shipping or blocking changes based on unreliable data.
3 / 15
What is the 'Minimum Detectable Effect (MDE)'?
The MDE is set before the experiment. If you need to detect a 2% relative improvement in conversion, you set MDE=2% and calculate the required sample size. Setting a smaller MDE requires exponentially more users.
4 / 15
What is 'power analysis' in the context of experiment planning?
Power analysis (sample size calculation) is done before launching to ensure the experiment is neither underpowered (too few users, misses real effects) nor overlong (wastes time on an effect already detectable). It uses the trade-off between power, MDE, and sample size.
5 / 15
What is the problem with 'stopping early due to reaching statistical significance'?
'Peeking' and stopping early (without pre-registered stopping rules) violates frequentist statistical assumptions and dramatically inflates Type I error rates. Sequential testing methods (like mSPRT or Bayesian approaches) allow valid early stopping with controlled error rates.
6 / 15
Review Comment: Alex writes in the code review: 'We're aiming for 80% power to detect a 10K user dropoff with this A/B test. Sounds good?' What does Alex *really* mean regarding statistical power?
Alex is referring to power analysis. '80% power' means there's an 80% chance of detecting a meaningful effect if it actually exists. The key misunderstanding here is that power isn't about guaranteeing results; it's about increasing the chances of finding a real difference when one is present, given a specific sample size and significance level.
7 / 15
Slack Message: Sarah from Product sends this message to the team: 'Just ran the power analysis. We've got a really low MDE – only 2% change! Seems like we could have stopped early.' What is the most significant risk Sarah's statement implies?
Sarah's focus on a low MDE (Minimum Detectable Effect) is dangerous. A small MDE means we need *huge* sample sizes to achieve adequate power. If we stop early because of statistical significance, we're highly likely to miss a real effect—a false negative or Type II error. This highlights the importance of considering both the MDE and the desired power level.
8 / 15
PR Description: When designing this new feature's A/B test, we need to ensure sufficient statistical power. We've defined a MDE of 5% and targeting 90% power at a significance level of 0.05.
What does the MDE (Minimum Detectable Effect) represent in this context?
The MDE (Minimum Detectable Effect) *is* precisely the smallest difference between variants that we can reliably detect with high confidence. It's a fundamental concept in power analysis; it's not about probability or deviation from the null hypothesis. Understanding this definition is crucial for interpreting and planning experiments.
9 / 15
Standup Update: 'I finished running the A/B test on the new onboarding flow. We got a p-value of 0.03 – statistically significant! We're rolling this out to all users.' What is a key concern regarding this statement, considering the context of power analysis?
A low p-value (0.03) only indicates statistical significance at a chosen alpha level (usually 0.05). However, it doesn't necessarily mean the effect is practically meaningful or that the test had sufficient power to detect it reliably. The team needs to consider the MDE and the sample size used to determine if the results are robust.
10 / 15
Code Review Comment: 'Before stopping this experiment early, we need to conduct a power analysis. Without it, we risk concluding that there's a significant effect when there isn't – a classic Type I error.' What does this comment primarily highlight?
This comment emphasizes the crucial role of power analysis in preventing Type I errors (false positives). Insufficient power means we're more likely to incorrectly conclude there's a significant effect when there isn't. It's about balancing the risk of both types of errors—Type I and Type II.
11 / 15
Review Comment: Alex writes in the code review: 'We're aiming for 80% power to detect a 10K user dropoff with this A/B test. Sounds good?' What does Alex *really* mean regarding statistical power?
Alex is referring to power analysis. '80% power' means there's an 80% chance of detecting a meaningful effect if it actually exists. The key misunderstanding here is that power isn't about guaranteeing results; it's about increasing the chances of finding a real difference when one is present, given a specific sample size and significance level.
12 / 15
Slack Message: Sarah from Product sends this message to the team: 'Just ran the power analysis. We've got a really low MDE – only 2% change! Seems like we could have stopped early.' What is the most significant risk Sarah's statement implies?
Sarah's focus on a low MDE (Minimum Detectable Effect) is dangerous. A small MDE means we need *huge* sample sizes to achieve adequate power. If we stop early because of statistical significance, we're highly likely to miss a real effect—a false negative or Type II error. This highlights the importance of considering both the MDE and the desired power level.
13 / 15
PR Description: When designing this new feature's A/B test, we need to ensure sufficient statistical power. We've defined a MDE of 5% and targeting 90% power at a significance level of 0.05.
What does the MDE (Minimum Detectable Effect) represent in this context?
The MDE (Minimum Detectable Effect) *is* precisely the smallest difference between variants that we can reliably detect with high confidence. It's a fundamental concept in power analysis; it's not about probability or deviation from the null hypothesis. Understanding this definition is crucial for interpreting and planning experiments.
14 / 15
Standup Update: 'I finished running the A/B test on the new onboarding flow. We got a p-value of 0.03 – statistically significant! We're rolling this out to all users.' What is a key concern regarding this statement, considering the context of power analysis?
A low p-value (0.03) only indicates statistical significance at a chosen alpha level (usually 0.05). However, it doesn't necessarily mean the effect is practically meaningful or that the test had sufficient power to detect it reliably. The team needs to consider the MDE and the sample size used to determine if the results are robust.
15 / 15
Code Review Comment: 'Before stopping this experiment early, we need to conduct a power analysis. Without it, we risk concluding that there's a significant effect when there isn't – a classic Type I error.' What does this comment primarily highlight?
This comment emphasizes the crucial role of power analysis in preventing Type I errors (false positives). Insufficient power means we're more likely to incorrectly conclude there's a significant effect when there isn't. It's about balancing the risk of both types of errors—Type I and Type II.
What will I practise in "Statistical Power Vocabulary"?
Practice English vocabulary for experiment statistical power: sample size, underpowered experiments, minimum detectable effect, power analysis, and early stopping.
How many exercises are in this module?
This module has 15 multiple-choice exercises, each with instant feedback and a full explanation of the correct answer.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to use with no account, sign-up, or paywall.
Do I need to create an account to do these exercises?
No account is required. Just click an option to answer — your score for this session is tracked automatically in the progress bar above.
What happens if I choose the wrong answer?
You'll immediately see which answer was correct, plus a full explanation covering the vocabulary and reasoning behind it — mistakes are where most of the learning happens.
Can I retry the exercises if I want a higher score?
Yes — use the "Try again" button on the results screen to reset and go through all the questions again.
Is my progress saved if I close the page?
No. Progress is tracked only for your current visit; reloading or leaving the page resets the counter. This keeps the exercise simple and account-free.
Where can I find more Growth & Experimentation Language exercises?
Browse the full Growth & Experimentation Language hub for related drills, or check the "Next up" link below to continue with a connected topic.
How is this different from reading an article on the same topic?
Articles explain vocabulary and concepts in prose; this exercise tests and reinforces that vocabulary through active recall with immediate feedback — the two work best together.
Who writes these exercises?
Every exercise is written by the CoderSlingo team, drawing on real workplace English used in IT roles, then reviewed for accuracy and clarity.