Practice English vocabulary for experiment statistical power: sample size, underpowered experiments, minimum detectable effect, power analysis, and early stopping.
0 / 5 completed
1 / 5
What does '10K users per variant for 80% power' mean?
Statistical power (80% is the conventional minimum) is the probability of detecting a real effect when it exists. The required sample size is calculated from the desired power, significance level, and the minimum detectable effect — more power or smaller effects need more users.
2 / 5
What is an 'underpowered experiment' and what is its risk?
Underpowered experiments frequently produce false negatives — concluding 'no effect' when there actually is one. Teams sometimes make decisions based on these inconclusive results, shipping or blocking changes based on unreliable data.
3 / 5
What is the 'Minimum Detectable Effect (MDE)'?
The MDE is set before the experiment. If you need to detect a 2% relative improvement in conversion, you set MDE=2% and calculate the required sample size. Setting a smaller MDE requires exponentially more users.
4 / 5
What is 'power analysis' in the context of experiment planning?
Power analysis (sample size calculation) is done before launching to ensure the experiment is neither underpowered (too few users, misses real effects) nor overlong (wastes time on an effect already detectable). It uses the trade-off between power, MDE, and sample size.
5 / 5
What is the problem with 'stopping early due to reaching statistical significance'?
'Peeking' and stopping early (without pre-registered stopping rules) violates frequentist statistical assumptions and dramatically inflates Type I error rates. Sequential testing methods (like mSPRT or Bayesian approaches) allow valid early stopping with controlled error rates.