Learn vocabulary for formulating and discussing experiment hypotheses: null hypothesis, statistical significance, p-value, Type I and Type II errors.
0 / 5 completed
1 / 5
What does 'we hypothesize that X will lead to Y' communicate in a growth experiment?
A hypothesis statement frames the experiment's purpose: a specific, testable prediction linking a change (X) to an expected measurable outcome (Y). Good hypotheses are falsifiable and specify both the direction and magnitude of the expected effect.
2 / 5
What is the 'null hypothesis' in the context of an A/B test?
The null hypothesis (H₀) assumes no effect: the change being tested has no impact on the metric. Statistical testing aims to determine whether observed data provides sufficient evidence to reject H₀ in favour of the alternative hypothesis (H₁).
3 / 5
What does 'the test reached significance at p < 0.05' mean?
A p-value below 0.05 means there is less than a 5% chance of seeing this result by random chance if the null hypothesis were true. This threshold (alpha) is a convention — lower p-values indicate stronger evidence against the null hypothesis.
4 / 5
What is a 'Type I error' in hypothesis testing?
A Type I error (false positive) occurs when you conclude the treatment had an effect, but the observed difference was actually due to chance. The significance level (alpha) is the maximum acceptable probability of making a Type I error.
5 / 5
What is a 'Type II error' in hypothesis testing?
A Type II error (false negative) occurs when a real effect exists but the experiment fails to detect it — often due to insufficient statistical power, which is related to sample size and effect size. Power analysis before running an experiment helps minimise Type II error risk.