Statistical Significance — Vocabulary for Product Teams
Learn to explain statistical significance, p-values, and Type I/II errors in plain English.
0 / 15 completed
1 / 15
What does p-value mean in A/B testing?
The p-value is the probability of seeing results as extreme as these if the null hypothesis (no difference) is true. A low p-value (e.g., p=0.03) suggests the result is unlikely due to chance alone.
2 / 15
What is a 'false positive' (Type I error) in A/B testing?
A false positive (Type I error) means you concluded the treatment works, but it actually doesn't — the observed result was due to chance. The significance level (e.g., 5%) is the false positive rate you accept.
3 / 15
What is a 'false negative' (Type II error) in A/B testing?
A false negative (Type II error) means a real effect existed, but the experiment failed to detect it — typically due to insufficient sample size (underpowered experiment).
4 / 15
What is 'statistical power' in A/B testing?
Statistical power is the probability of correctly rejecting the null hypothesis when a real effect exists. Typically set at 80% — meaning 20% risk of a false negative when the effect equals the MDE.
5 / 15
What does 'peeking' (or 'optional stopping') mean in A/B testing?
Peeking means checking significance repeatedly and stopping when p < 0.05 is first reached — dramatically increasing false positive rates. The solution is pre-registering the end date and using sequential testing methods.
6 / 15
Code Review Comment: 'The statistical significance for this feature is low. We need more data before rolling it out to production.' What does 'low statistical significance' likely imply in this context?
'Low statistical significance' means the observed difference between groups in an experiment isn't strong enough to be confidently attributed to the feature itself. It strongly suggests a random fluctuation or that the sample size wasn't large enough to reliably detect a real effect. The common misconception is that 'low' always means 'bad,' but it simply indicates uncertainty.
7 / 15
Slack Message: 'Hey team, we ran an A/B test on the checkout flow and saw a 2% increase in conversions with version B. However, the p-value was only 0.15. Should we deploy it immediately?' What does a p-value of 0.15 suggest?
A p-value represents the probability of observing results as extreme as, or more extreme than, those obtained in an experiment *if* there is no true effect. A p-value of 0.15 (or 15%) indicates a high chance that the observed increase is simply due to random variation – meaning it's not statistically significant and shouldn't be treated as a reliable finding without further investigation.
8 / 15
PR Description: 'Implemented changes to the user onboarding flow. A/B test results showed a statistically significant improvement in completion rates (p < 0.05).'
'p < 0.05' is a standard threshold in statistical hypothesis testing. It signifies that there's less than a 5% chance of observing the same or more extreme results if the new onboarding flow *didn't* actually improve completion rates – therefore, it's considered statistically significant and provides evidence to support the change.
9 / 15
Standup Update: 'We ran an A/B test on the new button color. The results showed a statistically significant difference in click-through rates (p = 0.03). We're planning to roll it out to all users.' What is the primary reason for considering this result 'statistically significant'?
Statistical significance hinges on the probability of observing the results if there's no real effect. A p-value of 0.03 (or 3%) indicates that there's only a 3% chance of seeing such a difference in click rates by random variation – meaning the observed change is likely genuine and not just noise.
10 / 15
API Response (Partial): `{'metric': 'conversion_rate', 'p_value': 0.28, 'sample_size': 15000}` What does a p-value of 0.28 suggest about the observed change in conversion rate?
A p-value of 0.28 means there's a 28% chance that the observed change in conversion rate (even if it exists) is simply due to random fluctuations within the sample. This is above the typical significance threshold (usually <0.05), indicating the effect isn't considered statistically significant and should be treated with caution.
11 / 15
Code Review Comment: 'The statistical significance for this feature is low. We need more data before rolling it out to production.' What does 'low statistical significance' likely imply in this context?
'Low statistical significance' means the observed difference between groups in an experiment isn't strong enough to be confidently attributed to the feature itself. It strongly suggests a random fluctuation or that the sample size wasn't large enough to reliably detect a real effect. The common misconception is that 'low' always means 'bad,' but it simply indicates uncertainty.
12 / 15
Slack Message: 'Hey team, we ran an A/B test on the checkout flow and saw a 2% increase in conversions with version B. However, the p-value was only 0.15. Should we deploy it immediately?' What does a p-value of 0.15 suggest?
A p-value represents the probability of observing results as extreme as, or more extreme than, those obtained in an experiment *if* there is no true effect. A p-value of 0.15 (or 15%) indicates a high chance that the observed increase is simply due to random variation – meaning it's not statistically significant and shouldn't be treated as a reliable finding without further investigation.
13 / 15
PR Description: 'Implemented changes to the user onboarding flow. A/B test results showed a statistically significant improvement in completion rates (p < 0.05).'
'p < 0.05' is a standard threshold in statistical hypothesis testing. It signifies that there's less than a 5% chance of observing the same or more extreme results if the new onboarding flow *didn't* actually improve completion rates – therefore, it's considered statistically significant and provides evidence to support the change.
14 / 15
Standup Update: 'We ran an A/B test on the new button color. The results showed a statistically significant difference in click-through rates (p = 0.03). We're planning to roll it out to all users.' What is the primary reason for considering this result 'statistically significant'?
Statistical significance hinges on the probability of observing the results if there's no real effect. A p-value of 0.03 (or 3%) indicates that there's only a 3% chance of seeing such a difference in click rates by random variation – meaning the observed change is likely genuine and not just noise.
15 / 15
API Response (Partial): `{'metric': 'conversion_rate', 'p_value': 0.28, 'sample_size': 15000}` What does a p-value of 0.28 suggest about the observed change in conversion rate?
A p-value of 0.28 means there's a 28% chance that the observed change in conversion rate (even if it exists) is simply due to random fluctuations within the sample. This is above the typical significance threshold (usually <0.05), indicating the effect isn't considered statistically significant and should be treated with caution.
What will I practise in "Statistical Significance — Vocabulary for Product Teams"?
Learn to explain statistical significance, p-values, and Type I/II errors in plain English.
How many exercises are in this module?
This module has 15 multiple-choice exercises, each with instant feedback and a full explanation of the correct answer.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to use with no account, sign-up, or paywall.
Do I need to create an account to do these exercises?
No account is required. Just click an option to answer — your score for this session is tracked automatically in the progress bar above.
What happens if I choose the wrong answer?
You'll immediately see which answer was correct, plus a full explanation covering the vocabulary and reasoning behind it — mistakes are where most of the learning happens.
Can I retry the exercises if I want a higher score?
Yes — use the "Try again" button on the results screen to reset and go through all the questions again.
Is my progress saved if I close the page?
No. Progress is tracked only for your current visit; reloading or leaving the page resets the counter. This keeps the exercise simple and account-free.
Where can I find more Growth & Experimentation Language exercises?
Browse the full Growth & Experimentation Language hub for related drills, or check the "Next up" link below to continue with a connected topic.
How is this different from reading an article on the same topic?
Articles explain vocabulary and concepts in prose; this exercise tests and reinforces that vocabulary through active recall with immediate feedback — the two work best together.
Who writes these exercises?
Every exercise is written by the CoderSlingo team, drawing on real workplace English used in IT roles, then reviewed for accuracy and clarity.