Experiment Results Communication — Vocabulary and Language
Learn to communicate A/B test results clearly using statistical and business language.
0 / 14 completed
1 / 14
What does 'the experiment reached significance' mean?
Reaching significance (or statistical significance) means the p-value fell below the threshold (e.g., p < 0.05), indicating the observed difference is unlikely to be due to random variation.
2 / 14
What is an 'effect size' in experiment communication?
Effect size is the magnitude of the difference — e.g., '2.3% lift in conversion rate'. A result can be statistically significant but have a very small effect size, making it practically meaningless.
3 / 14
What does 'confidence interval' communicate in experiment results?
A confidence interval (e.g., 95% CI: 1.8%–2.8%) expresses the range of plausible values for the true effect — acknowledging uncertainty rather than stating the point estimate as exact truth.
4 / 14
What does 'underpowered experiment' mean?
An underpowered experiment lacks the sample size to reliably detect the target effect size — increasing the risk of false negatives (missing a real effect) and unreliable results.
5 / 14
What is the correct way to describe a 'neutral result' in experiment communication?
A neutral result should be communicated precisely: no significant difference within the experiment's power. It does not mean the change has zero effect — it means the experiment could not detect an effect of the specified minimum size.
6 / 14
John, the data scientist, just posted this comment on your PR: 'The p-value is 0.03. This suggests a statistically significant difference between the control and treatment groups. However, the effect size is small.' What does John likely mean when he mentions a 'small effect size'?
John is highlighting a crucial distinction. A statistically significant p-value (0.03) only demonstrates a *likelihood* of a difference; it doesn't tell us how *large* that difference actually is. A small effect size indicates the observed difference could easily be due to random variation, even if the p-value suggests a real effect. This emphasizes caution when interpreting the results.
7 / 14
Sarah from the marketing team needs you to summarize the key findings of an A/B test for a new landing page design. You provide her with the following data: Mean conversion rate (Treatment): 5%, Mean conversion rate (Control): 3%. What is the most appropriate way to communicate this information concisely to Sarah?
Sarah is likely a non-technical stakeholder. While 'statistically significant' is accurate, it's jargon she won't understand. Focusing on the *magnitude* of the difference (20% higher) provides immediate context and impact. Avoid complex statistical terms unless necessary for clarity – in this case, they would obscure the key finding.
8 / 14
John, the data scientist, just posted this comment on your PR: 'The p-value is 0.03. This suggests a statistically significant difference between the control and treatment groups. However, the effect size is small.' What does John likely mean when he mentions a 'small effect size'?
John is highlighting a crucial distinction. A statistically significant p-value (0.03) only demonstrates a *likelihood* of a difference; it doesn't tell us how *large* that difference actually is. A small effect size indicates the observed difference could easily be due to random variation, even if the p-value suggests a real effect. This emphasizes caution when interpreting the results.
9 / 14
Sarah from the marketing team needs you to summarize the key findings of an A/B test for a new landing page design. You provide her with the following data: Mean conversion rate (Treatment): 5%, Mean conversion rate (Control): 3%. What is the most appropriate way to communicate this information concisely to Sarah?
Sarah is likely a non-technical stakeholder. While 'statistically significant' is accurate, it's jargon she won't understand. Focusing on the *magnitude* of the difference (20% higher) provides immediate context and impact. Avoid complex statistical terms unless necessary for clarity – in this case, they would obscure the key finding.
10 / 14
Mark: 'The experiment showed a trend towards increased user engagement with the new feature, but our p-value is only 0.1. What does this tell me?'
A p-value of 0.1 (or 10%) indicates a low probability of observing the data if there were no true effect. It doesn't prove significance; it simply suggests a trend that requires replication and larger sample sizes to confidently conclude an impact. A small p-value is crucial for demonstrating statistical significance.
The p-value of 0.02 (less than 0.05) represents statistical significance. This means that the observed difference in conversion rates between the treatment and control groups is unlikely to have occurred by chance alone. However, a full interpretation requires considering both the p-value and potential effect sizes.
12 / 14
During a standup meeting, you're asked to briefly describe the results of an A/B test. Which statement is MOST appropriate?
When communicating experimental results briefly, focus on the key takeaway. Highlighting a positive percentage change (even with caveats) is more digestible than getting bogged down in technical details like p-values. Avoid ambiguity and indicate that a statistically significant result has been observed.
13 / 14
PR Description: 'The experiment demonstrated a positive impact on user retention, with a p-value of 0.07 and an effect size of 0.3.'
Effect size (0.3) provides crucial context alongside a p-value of 0.07. A small effect size indicates that the observed difference is not practically significant – even though statistically significant, the impact might be too small to justify widespread implementation. Combining both metrics offers a more nuanced interpretation.
14 / 14
You're reviewing a colleague's PR that includes A/B test results. They state: 'The control group had a conversion rate of 3%, and the treatment group had a conversion rate of 4%. The p-value was 0.15.' What's your primary concern?
A p-value of 0.15 is above the conventional threshold for statistical significance (typically 0.05). This suggests that any observed difference in conversion rates could be due to random chance rather than a genuine effect of the treatment. The primary concern should be whether the effect size justifies further investigation, especially with a higher p-value.
What will I practise in "Experiment Results Communication — Vocabulary and Language"?
Learn to communicate A/B test results clearly using statistical and business language.
How many exercises are in this module?
This module has 14 multiple-choice exercises, each with instant feedback and a full explanation of the correct answer.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to use with no account, sign-up, or paywall.
Do I need to create an account to do these exercises?
No account is required. Just click an option to answer — your score for this session is tracked automatically in the progress bar above.
What happens if I choose the wrong answer?
You'll immediately see which answer was correct, plus a full explanation covering the vocabulary and reasoning behind it — mistakes are where most of the learning happens.
Can I retry the exercises if I want a higher score?
Yes — use the "Try again" button on the results screen to reset and go through all the questions again.
Is my progress saved if I close the page?
No. Progress is tracked only for your current visit; reloading or leaving the page resets the counter. This keeps the exercise simple and account-free.
Where can I find more Growth & Experimentation Language exercises?
Browse the full Growth & Experimentation Language hub for related drills, or check the "Next up" link below to continue with a connected topic.
How is this different from reading an article on the same topic?
Articles explain vocabulary and concepts in prose; this exercise tests and reinforces that vocabulary through active recall with immediate feedback — the two work best together.
Who writes these exercises?
Every exercise is written by the CoderSlingo team, drawing on real workplace English used in IT roles, then reviewed for accuracy and clarity.