A/B Test Hypothesis Writing — Vocabulary and Language
Learn to write clear A/B test hypotheses using the standard hypothesis structure.
0 / 12 completed
1 / 12
What is the standard A/B test hypothesis structure?
The standard A/B test hypothesis: 'We believe [change] will cause [metric outcome] for [user segment] because [rationale]' — all four components make the hypothesis testable and the rationale explicit.
2 / 12
What is a 'primary metric' in an A/B test?
The primary metric is the single key metric that determines experiment success. Having one primary metric prevents cherry-picking and keeps the experiment focused.
3 / 12
What is a 'guardrail metric' in A/B testing?
A guardrail metric is a critical business metric that must not degrade. If the primary metric improves but a guardrail (e.g., revenue, NPS, latency) degrades significantly, the experiment should not ship.
4 / 12
What does 'minimum detectable effect' (MDE) mean in A/B testing?
MDE is the smallest effect size you want to be able to detect reliably. A smaller MDE requires a larger sample size. Teams specify MDE based on what effect would be practically meaningful.
5 / 12
What is 'novelty effect' in A/B test analysis?
Novelty effect is when users engage more with a new feature simply because it is new — causing a temporary metric lift that fades over time. Running experiments longer helps distinguish novelty effects from real impact.
6 / 12
John from the Growth team just sent a Slack message: 'We're running an A/B test on the signup flow. Variant A has a 5% conversion rate, and Variant B is at 7%. I need to determine if this difference is statistically significant.' What's the most appropriate follow-up question for you to ask John to help refine his hypothesis?
The core of a good A/B test hypothesis isn't just about the observed difference (5% vs. 7%). It *requires* understanding the sample size – without knowing how many users are in each variant, you can't assess statistical significance. Options C and D represent potential distractions from the fundamental requirements for testing.
7 / 12
John from the Growth team just sent a Slack message: 'We're running an A/B test on the signup flow. Variant A has a 5% conversion rate, and Variant B is at 7%. I need to determine if this difference is statistically significant.' What's the most appropriate follow-up question for you to ask John to help refine his hypothesis?
The core of a good A/B test hypothesis isn't just about the observed difference (5% vs. 7%). It *requires* understanding the sample size – without knowing how many users are in each variant, you can't assess statistical significance. Options C and D represent potential distractions from the fundamental requirements for testing.
8 / 12
Code Review Comment: 'The PR description mentions an A/B test with a 20% lift in click-through rate. However, it doesn't specify the baseline conversion rate or the sample size used to determine this lift. This makes it difficult to assess the validity of the claim.' What is the most appropriate response for you to add to this comment?
The code review comment highlights a critical missing piece of information in the PR description. A robust A/B test hypothesis requires defining both the baseline and using an adequate sample size to calculate meaningful metrics. Ignoring this feedback would compromise the rigor of the testing process and risk misinterpreting results. Option A is too vague; options C & D are premature conclusions without sufficient data.
9 / 12
Slack Message: '@alice I'm seeing a 15% drop in user engagement after deploying the new onboarding flow. We ran an A/B test – Variant B (new flow) vs. Variant A (old flow). Variant A had an average session duration of 8 minutes, while Variant B is down to 6.5 minutes. Should we roll back?' What would be your *initial* response?
While a drop in session duration is concerning, jumping to a rollback without further investigation is premature. A/B testing analysis needs more than just one metric. The initial response should focus on gathering additional data to understand *why* the engagement dropped and rule out external influences. Options C & D are insufficient; option A risks making a hasty decision based on incomplete information.
10 / 12
PR Description: 'Implemented changes to the user profile page as part of an A/B test. Variant A (control) had a 3% completion rate for profile updates. Variant B (new design) achieved a 4.5% completion rate. The experiment ran for two weeks with a sample size of 10,000 users.' Which statement best summarizes the key findings of this A/B test?
This PR description provides enough information to state that Variant B showed a statistically significant improvement over Variant A. The percentage increase (4.5% vs 3%) is important context, but 'statistically significant' reflects the rigorous process of A/B testing – simply stating a larger number without statistical validation isn't accurate. Option A exaggerates the result; option D incorrectly blames the sample size.
11 / 12
API Response (from an A/B testing platform): `{"metric": "conversion_rate", "variant": "B", "value": 8.2, "sample_size": 5000, "confidence_interval": [7.8, 8.6], "p_value": 0.03}` What does the 'p-value' in this response indicate?
The p-value (0.03) represents the probability of observing a difference as large as or larger than the one seen in this A/B test *if there were actually no difference between Variant A and Variant B*. A small p-value (typically < 0.05) suggests that the observed result is unlikely to be due to random chance, providing evidence for a real effect. The other options are incorrect interpretations of this metric.
12 / 12
Stand-up Update: 'I'm running an A/B test on the call to action button color in our checkout flow. Variant A (blue) has a 2% conversion rate, and Variant B (green) is at 2.5%. We're tracking this against the overall conversion rate of 1.8%.' During the daily stand-up, which question would you ask to best understand the test's progress?
While all options are potentially relevant, understanding the sample size is *crucial* for assessing the statistical significance of the results. Knowing the number of users exposed to each variant allows you to determine if the observed difference (0.7% lift) is meaningful or simply due to random variation within that specific group. The other options provide less critical information at this stage.
What will I practise in "A/B Test Hypothesis Writing — Vocabulary and Language"?
Learn to write clear A/B test hypotheses using the standard hypothesis structure.
How many exercises are in this module?
This module has 12 multiple-choice exercises, each with instant feedback and a full explanation of the correct answer.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to use with no account, sign-up, or paywall.
Do I need to create an account to do these exercises?
No account is required. Just click an option to answer — your score for this session is tracked automatically in the progress bar above.
What happens if I choose the wrong answer?
You'll immediately see which answer was correct, plus a full explanation covering the vocabulary and reasoning behind it — mistakes are where most of the learning happens.
Can I retry the exercises if I want a higher score?
Yes — use the "Try again" button on the results screen to reset and go through all the questions again.
Is my progress saved if I close the page?
No. Progress is tracked only for your current visit; reloading or leaving the page resets the counter. This keeps the exercise simple and account-free.
Where can I find more Growth & Experimentation Language exercises?
Browse the full Growth & Experimentation Language hub for related drills, or check the "Next up" link below to continue with a connected topic.
How is this different from reading an article on the same topic?
Articles explain vocabulary and concepts in prose; this exercise tests and reinforces that vocabulary through active recall with immediate feedback — the two work best together.
Who writes these exercises?
Every exercise is written by the CoderSlingo team, drawing on real workplace English used in IT roles, then reviewed for accuracy and clarity.