Learn vocabulary for designing experiments: treatment vs control, randomization unit, novelty effect, minimum detectable effect, and early stopping.
0 / 5 completed
1 / 5
What is the difference between the 'treatment group' and the 'control group' in an experiment?
The treatment group (also called the variant or experiment group) receives the new feature or change being evaluated. The control group experiences the unchanged baseline. Comparing outcomes between the two isolates the effect of the change.
2 / 5
What is the 'randomization unit' in an A/B test?
The randomization unit defines what gets assigned to each group. User-level randomization is most common (each user sees one variant consistently), but experiments may randomize at the session, device, or geo level depending on the context and risk of cross-contamination.
3 / 5
What is the 'novelty effect' in experiment analysis?
The novelty effect can make a treatment look more effective than it really is if the experiment is measured too early. Running experiments for a sufficient duration (usually at least one to two weeks) helps the novelty effect dissipate and reveals the true steady-state impact.
4 / 5
What is the 'minimum detectable effect' (MDE) in experiment planning?
The MDE is defined upfront: if you expect a treatment to improve conversion by at least 2%, you design the experiment to detect a 2% lift with sufficient power. Setting a smaller MDE requires a larger sample size and a longer experiment runtime.
5 / 5
What does 'we're stopping early due to harm' mean in an experiment context?
Early stopping due to harm is a pre-specified protocol: if a critical metric — such as error rate, user complaints, or revenue — deteriorates beyond an acceptable threshold during the experiment, the team stops the test and rolls back to protect users. This is distinct from stopping early because results look positive.