Growth Experimentation Language

A well-formed A/B test hypothesis follows the structure: "We believe that [change] will cause [outcome] for [audience] because [rationale]." For example: "We believe that changing the checkout button from grey to green will increase conversion rate for mobile users because green is a stronger call-to-action colour." Strong hypotheses include a specific metric, a target audience, and a causal explanation. Vague hypotheses like "this will make users happier" lack measurability and are rejected in rigorous growth teams.

Statistical significance is the confidence level that an observed difference between control and variant is not due to chance. Teams say "the result is statistically significant at the 95% confidence level" or "we have not reached significance — we need more traffic." The p-value is the most common metric discussed: "a p-value below 0.05 means we reject the null hypothesis." Complementary terms include "power" (the probability of detecting a real effect) and "sample size" (the number of users needed for reliable results).

MAU (Monthly Active Users) and DAU (Daily Active Users) are core engagement metrics. Teams say "our DAU/MAU ratio is 35%, which indicates strong daily engagement" or "MAU grew 12% month-over-month." Retention is discussed as "Day-1 retention" (the percentage of users who return the day after first use) or "Week-4 retention." Phrases include "our 30-day retention dropped by 3 percentage points" or "we are targeting a 40% Day-7 retention rate for the new onboarding flow."

The control group is the existing version shown to users who are not in the experiment. The variant (or treatment) is the new version being tested. Teams say "we split traffic 50/50 between control and variant" or "we are running a multivariate test with three variants." A "holdout group" is a small percentage of users permanently excluded from experiments to measure long-term effects. "Novelty effect" describes the tendency for new features to show inflated early engagement that normalises over time.

Conversion rate is the percentage of users who complete a desired action, such as signing up or purchasing. Engineers and product managers say "the landing page has a 4.2% conversion rate" or "we improved checkout conversion by 1.8 percentage points." A conversion funnel is the sequence of steps users take toward conversion, and "funnel drop-off" describes where users abandon the process. Phrases include "we are seeing 60% drop-off at the payment step" and "our goal is to reduce funnel abandonment by 15%."

Guardrail metrics are secondary metrics monitored during an experiment to ensure the variant does not cause unintended harm. Teams say "the variant improved click-through rate but triggered a guardrail — session length dropped by 8%." Common guardrails include error rates, page load time, and support ticket volume. When a guardrail is hit, teams say "we are pausing the experiment because it is degrading a guardrail metric" or "we need to investigate the trade-off before shipping."

The novelty effect is the artificial boost in engagement that new features receive simply because they are new. Teams say "early results look strong but may be inflated due to novelty effect — we should run the experiment for at least four weeks" or "the treatment showed 20% higher engagement in week one but normalised to 5% by week three." Accounting for novelty effect is called "waiting for regression to the mean" and is cited when arguing for longer experiment runtimes.

Feature rollouts use staged vocabulary: "canary release" (exposing a feature to a small percentage first), "percentage rollout" ("we are at 5% traffic and will ramp to 100% over two weeks"), and "kill switch" (the ability to turn a feature off instantly if issues arise). Teams say "we are gating this feature behind a flag" and "we will ramp the rollout after confirming no degradation in error rates." A "soft launch" is a limited public release before full rollout.

AARRR stands for Acquisition, Activation, Retention, Referral, and Revenue — often called the "Pirate Metrics" framework. Teams say "our biggest growth lever is activation — 40% of new signups never complete onboarding" or "referral is our cheapest acquisition channel with a viral coefficient of 0.8." Each stage is discussed with its own metric: "time to first value" for Activation, "cohort retention curves" for Retention, and "average revenue per user (ARPU)" for Revenue.

Experiment velocity refers to how many experiments a team runs per unit of time. Growth teams say "we run 20 experiments per week" or "we need to increase our experimentation velocity to learn faster." Bottlenecks are discussed as "our experiment velocity is limited by engineer bandwidth to implement variants" or "the review process is slowing down experiment launches." A high-velocity experimentation culture is described as "building a culture of rapid hypothesis testing and evidence-based decision making."