Experimentation Platform Vocabulary

Proper random assignment is foundational to A/B testing. The experimentation platform uses a hash function (typically on user ID + experiment ID) to deterministically but pseudo-randomly assign each user to a variant, ensuring unbiased, balanced groups.

Consistent assignment (also called sticky assignment) is achieved by hashing the user ID. The same user ID always produces the same variant, so users have a consistent experience and assignment doesn't change between sessions.

The exposure event (or assignment event) is logged when a user is actually exposed to the variant (e.g., sees the new button). Only users who were exposed are included in the analysis. This prevents the dilution effect from users who were assigned but never reached the experiment.

Many teams implement experiments through their feature flag system. The flag's targeting rules (% of users, specific segments) define variant assignment. This allows changing experiment exposure without deploying new code and enables instant kill switches.

Guardrail metrics (like page load time, error rate, or revenue per user) protect against experiments that improve one metric at the cost of another. If the treatment improves conversion but crashes error rates past a threshold, the guardrail triggers a stop.