Holdout
A share of visitors kept on the original experience, so the effect of a personalization or a programme of changes can be measured.
Also called holdout group
A holdout keeps some visitors away from a change that everyone else gets. Comparing the two groups measures what the change is worth, long after its own test has ended.
Holdouts are common for personalization, where there is no randomized test by default, and for measuring the combined effect of many shipped changes.
A store shows a returning-customer welcome to its returning visitors but keeps 10% of them on the standard homepage, and compares orders.
Related terms
- PersonalizationShowing different visitors different versions of a page based on who they are or what they do, instead of the same page for everyone.
- AudienceA group of visitors defined by rules, such as device, country, traffic source or behaviour, that a test or personalization targets.
- A/B testAn experiment that shows two versions of a page to randomly split visitors and compares which one performs better on a chosen goal.