The short version
- A holdout is a share of shoppers who keep seeing your original store while everyone else gets a change you have already decided to ship.
- Keep one through BFCM for changes you shipped in October or November, and for anything personalized to a segment. It is the only clean way to measure them during the sale.
- 20% is a sensible default. Smaller holdouts cost less but take months to show a modest lift; larger ones measure faster but keep more shoppers on the old store.
- Never hold out the sale itself unless you mean to test it. Shoppers in a holdout should still get your Black Friday offer.
- Read it after the refunds settle, on revenue or profit per visitor, and compare the sale weeks with the weeks before.
What a holdout is
An A/B test has a control: the original version, kept so you can choose between versions. A holdout is a control you keep after the choice is made. The change goes to most shoppers, and a fixed share keeps the original so the change goes on being measured.
| A/B test control | Holdout | |
|---|---|---|
| Question | Which version should we ship? | What is the version we shipped still adding? |
| Typical share | Half the traffic, or an equal share per version | 10% to 30% of the audience |
| How long it runs | Until the planned sample is reached | Weeks or months, often through a season |
| What you do with the answer | Ship the winner | Keep the change, adjust it, or remove it |
Why keep one through BFCM
- Before and after comparisons don't work in November. Conversion, order value and traffic all rise for the sale whatever you change, so a change shipped on 2 November looks good by 30 November for reasons that have nothing to do with it. A holdout sees the same sale at the same time, so the season cancels out.
- Sale shoppers are different shoppers. More first-time visitors, more deal hunters, more gift buyers, more mobile. A change that won in October was measured on October's shoppers. The holdout tells you whether it still helps the people who arrive in November.
- Lifts often shrink. Part of an early lift can be the novelty effect: returning shoppers react to something new, then get used to it. A holdout keeps measuring after the novelty wears off.
- Personalization needs one to mean anything. Showing returning visitors a different homepage only tells you something if some returning visitors don't see it.
How big to make it
A holdout is measured like any test, and the smaller group sets the pace. A 10% holdout only collects one visitor for every nine that see the change, so it needs far longer to reach the same sample size as an even split.
- 10% holdout
- 178 days
- 20% holdout
- 89 days
- 30% holdout
- 60 days
- 50% holdout
- 36 days
Illustrative store. Computed with Benchmyrk's planning code, the same estimate the test wizard shows: 95% confidence, 80% power, and the holdout must reach the planned sample on its own.
At these numbers a 10% holdout would need about half a year to confirm a 10% lift. It can still show the direction and catch a change that clearly hurts, but its interval will stay wide through the season. A 20% holdout gets there in about three months, which covers November and December.
The cost of a holdout is the lift the held-out shoppers don't get. It is roughly the holdout share times the lift: if a change lifts orders by 5%, a 20% holdout gives up about 1% of that audience's orders while it runs. That is the price of knowing whether the 5% is real.
- 20% for most changes and personalizations you want measured through the season.
- 30% to 50% for a change you are still unsure about. At 50% you are effectively still running the test.
- 10% for a change you are confident in, where you mainly want to catch it turning harmful.
What to hold out, and what not to
- Hold out: winners shipped in October and early November, personalizations for a segment, and segment rollouts based on a breakdown you didn't plan for.
- Don't hold out the sale. Shoppers in a holdout should see the same Black Friday offer, prices and shipping as everyone else. Holding them out of the offer is a discount test, with the support questions that come with it.
- Don't hold out fixes. A corrected price, a working checkout button or a required legal notice goes to everyone.
- Keep the share fixed until you read it. Changing a holdout's size part way through mixes shoppers who saw different things at different times.
- Keep holdouts apart. Two personalizations that change the same part of the page for the same shoppers make both holdouts harder to read.
Reading a holdout after the sale
- Same rules as any test. No verdict before 7 full days, enough visitors in both groups, and an interval, not just a number. See how to read an A/B test result.
- Wait for refunds. BFCM orders come back for weeks. Read revenue per visitor after refunds before you decide.
- Look at the sale weeks on their own. Set the results to the Black Friday to Cyber Monday dates and compare with the weeks before. A lift that holds in both is worth more than one that only shows up in one.
- Expect it to be smaller than the original test. Tests that win are often the ones that got a lucky draw, and novelty fades. A smaller lift that is still clearly positive is a good result.
- Then decide: keep the change and end the holdout, keep holding out into January, or remove the change.