A/B/n test
An A/B test with more than one challenger: the control is compared with two or more variants in the same experiment.
An A/B/n test splits traffic between the control and several variants at once, which is faster than running the same ideas one after another.
The cost is traffic and statistics: each variant gets a smaller share, and every extra comparison with the control is another chance for a false winner, so the p-values need a correction for multiple comparisons.
A store tests three free-shipping messages against no message, splitting product-page traffic four ways at 25% each.
Related terms
- A/B testAn experiment that shows two versions of a page to randomly split visitors and compares which one performs better on a chosen goal.
- Multiple comparisonsThe problem that the more comparisons you make in one test, the more likely one looks significant by chance.
- Holm correctionA way to adjust p-values when a test compares several variants with the control, so the chance of any false winner stays at the chosen level.
- Sample sizeHow many visitors a test needs per variant to detect the minimum detectable effect with the chosen confidence and power.