Multi-armed bandit
A test that shifts traffic toward the better-performing variant while it runs, instead of keeping a fixed split.
Also called auto-allocation, bandit test
A bandit test updates the traffic split as evidence comes in, sending more visitors to the variant most likely to be best. Fewer visitors see a weak variant, which suits short campaigns where earning matters more than measuring.
The trade-off is a less precise estimate of how big the difference is, and standard p-values don't apply to a moving split.
During a one-week sale, a store tests three banners. After a day, the strongest gets most of the traffic while the others keep a minimum share.
Related terms
- Traffic allocationHow much of the eligible traffic enters a test, and how it is divided between the variants.
- Expected lossIn Bayesian testing, how much you would lose on average by choosing a variant if it turned out not to be the best.
- A/B/n testAn A/B test with more than one challenger: the control is compared with two or more variants in the same experiment.