Expected loss
In Bayesian testing, how much you would lose on average by choosing a variant if it turned out not to be the best.
Expected loss weighs how likely a variant is to be worse by how much worse it would be. A variant with a 90% chance to beat control but a tiny downside when it doesn't has a small expected loss.
It is useful when the cost of a wrong choice matters more than certainty, for example on low-risk copy changes.
Shipping the new header has an expected loss of 0.02 percentage points of conversion: even if it's not better, the downside is negligible.
Related terms
- Bayesian A/B testingAnalysing a test by updating a probability model with the data, so results read as probabilities such as the chance a variant beats the control.
- Chance to beat controlThe probability, given the data so far, that a variant's true rate is higher than the control's.
- Multi-armed banditA test that shifts traffic toward the better-performing variant while it runs, instead of keeping a fixed split.