Bayesian A/B testing
Analysing a test by updating a probability model with the data, so results read as probabilities such as the chance a variant beats the control.
Bayesian analysis starts with a prior belief about each variant's rate and updates it with the observed data into a posterior distribution. From the posteriors you can read direct statements: the probability the variant is better, or the expected loss of choosing it.
It doesn't remove the need for enough data or for a decision rule; it changes how the evidence is expressed.
After two weeks, the posterior says the new product page has a 97% chance of a higher order rate than the current one.
Related terms
- Chance to beat controlThe probability, given the data so far, that a variant's true rate is higher than the control's.
- Credible intervalThe Bayesian counterpart of a confidence interval: a range that contains the true value with a stated probability, given the model and the data.
- Expected lossIn Bayesian testing, how much you would lose on average by choosing a variant if it turned out not to be the best.
- Frequentist A/B testingAnalysing a test with p-values and confidence intervals, which describe how surprising the data would be if the variant made no difference.