Holm correction
A way to adjust p-values when a test compares several variants with the control, so the chance of any false winner stays at the chosen level.
Also called Holm-Bonferroni method
With several comparisons, the chance that at least one looks significant by luck grows. The Holm method sorts the p-values from smallest to largest and tests each against a stricter threshold, relaxing it step by step.
It controls the chance of any false winner as strictly as the Bonferroni correction, but finds real winners more often.
Three variants have raw p-values of 0.01, 0.03 and 0.04. Holm-adjusted, they are 0.03, 0.06 and 0.06: only the first stays significant at 5%.
Related terms
- Multiple comparisonsThe problem that the more comparisons you make in one test, the more likely one looks significant by chance.
- A/B/n testAn A/B test with more than one challenger: the control is compared with two or more variants in the same experiment.
- P-valueThe probability of seeing a difference at least as large as the observed one if the variant actually made no difference.
- False positiveCalling a winner that isn't real: the test finds a difference although the variant made none.
Where it comes up
- How to read an A/B test result · Guide