False negative
Missing a real effect: the test ends without a winner although the variant really was better.
Also called Type II error
A false negative happens when a test lacks the visitors to detect an effect that is there. Its rate is one minus the power: at 80% power, one real effect in five of the planned size is missed.
Underpowered tests are the usual cause. They also exaggerate the effects they do find, because only lucky draws cross the line.
A store runs a test for one week on a low-traffic page, sees no significant difference and drops an idea that would have lifted orders by 8%.
Related terms
- Statistical powerThe chance that a test detects an effect of a given size when that effect is really there.
- Sample sizeHow many visitors a test needs per variant to detect the minimum detectable effect with the chosen confidence and power.
- Minimum detectable effect (MDE)The smallest lift a test is planned to detect reliably; it sets how many visitors the test needs.