Statistical power
The chance that a test detects an effect of a given size when that effect is really there.
Power is the probability of a significant result when the true effect equals the minimum detectable effect. 80% is the usual target; 90% needs about a third more visitors.
Low power means real improvements often end as "no difference", and the wins that do appear tend to be overstated.
A test planned at 80% power for a 10% lift finds a real 10% lift four times out of five.
Related terms
- Sample sizeHow many visitors a test needs per variant to detect the minimum detectable effect with the chosen confidence and power.
- Minimum detectable effect (MDE)The smallest lift a test is planned to detect reliably; it sets how many visitors the test needs.
- False negativeMissing a real effect: the test ends without a winner although the variant really was better.