Statistical significance
A result is significant when its p-value is below a chosen threshold, such as 0.05, meaning chance alone is an unlikely explanation.
Also called significance level, confidence level
The significance level (alpha) is the false-positive rate you accept: 5% at a 95% confidence level. A significant result means the data would be unusual if the variant made no difference.
Significant doesn't mean large or important. With enough traffic, a tiny difference can be significant; read the lift and its interval to judge whether it matters.
At a 95% confidence level, a p-value of 0.03 is significant and 0.08 is not.
Related terms
- P-valueThe probability of seeing a difference at least as large as the observed one if the variant actually made no difference.
- False positiveCalling a winner that isn't real: the test finds a difference although the variant made none.
- Confidence intervalA range of values for the true effect that is consistent with the data, at a stated confidence level such as 95%.
- Statistical powerThe chance that a test detects an effect of a given size when that effect is really there.