Confidence interval
A range of values for the true effect that is consistent with the data, at a stated confidence level such as 95%.
Also called 95% interval
A confidence interval puts a range around an estimate. At 95%, the method that produces it covers the true value 95% of the time. If an interval for the lift excludes zero, the data rule out "no effect" at that level.
Its width shows precision: wide early in a test, narrower as visitors accumulate.
A lift of +12% with a 95% interval from +3% to +21% means the change most likely helps, by somewhere between a little and a lot.
Related terms
- LiftHow much better or worse a variant performs than the control, as a percentage of the control's value.
- Credible intervalThe Bayesian counterpart of a confidence interval: a range that contains the true value with a stated probability, given the model and the data.
- P-valueThe probability of seeing a difference at least as large as the observed one if the variant actually made no difference.
- Statistical significanceA result is significant when its p-value is below a chosen threshold, such as 0.05, meaning chance alone is an unlikely explanation.
Where it comes up
- How to read an A/B test result · Guide