The short version
- A p-value is the chance of a difference at least this large if the variant truly did nothing. Small means surprising, not "probably true".
- A chance to beat control is the probability the variant's true rate is higher than the control's, given the data and a starting assumption (the prior).
- With a flat prior and normal traffic, the two carry much the same information: the chance to beat control is close to 1 minus half the two-sided p-value.
- They disagree near the line, because the usual thresholds (p below 0.05, chance above 95%) are not the same bar.
- Checking every day and stopping on a good number inflates false winners under both. The fix is a stopping rule built for it, such as an always-valid p-value.
- Benchmyrk shows both and calls a winner only when both agree.
What question does each approach answer?
Frequentist testing answers "if the variant made no difference, how often would I see a gap this big?"; Bayesian testing answers "given what I saw, how likely is the variant to be better, and by how much?". The first is about the data under an assumption, the second about the variant itself.
| Frequentist | Bayesian | |
|---|---|---|
| Headline number | p-value | Chance to beat control |
| Range of likely lifts | Confidence interval | Credible interval |
| Starting assumption | The variant does nothing (the null hypothesis) | A prior: what rates were plausible before the test |
| What it promises | Looked at once, at a fixed sample size, false winners at most 5% of the time | A probability statement that is correct if the model and prior are right |
| What it does not say | How likely the variant is to be better | How often the rule produces false winners |
| Typical threshold | p below 0.05 | Chance to beat control above 95% |
Neither is more honest than the other. The p-value makes a promise about a procedure; the chance to beat control makes a statement about this one test. The Bayesian number is easier to explain to a team, which is why many tools lead with it, but it inherits its meaning from the model and the prior.
What does a p-value mean, exactly?
A p-value is the probability of a result at least as extreme as the one observed, assuming the variant had no effect and the test's other assumptions hold. A p-value of 0.03 says the gap would be rare under no effect. It does not say there is a 97% chance the variant works.
Kohavi, Deng and Vermeer list that misreading among the most common in online experiments: a significant p-value does not mean a 5% chance the result is a false positive. How often a "win" is real depends on how often the ideas you test actually work, which the p-value doesn't know. When most ideas don't work, a fair share of significant results are false ones.
What does "chance to beat control" mean?
The chance to beat control is the probability, under the model, that the variant's true rate is higher than the control's. Benchmyrk starts every conversion goal from a flat prior (every rate equally plausible) and computes the probability exactly from a Beta-Binomial model, not by simulation, so the same data always gives the same number.
It says nothing about size. A variant can have a 99% chance to beat control with a lift too small to pay for itself, so read it with the credible or confidence interval. Benchmyrk also reports expected loss: the average you would give up by choosing the variant if it turns out worse.
chance to beat control ≈ 1 − p / 2When do Bayesian and frequentist results disagree?
They disagree when a result sits between the two thresholds. A two-sided p-value of 0.05 corresponds to roughly a 97.5% chance to beat control, so a variant at 95% can look like a win in Bayesian terms while the p-value is still near 0.1.
- Control
- 10,000 visitors, 280 orders (2.80%)
- Variant
- 10,000 visitors, 320 orders (3.20%)
- Lift and 95% interval
- +14.3% (−2.4% to +33.8%)
- Chance to beat control
- 95.1%
- Two-sided p-value
- 0.097
- Verdict after 21 days
- Still collecting evidence
The chance to beat control (95.1%) clears a 95% bar; the p-value (0.097) does not clear 0.05, and the interval still includes a small loss. Both are describing the same evidence: about 95.1% is what 1 − p/2 gives.
Neither number is wrong. The variant is probably better, but not by a margin the data can pin down yet. The planned sample size for this baseline is 57,135 visitors per variant, so the honest call is to keep the test running.
Example numbers, computed with Benchmyrk's statistics engine at default settings (95% confidence, 80% power, 10% minimum detectable effect). Not data from a store.
The other common disagreement is with very small or very lopsided samples, where the prior matters more. A flat prior keeps that effect small; a strong prior can pull a result towards whatever the prior assumed, which is why it should be agreed before the test, not tuned after.
Does peeking affect Bayesian tests too?
Yes. Checking a test every day and stopping the first time the number looks good raises the share of wrong decisions under both approaches. The Bayesian number is still a correct probability at each look, but a rule that stops on it whenever it crosses a threshold no longer keeps the error rate you expected.
| Simulation | Looked once at the end | Stopped on the first good look |
|---|---|---|
| Frequentist p below 0.05, A/A tests, daily checks from day 7 over 28 days (our peeking simulation, 1,000 tests) | 5.3% | 18.9% |
| Bayesian expected-loss rule, variant truly worse, daily checks over 20 days (Robinson 2015) | 2.5% | 11.8% |
David Robinson's simulation found the expected-loss rule shipped a worse variant in 2.5% of tests when judged once, and 11.8% with daily peeking, "more than four times as likely". His point is that Bayesian methods don't promise to control false positives in the first place. In our own simulation, the same daily checks with Benchmyrk's always-valid rule produced false winners in 1.3% of tests.
How does Benchmyrk report both?
Every result shows the frequentist and the Bayesian view together, and a winner needs both. The p-value it decides on is always-valid until the planned sample size is reached, so looking early does not create false winners.
- Lift
- +16.6% in both
- Chance to beat control, 12,350 visitors per variant
- 98.6%
- Fixed-horizon p-value
- 0.027
- Always-valid p-value (decides before the planned size)
- 0.350
- Verdict
- Still collecting evidence
- Chance to beat control, 24,700 visitors per variant
- 99.9%
- Always-valid p-value
- 0.047
- Verdict
- Winner found
At 12,350 visitors per variant, the chance to beat control and the classic p-value both look decisive, but the planned size is 53,211 visitors per variant, and the always-valid p-value hasn't crossed 0.05. With twice the visitors it has, and the verdict changes.
Example numbers, computed with Benchmyrk's statistics engine at default settings. Not data from a store.
Which should you use?
Use whichever your team reads correctly, and decide the stopping rule before launch. The choice of school matters far less than three habits: a planned sample size, a stopping rule that allows for checking early, and one primary goal agreed in advance.
- Explaining results to people outside the team: lead with the chance to beat control and the interval.
- Controlling how many false winners you ship over a year: rely on the p-value with a sequential or fixed-horizon rule.
- Either way: don't read a 95% chance to beat control as "95% sure the headline lift is real". It is the chance the variant is better at all.