How to read an A/B test result

The conversion rate is the share of visitors in a variant who converted on the goal at least once: converting visitors divided by visitors. Benchmyrk counts visitors, not page views, so one shopper who reloads the page ten times counts once.

The lift is the variant's rate relative to the control's. A control at 3.00% and a variant at 3.50% is a lift of +16.6%. Lift is always relative: a "+17%" lift on a 3.0% rate is about half a percentage point.

The observed lift is one estimate. The 95% interval next to it is the range of lifts that fit the data. A narrow interval means the estimate is precise; a wide one means it could still move a lot.

• If the whole interval is above zero, the data rule out "no effect" at the 95% level for this comparison.
• If it includes zero, the variant could be better, the same or worse.
• Its width shrinks roughly with the square root of the number of visitors: four times the traffic halves it.

Read the low end as well as the middle. A lift of +16% with an interval from +2% to +34% says the change helps, but it could be a small help. Plan the rollout around the low end, not the headline number.

The chance to beat control is a Bayesian probability: given the data so far, how likely the variant's true rate is higher than the control's. Benchmyrk calculates it exactly from a Beta-Binomial model for conversion goals.

It is not the chance that the lift is as large as observed, and it says nothing about how big the difference is. A variant can have a 99% chance to beat control with a lift that is too small to matter. That is why it sits next to the interval, and why a winner needs both: at least a 95% chance to beat control and a significant p-value.

A p-value is the probability of seeing a difference at least this large if the variant truly made no difference. The classic rule is to call a result when p is below 0.05. That rule assumes you look once, at a sample size you fixed in advance.

Most teams look every day and would stop the moment the number turns green. Looked at that way, the classic p-value finds false winners far more often than 5% of the time (our A/A simulation puts it near one in five over four weeks of daily checks). So until a test reaches its planned sample size, Benchmyrk decides on an always-valid p-value, which stays valid however often anyone looks. After that, the classic fixed-horizon test applies.

With two or more variants, each comparison with the control gets its own p-value, and the more variants you add, the more likely one of them looks good by chance. Benchmyrk adjusts the p-values with the Holm correction, so adding variants doesn't raise the chance of a false winner.

• The traffic split. If you planned 50/50 and the variants got noticeably different numbers of visitors, something is filtering visitors unevenly and the result can't be trusted. Benchmyrk runs a sample ratio check and flags the test "Check traffic split" instead of showing a winner. See the SRM guide.
• Full weeks. Shoppers behave differently on weekdays and weekends. Benchmyrk gives no verdict before 7 full days; running whole weeks is better still.
• Guardrails. A variant that lifts orders but raises refunds or hurts a guardrail metric isn't a win. Check the guardrail goals before shipping.
• What changed during the test. A sale, a campaign or a theme update that hits one variant harder than the other can move the numbers. Note them with the decision; the audit log shows changes made to the test itself.

Conversion rate counts orders; revenue per visitor counts money. They can point different ways: a discount banner can raise the conversion rate and lower revenue per visitor, and a premium bundle can do the opposite. For anything that touches price, offers or order size, judge the test on revenue or gross profit per visitor.

Revenue per visitor is noisier than conversion rate, because order values vary, so it needs more visitors for the same certainty. Benchmyrk compares it with Welch's t-test, which doesn't assume both variants have the same spread, and takes refunds and cancellations off the variant that made the sale. Gross profit per visitor also takes off product costs from Shopify, payment fees and shipping costs; there is no profit verdict while costs are known for less than 90% of order value.

Breaking a result down by device, new or returning visitors, traffic source or country is useful for ideas. It is not a way to rescue a test. With ten segments, one will usually look like a winner by chance. Benchmyrk labels segment results exploratory and doesn't adjust their intervals for the number of segments. If mobile shoppers seem to love the variant, run a test targeted at mobile to confirm it.

1. 1.Read the verdict on the primary goal.
2. 2.Check the traffic split, the days run and the guardrails.
3. 3.Look at the interval's low end to size the decision.
4. 4.Ship the winner (a staged rollout lets guardrails watch each step), or keep the control.
5. 5.Record the decision and why, so the next test starts from it.

A test that ends with no meaningful difference is still an answer: the change didn't matter enough to detect at the size you planned for. Keep whichever version is simpler to maintain, and use the sample size calculator to plan a bolder test next time.