After Cyber Monday: analyze the tests and the sale honestly

BFCM orders are returned more than usual, and returns arrive over weeks. A version that sold more on impulse can look best on 1 December and worst on 15 January. If your return window is 30 days, the last Cyber Monday refunds arrive around the end of December.

Before reading any revenue or profit result from November:

• Check that refunds reduce credited revenue in your results.
• Set the read date after the return window, and write it down.
• Read conversion rate earlier if you like, but don't decide on it alone.

Your December report will show that November revenue was far above October. That's the sale: the traffic, the offer and the season. None of it is evidence that a test worked.

A test result is the difference between two groups who were in the store at the same time, in the same sale, with only the tested change between them. That comparison is the only part of the report that tells you what a change added. Two common mistakes:

1. 1.Before-and-after. "We shipped the new product page on 13 November and conversion rose 40% in sale week." Everything else changed too. A before-and-after comparison can't separate the page from the sale.
2. 2.Adding up test lifts. Three tests that each won by 5% didn't add 15% to the sale. Lifts measured in different periods, on different pages and different shoppers, don't add up, and winners' measured lifts tend to be overestimates of the true effect.

If you want a number for the whole program, you need a group that saw none of your changes. That's a holdout, and it has to be set up before the sale.

For each test that ran over the peak, read the primary goal once, after the read date, with its 95% interval and the chance of beating the control. The how to read an A/B test result guide explains each number.

How to report these honestly:

• The delivery line won, for BFCM shoppers. The true effect could be as small as 0.6% or as large as 15%. A delivery promise matters more in the last week before holiday cut-offs. Ship it, and re-test it in January before treating 7.5% as its normal effect.
• The progress bar shows no difference. Both tests reached their planned sample, and this one wasn't significant. "No difference" here means a change of the size you planned to detect was ruled out, not that the bar does nothing. An 86% chance to beat control is not a win.

After a big sale, everyone slices the results: by device, country, new or returning, traffic source. Some slice will always look like a winner. With enough slices, that happens by chance alone.

Segment results are exploratory. A segment "win" is a hypothesis for the next test: write it down, with the reason it might be real, and test it in that segment in January. If you do ship to a segment, use a lift adjusted for the number of segments you looked at, not the raw one.

Two effects make peak results look larger than they will be later:

1. 1.Novelty. A new design attracts attention from returning customers because it's new. That fades.
2. 2.Season. BFCM shoppers are more motivated, more deal-focused, and more often first-time buyers. A result from that week describes them.

Neither makes the result wrong. Both are reasons to re-test peak winners in a normal period before treating them as permanent.

For each test, write down five things while you still remember them:

1. 1.The hypothesis and its mechanism.
2. 2.The primary goal, the result, its interval and the verdict.
3. 3.What ran alongside it (sale dates, offers, other tests).
4. 4.What you decided, and why.
5. 5.The re-test, if any, with a date.

A year from now, this record is what turns "we tried that once" into knowledge. It is also the starting point for next year's BFCM plan.

If you tested next-order codes on the thank-you page (part 7), their answer isn't ready in December. Read repeat orders at 30 days in early January and at 90 days in late February. Count only orders from buyers in each version, and subtract refunds and cancellations.

• Reporting the sale's growth as the tests' effect.
• Reading revenue before refunds.
• Adding up lifts from different tests.
• Shipping a segment win without re-testing it.
• Making a peak winner the permanent default without a January re-test.
• Not writing anything down. The next sale starts from what you recorded.

1. 1.A read date after the return window for every revenue or profit result.
2. 2.Each test read once, with its interval and verdict.
3. 3.Peak winners marked "won for BFCM shoppers", with a January re-test date.
4. 4.Segment wins written down as hypotheses.
5. 5.Thank-you code reads scheduled at 30 and 90 days.
6. 6.Every test written into the learning record.
7. 7.Next year's first draft of the BFCM plan started from that record.

Reporting to clients after the sale? The agency track on the BFCM hub covers the post-BFCM client report. The full season plan is in What to test before Black Friday and Cyber Monday on Shopify.