Measuring tests when BFCM traffic spikes

During BFCM week three things change at once:

1. 1.Who arrives. Email and paid campaigns bring a larger share of past customers and deal seekers. A page that persuades a returning customer may do nothing for a first-time visitor in January.
2. 2.Why they buy. Many shoppers have already decided to buy something; the question is what and how much. Tests that help the decision (product pages) behave differently from tests that help the basket (thresholds, bundles).
3. 3.The calendar. Thanksgiving, Black Friday, the weekend and Cyber Monday each have their own pattern. A four-day window is one of each, not a typical week.

None of this makes a peak result wrong. It makes it narrow. Treat a peak winner as "won for BFCM shoppers", and re-test in a normal period before making it the default. Novelty effects fade as well, so a change that looked strong in a week of new visitors can settle lower.

The same four inputs as always decide the sample: the baseline rate, the smallest change worth detecting (relative), confidence (95%) and power (80%). What changes at the peak is the time you have. The sample size and test duration guide explains the inputs.

A brand at 10,000 visitors a day over the weekend can detect a 14% change in conversion rate in four days. Most good changes are smaller than that. So a four-day test at that traffic will usually end "inconclusive", which is fine if you planned for it and costly if you expected an answer. At 50,000 a day the window starts to be useful, but only for conversion rate. Revenue and profit per visitor need more visitors still.

The practical rule: run a peak test only if a change of the size in your row would matter to you, and if you've decided in advance what you'll do when the answer is "inconclusive".

Black Friday brings more large orders: gift buyers, stock-ups, wholesale customers using the retail site. One order ten times your average can swing revenue per visitor in a version with a few hundred orders.

• Cap large orders, decided before launch. Capping each order at, say, the 99th percentile of order values, applied to every version alike, keeps one order from deciding the test. Choosing the cap after seeing results is a way to pick the answer you prefer, so it must be fixed at launch.
• Plan for refunds. Peak orders are refunded more than usual, and refunds arrive over weeks. A version that sells more items on impulse can look better on day 5 and worse on day 40. Read revenue and profit after the return window (part 10).
• Keep one primary goal. Revenue per visitor or profit per visitor, decided before launch. Conversion rate alone can rise while revenue per visitor falls, especially with deeper offers. See revenue per visitor vs conversion rate.

There are two honest reasons to keep a test running through the sale, and they need different setups.

Earning. You have two versions you'd be happy to run, and you want more of the sale to go through the better one. An auto-allocating (bandit) test moves traffic toward the version that's ahead as data comes in. It earns more during the sale. It gives a weaker answer about the size of the difference, because the losing version gets fewer visitors.

Learning. You want to know which version is better, and by how much, for BFCM shoppers. Keep a fixed split for the whole window. Accept the detection limits in the table above.

Pick one before launch. Switching from fixed to auto-allocate halfway through mixes two designs and makes the result hard to read.

Sample ratio mismatch (SRM) means the split you got differs from the split you set by more than chance allows. A chi-square test checks it, and Benchmyrk flags it at p < 0.001. Research on SRM in large experimentation programs lists the usual causes: redirects that lose visitors in one version, bots, caching, and logging that fails differently between versions (Fabijan and colleagues, KDD 2019).

Peak traffic makes each cause more likely:

• Caching. CDNs and apps cache more aggressively under load. A cached page can show one version to everyone who gets it.
• Bots. Price scrapers and resellers' bots spike in sale week, and may only see one version.
• Redirects. A split URL test adds a page load to one version. On a slow sale-day connection, more visitors abandon that redirect.
• Script errors. An app that fails under load may break one version only.

If SRM fires, don't trust the result. Find the cause first. The SRM guide goes through each.

You will look at results every day of the sale. That is fine if the method allows it. With a fixed-horizon test, checking repeatedly and stopping at the first "significant" reading raises the false-winner rate well above 5% (why repeated checking inflates false winners). An always-valid sequential method lets you look without that penalty. Either way there's no verdict before 7 full days, and a four-day peak test can't produce one, so what you check daily is the split, the guardrails and orders arriving, not the winner.

• Treating a BFCM winner as the new default without re-testing in a normal period.
• Starting a peak test with no traffic maths. It will end "inconclusive" and feel like a waste.
• Choosing the large-order cap after seeing results.
• Reading revenue before refunds.
• Switching to auto-allocate mid-test.
• Ignoring SRM because the sale is busy. A result with SRM is not a result.

1. 1.For each peak test: the smallest change it can detect in your window, written down.
2. 2.One primary goal, chosen before launch.
3. 3.Large-order cap set before launch, or deliberately off.
4. 4.Earning (auto-allocate) or learning (fixed split) decided before launch.
5. 5.Guardrails set: checkout completion and page errors.
6. 6.A daily check of split, guardrails and orders during 26 to 30 November.
7. 7.A read date after the return window.

Reporting peak results to clients? The agency track on the BFCM hub covers reporting test results to clients during BFCM. The full season plan is in What to test before Black Friday and Cyber Monday on Shopify.