What not to test over BFCM

During BFCM, shoppers compare more than at any other time of year. They open the store on two devices, send a link to a friend, check a deal site, and screenshot the price. If one version says 25% off and the other 15% off, or one shows $48 and the other $54, some of them will see both.

Three things then go wrong:

1. 1.Trust. A shopper who finds a better price on a friend's phone feels cheated, even if the lower price was a test. Support contacts and refund requests follow.
2. 2.The test itself. Shoppers who see both versions don't behave like either group. The comparison stops being clean.
3. 3.Price-reference rules. In the EU, any announced price reduction must show the lowest price applied in the previous 30 days (Directive (EU) 2019/2161, which added Article 6a to Directive 98/6/EC). In the US, the FTC's guides on former price comparisons say a "was" price should be one the item was genuinely offered at for a reasonable period (16 CFR 233.1). A price that differs by test version makes it harder to show a single honest reference price. This is information, not legal advice.

Instead: test the format of the offer in October (percent off vs spend tiers, at a similar cost), settle the depth from your margin, and run one depth for everyone during the sale. Part 4 has the break-even maths.

A checkout block, a new theme, or a new app is a change to the store, whether or not it's labeled a test. After 16 November, there's no time to find out what it breaks.

This includes apps installed "for the sale": a countdown timer, a popup, a new upsell widget. Each one adds scripts to every page. If you want to know what an app is worth, test it in October by holding it out from a share of visitors, and decide before the freeze. Part 3 lists what counts as a change.

Instead: install, test and decide in October. If an app didn't earn its place by mid-November, it doesn't go live for the sale.

Each extra version needs its own sample, and comparing several versions with the control needs a correction for multiple comparisons, which raises the sample again. A page that gets a modest share of your traffic can't support that, least of all in four days.

Even two versions need six weeks on this page. Four versions need almost four months. Over a four-day sale, none of these can produce an answer, and part 8 shows how large a change even a whole-store peak test needs. Adding versions doesn't give you more answers; it gives you more chances of a false winner and longer waits for a real one.

Instead: run two versions on your highest-traffic pages, in October. Keep the other ideas for January.

Showing different offers to different audiences is normal: a welcome code for first-time visitors, a loyalty offer for subscribers. It becomes a problem when shoppers can see the difference and can't see the reason. "Returning customers get 10%, new customers get 25%" is the classic case, and in BFCM week those customers talk to each other.

Instead: make any audience-specific offer explainable in one sentence ("subscribers get early access", "first order gets free shipping"), and keep the sale's headline offer the same for everyone.

During the sale, the cost of a broken test is measured in hours of revenue. Before anything runs over the peak, you should be able to answer, in writing:

• Who can stop it?
• How, in one action?
• What do shoppers see when it stops?
• Does stopping it leave anything behind (a changed price, a discount, a checkout block)?

If any answer is "I'd have to ask the agency", it doesn't run over the sale.

Instead: write the stop instruction next to every live test, app and discount, and test it once before 16 November.

• Tests that overlap on the same element. Two tests changing the same banner or the same price can't both be read. Make them mutually exclusive, or run one.
• A change in consent or tracking. It changes who is counted halfway through your tests. Part 2 covers this.
• Brand-new hypotheses. BFCM is a bad time to learn something new. It's a good time to use what you learned in October.

• "It's only a small test." Small tests are the ones that can't reach a sample in four days.
• Testing depth "just for a day". Shoppers who saw both prices remember both.
• Approving an exception in a chat thread without who, why and how to stop.
• Keeping a losing app live because it's paid for.

1. 1.No running test splits price or offer depth over 26 to 30 November.
2. 2.No theme publish, new app or checkout block after 16 November.
3. 3.Every peak test has two versions and a written detection limit (part 8).
4. 4.Audience-specific offers explainable in one sentence.
5. 5.A written one-click stop, with an owner, for every live test, app and discount.
6. 6.Overlapping tests on the same element made mutually exclusive.

Running tests for several clients? The agency track on the BFCM hub covers what agencies shouldn't test over BFCM. The full season plan is in What to test before Black Friday and Cyber Monday on Shopify.