Planning BFCM tests across a client portfolio

These dates come from our general guide to testing before Black Friday. The freeze date is a suggestion, not a Shopify rule. Move it earlier for clients whose developers or app vendors need more lead time.

The clients who ask loudest for "a Black Friday test" are not always the ones who can run one. What decides it is how many visitors reach the tested pages in the time left, and how big a change you need to detect.

For each client, write down four numbers:

1. 1.Daily visitors to the pages you would test. Not sessions to the whole store, only the pages the test changes.
2. 2.The baseline rate of the primary goal on those pages, for example purchases per visitor.
3. 3.The smallest relative change worth acting on (the minimum detectable effect, or MDE). A 10% relative lift means 2.5% to 2.75%.
4. 4.The days available. A test started on October 25 and read on November 15 has 21 days.

Put them into the sample-size calculator at 95% confidence and 80% power. If you need a refresher on why those four inputs matter, the sample size and test duration guide covers it.

Revenue and profit goals need more visitors than conversion rate, because order values vary a lot from one order to the next. Treat the conversion-rate numbers below as the best case.

From that table, sort each client into one of three groups:

• Test: the client can detect a change you'd act on, before November 15.
• Test one thing only: the client can run one well-planned test. Pick the one whose answer you will use during the sale, usually the offer format or the landing page your ads will send people to.
• Fix, don't test: the client can't reach a useful sample. Give them a list of fixes instead (broken links, slow pages, unclear delivery dates, a checkout that asks for too much). Say plainly why. A client who hears "your store can't measure a 10% change in three weeks" in October trusts you more in January than one who hears "the test was inconclusive" in December.

Once you know which clients will test, give every test the same four dates: planned start, planned end, a decision date and the freeze. Agencies get into trouble when each client's plan lives in a different spreadsheet and nobody can see that four tests end on the same Friday.

A shared calendar answers three questions every morning:

• Which tests start or end today, and for which client?
• Which decisions are due this week?
• Who is on point for each one?

Plan backward from the freeze. A three-week test that slips by a week now ends on November 22, inside the freeze. Decide in advance what happens then: you stop it and read what you have, or you let it run untouched through the peak and label the result as a sale-week result.

A single freeze date for every client is easier to staff, easier to explain and harder to break. From the freeze to the end of Cyber Monday:

• No new tests, no theme publishes and no new apps.
• No changes to prices, discounts, shipping or checkout, except the planned sale offers.
• Copy fixes only through the approval route you agreed with the client.

Part 3 of this series is a code-freeze agreement you can copy for each client; find it on the BFCM hub.

One Shopify detail affects every client with a sale offer: discounts only combine when each discount's settings allow it. Shopify allows up to 25 active automatic discounts per store, app discounts included (Shopify Help Center, read October 11, 2026). Benchmyrk's price, offer and shipping tests use up to four of them. If a client's sale offer, a loyalty perk and a test all use automatic discounts, count them before the freeze.

During the freeze nobody should be building. The work is watching, and it scales with the number of clients, not the number of tests. For each client, name one person who checks once a day from Thursday to Monday:

• Orders and revenue against the same days of the previous week, so a broken checkout shows up within hours.
• Guardrails on every test still running, and whether any test paused itself.
• The split between variants (a sample ratio mismatch is a stop signal; see the SRM guide).
• Support messages that mention a price, an offer or a banner.

Write down who can pause a test for each client without asking first, and how they tell the client afterward. Over a holiday weekend, waiting for a sign-off can cost more than the test was worth.

• Promising every client a Black Friday test. Traffic decides, and some clients will get a fix list.
• Starting tests after November 1. A two-week test started later ends inside the freeze.
• Running four variants for a mid-traffic client. Every extra variant splits the traffic further and adds a comparison to correct for. Use two arms before the sale.
• Different freeze dates per client. Your team ends up shipping for one client while another is locked, and mistakes follow.
• Reading results on the day they look good. Checking a fixed-horizon test daily and stopping at the first significant reading finds false winners far more often than 5% of the time; see our peeking simulation.

• [ ] Daily visitors to the pages you would test, and the baseline rate
• [ ] Sample size and days at 95% confidence and 80% power
• [ ] Group: test, test one thing, or fix first
• [ ] Each test's start, end, decision date and owner on the shared calendar
• [ ] Freeze date agreed in writing
• [ ] A named person for each day from Thanksgiving to Cyber Monday
• [ ] Who can pause a test without asking, and how the client is told

Running the same plan for a single brand rather than a portfolio? See the BFCM testing plan for brands. All three tracks start from the BFCM hub.