The short answer
- Every test result is only as good as its counting. Before BFCM, check that each checkout step and each purchase is recorded once, for the right version, and that test orders, internal visits and declined consent are handled the way you expect.
- Run an A/A test once the checks pass. Two identical versions for two weeks, from late October. It proves the split is even and the counting is consistent, before you rely on it for a real decision.
- Know what a split warning (SRM) means. If the split you got differs from the split you set by more than chance allows, don't trust the result. Find the cause first.
- Reconcile test orders with Shopify, and write down the expected gaps. Test results will never equal Shopify's order count, and that's correct. What matters is that you know why.
Who this is for: Shopify brands doing $1M to $10M a year about to rely on test results for BFCM decisions, especially if tracking was set up by someone who has since moved on.
The 2026 timeline
| When | What to do |
|---|---|
| Tue 20 Oct to Fri 23 Oct | Run the checks below. Fix what fails. |
| Mon 26 Oct to Mon 9 Nov | A/A test, after the checks above. |
| Mon 9 Nov | Reconcile two weeks of test orders with Shopify. Write down the expected gaps. |
| Mon 16 Nov | Freeze. No changes to tracking, consent banners or the pixel after this. |
| 26 Nov to 30 Nov | Check daily that orders are arriving and the split is even. |
Step 1: Check the checkout steps and purchases
On Shopify, checkout pages can't run storefront scripts, so checkout steps and purchases come from a web pixel. Web pixel extensions are available on every plan except Shopify Starter (Shopify developer docs, read 11 October 2026).
Walk the path yourself, on a phone and on a desktop:
- Open a product page in a private window. Add to cart. Open the cart.
- Start checkout. Fill in contact, then shipping, then payment.
- Place a test order.
- In your testing tool, confirm each step appeared once, for the visit you just made, and that the order arrived.
Then repeat with Buy it now or an accelerated checkout button. Those skip the cart page, so a funnel report should still count the later steps for those shoppers.
Test orders. Most tools leave Shopify test orders out of results by default, because a test order is not a sale. In Benchmyrk, test orders don't count in results unless "Include test orders" is turned on for a goal. Switch it on only for a dry run, then switch it off again.
Step 2: Check consent by region
Consent changes who is counted. Two separate questions:
- Can a shopper see a variant before they answer the consent banner? If not, everyone who hasn't answered sees the original, which dilutes the test.
- Is a shopper measured before they consent? If you only measure after consent, shoppers who decline are left out of results.
Both are legitimate choices. What matters is that the same rule applies to every version, so the comparison stays fair, and that you know your results describe consenting shoppers. If you sell in the EU and the US with different banner behavior per region, check both regions with a VPN or a colleague abroad.
Step 3: Leave your own team out
During BFCM preparation your team visits the store constantly: checking banners, placing test orders, reviewing landing pages. Those visits are not shoppers.
- Add your office IP ranges.
- Give remote staff and your agency a way to mark their browser as internal.
- Tell anyone placing test orders to use the internal setting.
Internal visitors should still see experiments, so they can QA them. Their visits, events and orders shouldn't count.
Step 4: Run an A/A test
An A/A test splits visitors between two identical versions. Nothing should differ. It checks three things at once:
- The split. Each version should get its share of visitors, within chance.
- The counting. Orders and revenue should be similar in both, within chance.
- The pipeline. Goals, attribution and reports work end to end on real traffic.
At 95% confidence, a correctly run A/A test will still show a "significant" difference about 5% of the time on a single fixed-horizon read. That's what 95% means. One A/A "winner" is not proof of a problem; a split warning is.
What a split warning (SRM) tells you
A sample ratio mismatch (SRM) means the split you got differs from the split you set by more than chance allows. It is checked with a chi-square test, and Benchmyrk flags it at p < 0.001. When it fires, don't trust the result until you've found the cause. Common causes: redirects that lose visitors in one version, bots, page caching, consent settings applied unevenly, internal traffic, or a script error in one version. The SRM guide goes through each.
Illustrative counts for a 50/50 A/A test. p-values computed with Benchmyrk's own SRM check (Pearson chi-square).
| Version A visitors | Version B visitors | B short by | p-value | Flagged at p < 0.001? |
|---|---|---|---|---|
| 60,000 | 59,000 | 1.7% | 0.0037 | No |
| 60,000 | 58,800 | 2.0% | < 0.001 | Yes |
| 60,000 | 58,500 | 2.5% | < 0.001 | Yes |
| 6,000 | 5,700 | 5.0% | 0.0055 | No |
At brand traffic levels, a gap of 2% is already very unlikely by chance and gets flagged. At a tenth of the traffic, even a 5% gap doesn't. The bigger your store, the smaller the tracking problem an SRM check can catch, which is one reason an A/A test is worth more to a $5M brand than to a new store.
Step 5: Reconcile orders with Shopify
Compare two weeks of orders in your testing tool with Shopify's order report for the same dates. They won't match. Write down why, so nobody panics in December.
| Gap | Why it's expected |
|---|---|
| Orders outside the online store | Point of sale, draft and manual orders, and orders from other sales channels have no storefront visit to tie them to. |
| Subscription renewals | Charged by your subscription app, not placed in a visit. |
| Shoppers who declined analytics | Not counted, in every version alike. |
| Test orders | Usually left out. |
| Shoppers not in any test | A test only counts visitors who entered it. Shopify counts everyone. |
| Refunds and cancellations | Reduce credited revenue, and can arrive weeks later. |
| Date ranges and time zones | Check both reports use the same dates and time zone. |
What isn't expected: purchases counted twice, orders credited to a version the shopper never saw, or a share of web orders without a tracked visit that jumps from one week to the next. Those are reasons to fix tracking before BFCM. The help article why test orders don't match Shopify's numbers goes through each line.
Mistakes to avoid
- Auditing on desktop only. Phone checkouts use different buttons, wallets and layouts. Check both.
- Leaving test orders switched on after a dry run.
- Changing the consent banner in November. It changes who is counted halfway through your tests.
- Ignoring a split warning because the lift looks good. A result with an SRM is not a result.
- Treating any gap with Shopify as a bug. Know which gaps are expected.
Checklist
- Each checkout step and the purchase recorded once, on phone and desktop.
- Buy it now and accelerated checkout paths checked.
- Test-order handling known, and switched back after any dry run.
- Consent behavior checked in each region you sell to.
- Office IPs, remote staff and the agency marked as internal.
- A/A test run for one to two weeks with no split warning.
- Two weeks of orders reconciled with Shopify, gaps written down.
An A/A test is an A/B test with an unchanged variant; see A/A test in the glossary.
Auditing several client stores? The agency track on the BFCM hub covers a tracking and QA audit for every client store. The full season plan is in What to test before Black Friday and Cyber Monday on Shopify.