The pre-BFCM tracking and QA audit for every client store

For each client store, confirm the parts that carry data into results are working, not just installed. On Shopify that means:

• The web pixel exists with the right site key and is sending events. Cart, checkout and purchase events come from it.
• The app embed is switched on in the live theme. A theme published last week may not have it.
• Webhooks for orders and refunds are arriving, because orders are counted from the order Shopify confirms, not from a page view.
• The plan is active, so experiments can run.

Do this again after any theme publish. One easy way for tracking to break in November is a new theme going live without the app embed turned on.

Compare a week of orders in your testing tool against the same week in Shopify. They rarely match exactly, and they shouldn't be expected to. What matters is that you can explain the gap.

Common reasons the numbers differ:

• Orders that never passed through a tracked visit: point of sale, draft orders, wholesale and other sales channels.
• Shoppers who declined analytics, where the store measures only with consent.
• Test orders placed by the team. Benchmyrk leaves them out of results unless "Include test orders" is turned on for a goal; other tools may not.
• Refunds and cancellations, which some reports subtract and others don't.
• Time zones, when the store and the report cut days at different hours.

Write down the gap as a share of orders and the reasons you found. A store where 40% of online orders can't be matched has a tracking problem to fix before testing; a store where the gap is small and explained is ready. The help article on why orders don't match Shopify lists what to check.

Clients selling into the EU, the UK or US states with privacy laws often run a consent banner. How the banner and the testing tool interact decides who gets counted.

For each client, find out:

• Which regions see a consent banner, and which tool runs it.
• Whether variants may be shown before consent, and whether visits may be measured before consent.
• Whether the banner talks to Shopify's Customer Privacy API, which lets apps check whether analytics is allowed for a visitor and records their choice. Whether processing is allowed depends on the merchant's settings, the visitor's region and the consent given (Shopify developer documentation, read October 11, 2026).

Consent affects every arm alike, so it doesn't bias a comparison by itself. It does shrink the sample. If 30% of a client's visitors decline analytics and the store measures only with consent, each day brings 70% of the sample, so the test needs about 43% more calendar time (1 ÷ 0.7 = 1.43). Put that into the plan from part 1. This is information, not legal advice: the client's own adviser decides what their banner must do.

During BFCM prep, agency and client teams visit the store dozens of times a day. Those visits and test orders shouldn't land in results. Exclude office IP ranges, mark team browsers as internal, and agree that test orders are placed only from those browsers. The help article on excluding internal traffic covers each option.

An A/A test runs two identical arms. Nothing differs, so the result should show no difference and a split close to the one you set. It checks the assignment, the counting and the order attribution on the real store.

Read two things:

1. 1.The split. A sample ratio mismatch (SRM) means the visitors you got per arm differ from the split you set by more than chance allows. It is tested with a chi-square test, and Benchmyrk flags it at p < 0.001. An SRM means something in the setup treats arms differently: redirects, bots, caching, consent settings, internal traffic or a script error in one arm. Don't trust results from a store with an unexplained SRM. The SRM guide has the causes.
2. 2.The result. It should be inconclusive. A single A/A test can still show a "significant" difference by chance, about 5% of the time at 95% confidence, so read it as a check on the setup, not a pass or fail.

For each store, record the date of the audit, each check and its outcome, the order-matching gap and its reasons, the consent settings, and the A/A split and result. Keep it with the client's test records. When someone asks in December why a result can be trusted, this is the page you show them.

• Auditing once. A theme publish or a new consent banner can break tracking overnight. Re-check on freeze day.
• Expecting a perfect order match. Explain the gap instead.
• Skipping the A/A test because "the tool is installed". Installation isn't the same as counting correctly.
• Treating a small SRM as a warning to ignore. A mismatch at p < 0.001 means the result can't be trusted, whatever the lift says.
• Letting the team's test orders into results. Mark browsers as internal before the prep weeks begin.

• [ ] Web pixel, app embed, webhooks and plan all working
• [ ] A week of orders compared with Shopify; gap and reasons written down
• [ ] Consent regions, banner tool and settings recorded
• [ ] Office IPs and team browsers excluded
• [ ] A/A test run; split checked; result inconclusive
• [ ] Audit written into the client's records with the date
• [ ] Health re-checked on the morning of November 16

Auditing one brand's store rather than a portfolio? Part 2 of the brand track covers the same ground for a single store; start from the BFCM hub. The general timeline is in the guide to testing before Black Friday.