What loads on the storefront
- The test settings, inline. On Shopify, the app embed prints the running tests' settings into the page's head: about 1.4 KB of HTML on the page we measured. No extra request is needed to know which tests run.
- bc.js, 11.3 KB. That is its size on the wire with brotli compression, served from Shopify's CDN and cached for a year. While a test runs it loads as a parser-blocking script, so variants apply before the page paints; otherwise it is deferred.
- Shopify's web pixel, which Shopify loads in its own sandbox to report checkout events.
- Heatmap and session replay scripts load only when a store turns those features on. They were off for this measurement.
How we measured
Lighthouse 12.8.2 (performance category only) in headless Chrome on a Dawn-based development store, on a product page while one test ran there (a headline change, 50/50). Each setting ran three times as served and three times with our scripts blocked, alternating, and we report medians. We ran both Lighthouse's default simulated throttling and DevTools throttling, on its mobile and desktop profiles. Lighthouse's visits were dropped by our bot filter, so the live test's numbers stayed clean while the script itself ran normally.
The results
| Setting | First contentful paint | Largest contentful paint | Total blocking time | Layout shift |
|---|---|---|---|---|
| Mobile, simulated, as served | 3,001 | 3,649 | 0 | 0 |
| Mobile, simulated, script blocked | 2,997 | 3,257 | 0 | 0 |
| Mobile, DevTools, as served | 2,122 | 3,176 | 58 | 0 |
| Mobile, DevTools, script blocked | 2,103 | 3,160 | 63 | 0 |
| Desktop, simulated, as served | 1,149 | 1,149 | 0 | 0 |
| Desktop, simulated, script blocked | 1,037 | 1,037 | 0 | 0 |
| Desktop, DevTools, as served | 1,262 | 1,262 | 0 | 0.0028 |
| Desktop, DevTools, script blocked | 749 | 749 | 0 | 0 |
- Main thread: close to nothing. The script used 1 to 6 ms of main-thread time and added no total blocking time in any run. The page's blocking time comes from Shopify's and the theme's scripts and is the same with ours blocked.
- Layout shift: 0 to 0.003, against Google's "good" threshold of 0.1.
- Render blocking: about 100 ms on a cold first view. Downloading a parser-blocking script sits on the critical path. With desktop simulated throttling that is 112 ms of first paint; under DevTools desktop throttling it was 513 ms. Repeat views load the script from cache.
- Mobile simulated runs are noisy. Largest paint ranged from 2.8 to 5.3 s within one setting, driven by the store's response time, so the 392 ms difference there isn't meaningful. The DevTools mobile runs are tight: 16 ms.
So the honest summary: our script costs a cold first view roughly a tenth of a second, and nothing after that. We don't say "zero impact", because it isn't.
What the measurement found: a flicker
Looking frame by frame, variant visitors briefly saw the original headline before the variant replaced it, about a third of a second after first paint. The cause was consent: on Shopify the script waits for the store's consent decision before it assigns a visitor, and Shopify's consent API was ready only after the page had painted.
The fix (7 October): while the consent decision is pending, the script now hides only the elements the page's tests could change, for every visitor alike, so both arms wait the same time and timing can't bias the result. It reveals them as soon as the decision arrives and the change applies, and never later than 800 ms. If consent is denied, the original page shows and nothing is stored or sent.
We verified the fix in a real browser against a storefront whose consent API answers 400 ms late, sampling the headline on every frame: the original is never visible to a variant visitor. We'll repeat the cold first-view measurement on the live development store and update this post with it.
Check it on your store
Run Lighthouse on one of your pages with a test running, then again with --blocked-url-patterns for our script, and compare medians of a few runs. If the numbers on your store differ from ours, tell us: we'd rather know.