Test typesShopify
Test a whole new Shopify theme on real shoppers.
Send a share of shoppers to an unpublished theme, such as a copy of yours with a new header and cart drawer, and compare orders and revenue with your live theme.
How a theme test works
Shoppers in the test group browse your unpublished theme through Shopify's own theme preview. Everyone else gets the live theme.
Prepare the theme
Duplicate your live theme in Online Store, Themes, make your changes, and turn on the Benchmyrk app embed in that theme's editor. Benchmyrk never writes theme files.
Launch checks
Before the test starts, Benchmyrk reads your themes from Shopify: each test theme exists and is unpublished, and the app embed is on in the live theme and every test theme.
First page
A shopper in the test group is sent once to the same page on the test theme, with Shopify's preview bar hidden by Shopify's own
pb=0setting. Campaign parameters such as UTM tags are kept.Browsing
Shopify keeps showing the test theme for the rest of the visit, so later pages need no redirect. Cart and checkout are your store's as usual.
Paid order
The shopper's group travels with the cart, so orders, revenue and gross profit per visitor are credited to the theme they browsed.
After the test
Pausing or ending the test sends shoppers on the test theme back to the live theme on their next page. You publish the winning theme in Shopify yourself.
| Step | What happens |
|---|---|
| Prepare the theme | Duplicate your live theme in Online Store, Themes, make your changes, and turn on the Benchmyrk app embed in that theme's editor. Benchmyrk never writes theme files. |
| Launch checks | Before the test starts, Benchmyrk reads your themes from Shopify: each test theme exists and is unpublished, and the app embed is on in the live theme and every test theme. |
| First page | A shopper in the test group is sent once to the same page on the test theme, with Shopify's preview bar hidden by Shopify's own pb=0 setting. Campaign parameters such as UTM tags are kept. |
| Browsing | Shopify keeps showing the test theme for the rest of the visit, so later pages need no redirect. Cart and checkout are your store's as usual. |
| Paid order | The shopper's group travels with the cart, so orders, revenue and gross profit per visitor are credited to the theme they browsed. |
| After the test | Pausing or ending the test sends shoppers on the test theme back to the live theme on their next page. You publish the winning theme in Shopify yourself. |
Watched while it runs
Theme changes caught
If a test theme is deleted or published, or the app embed is turned off, the test comes off your store and your team is told. It returns once fixed.
No redirect loops
If Shopify doesn't switch a shopper's theme, the test stays off for that browser for the session, and nobody is counted for a theme they didn't see.
Crawlers see the live theme
Search engine crawlers, audit tools and the theme editor always get your live theme, and canonical URLs don't change.
Why not Shopify Rollouts?
Shopify's Rollouts lets you run a theme experiment from your Shopify admin, with Shopify's own analytics. If those numbers are enough for you, it's a fine place to start.
Apps can read rollouts through the Admin API (since version 2026-10), but they can't create, run or end one, or see which version a given buyer got. So Benchmyrk can't run or measure a theme test through Rollouts, and uses Shopify's theme preview instead.
- What you get in Benchmyrk: revenue and gross profit per visitor, refunds taken off, and the same peeking-safe statistics as every test. How results are judged.
- Only changing one page type? A template test needs no second theme and no redirect between themes.
What to test first
A theme test is worth it when the change is too big for a page edit. These are examples of tests to run, not results.
- 01
Example: new header and navigation
A simpler menu or a sticky header across the whole store. Judge it on revenue per visitor, since it touches every page.
- 02
Example: cart drawer or cart page
A drawer that opens on add to cart against your cart page. Watch conversion and order value together.
- 03
Example: moving to a new theme
Before switching to a new theme or a major theme update, run it against the old one so a drop shows up in a test, not in your monthly numbers.
How long to run it
A theme test runs on every page, so every visitor counts toward it. Store-wide conversion is low, though, which makes the sample large.
Example: at a 2.5% conversion rate, spotting a 10% relative change (2.5% to 2.75%) needs about 64,199 visitors per version, at 95% confidence and 80% power. Run full weeks, at least two.
When the aim is to switch safely rather than to win, look at the interval for revenue per visitor: if its lower end is a drop you can live with, switching is low risk. Plan with the sample size calculator.
Good to know
- Shoppers in the test group load their first page twice (one redirect), as with every tool that tests whole Shopify themes. While Benchmyrk decides, the page is hidden for a capped time.
- Theme previews are a Shopify feature for checking a theme, not a documented way to serve shoppers. How long Shopify keeps a browser's preview isn't documented; if it lapses, the shopper is sent back to the test theme once more.
- Keep a test theme for 60 days after the test. A deleted test theme shows Shopify's error page to browsers still on it, and Benchmyrk's code can't run there.
- The theme changes the storefront only. Cart contents, prices and checkout are shared by every group.
- One theme test per store. Other tests on the same pages make the result harder to read.
- Products assigned to an alternate template need a template with the same name in every tested theme.
- Shoppers who don't allow analytics see the live theme and aren't counted.
- Theme tests are new: checked in a real browser against a store that answers like Shopify, not yet run on a live store.
Questions
Does Benchmyrk edit my theme?
No. You duplicate and edit the theme in Shopify. Benchmyrk only reads your themes to check they exist, are unpublished and have the app embed turned on.
Which Shopify plan do I need?
Any plan with an online store. Theme tests use Shopify's theme preview, not Rollouts, so they don't depend on the plans Rollouts experiments need.
Is a theme test bad for SEO?
Search engine crawlers always get your live theme, and the canonical URL of every page stays the same.
How do I keep the winning theme?
End the test and publish the winning theme in Shopify, in Online Store, Themes. Benchmyrk doesn't ask for permission to publish themes, so it can't do this for you.
Can I change the test theme while the test runs?
You can, but shoppers before and after the edit saw different themes, which muddies the result. Finish your edits, check the theme with a preview link, then start the test.
Try the new theme before you publish it.
We'll go through a theme test with you: preparing the copy, the launch checks and how the result is judged. Or install the Shopify app and set one up.