Benchmyrk
Book a demo

Changelog

What's new in Benchmyrk.

Every release in plain language, newest first. Customers also find these notes, with the status of every feature, under What’s new in the app.

RSS feed

  1. Guides, a glossary and a sample size calculator on the website

    • New guides on the website: how to read an A/B test result, how to plan sample size and test duration, price testing on Shopify without Plus, and what to do when a test's traffic split looks wrong. The worked examples are computed with the same statistics engine as your results.
    • A free sample size calculator at benchmyrk.com/tools/sample-size-calculator. It uses the same formula as the new-experiment wizard, so the numbers match what the app plans.
    • An A/B testing glossary with plain definitions, a public changelog with an RSS feed, and a docs page that links every setup guide in one place.

    Fixes from end-to-end checks on a live store

    • Shopify: a shopper in a test who clicked Add to cart within a second of the page loading could lose the item, because Benchmyrk's note on the cart (which variant they saw) made a second, empty cart at the same moment. Benchmyrk now waits for the shopper's own cart before adding its note. Reaches your store with the next app update.

    AI now switches between free models when one is busy

    • When the AI model a job starts on is busy or unavailable, Benchmyrk waits a moment and tries it once more, then moves on to the next free model on the list, so copy rewrites, variants, audits and the other AI jobs keep working.
    • A model that just said it was busy is skipped for a few minutes, so new jobs start on one that is answering.
    • If every free model is busy, the job stops within about a minute and a half (four minutes for deep audits) with: “All free AI models are busy right now. Try again in a few minutes.”
    • Owners and admins see each model on the list, its place in the order and whether it is working in Settings › Brand & AI.

  2. A new logo and a new website

    • Benchmyrk has its new logo, the "Step": the control and the challenger that steps up beside it. You'll see it in the app, in our emails and on the Chrome extension.
    • The public website is rebuilt with pricing, a Shopify page, security and privacy details, and a form to book a demo.
    • Visitors in the EEA, the UK and Switzerland now choose whether our website may use analytics cookies. Your storefronts are not affected.

    App impact tests

    • Find out what an app is worth: pick an app from your live Shopify theme (app embeds and app blocks are listed for you), choose how many visitors go without it, and Benchmyrk compares revenue per visitor, average order value and conversion.
    • Visitors in the app-off group get your store without the app: its scripts are stopped before they run and its elements are hidden. Everyone else sees no change.
    • Before launch, Benchmyrk checks the app is still on in your live theme and that its elements are on your example page.
    • Apps that work in checkout, in emails or on Shopify's servers can't be held out this way; the guide explains what can.

    Hydrogen and headless Shopify storefronts

    • Run experiments and feature flags on a Hydrogen storefront: your route loaders decide each visitor's variant on the server, and the page arrives already showing it, so nothing flickers. Visitors get the same variant as in Benchmyrk's storefront script, because both use the same visitor cookie and the same bucketing.
    • Purchases still count: a helper writes the visitor into the cart, and Shopify's order webhook attributes the order as it does on your theme.
    • Headless storefronts load Benchmyrk's script from Benchmyrk and send events straight to it. Add your storefront's address under Site settings, Headless storefront origins.
    • Price, template, offer and shipping tests keep running on your Shopify theme only; the guide lists what works headless.

    Multivariate tests

    • Test several elements of one page at once, for example the headline and the button: pick 2 or 3 elements, give each 2 to 4 versions, and every combination runs as its own arm (up to 16).
    • Before you create one, Benchmyrk shows how many visitors each combination needs and how long that takes at your page’s recent traffic, and warns you when the page doesn’t get enough.
    • In the visual editor you edit each version of each element on its own; Benchmyrk builds the combinations for you and won’t launch a test where two elements change the same thing.
    • Results show, for each element, which version beats the original, and whether two elements work better or worse together. Every combination is still listed below, and the AI explanation follows the same findings.

    Measured page-speed cost, and faster prices in price tests

    • We measured what Benchmyrk costs a mobile product page in a repeatable lab test: no layout shift, no long tasks, and content in a test hidden for about 10 ms while the change applies. On a first visit the page’s first paint came about 0.15 seconds later (the script downloads before the page shows); on repeat visits about 0.03 seconds.
    • Price tests: shoppers in the control group now see your prices immediately. Before, prices stayed hidden until the page finished loading for everyone in a price test.

    Target your Klaviyo lists and segments

    • Connect Klaviyo in Settings › Integrations with a read-only private API key, then choose the lists and segments you want to target. Benchmyrk syncs them every hour.
    • New targeting rule on Shopify stores: “Klaviyo list or segment”. It matches customers signed in to your store whose email is in the list or segment.
    • Benchmyrk keeps members as scrambled (hashed) emails only, never the emails themselves, and never sends them to your storefront. When your site’s consent setting follows Shopify’s cookie consent, the rule only applies to shoppers who allowed marketing.
    • Disconnecting deletes the key and every synced member at once.

    Shopify Markets: price tests in every currency, currency targeting

    • Price tests whose arms are all percent changes now include shoppers paying in other currencies. They get the same percentage off their market’s own price, on the page and at checkout.
    • A price test with a fixed-price arm still runs in your store currency only, and the Prices tab now says so before you launch.
    • New targeting rule on Shopify: the shopper’s currency (for example EUR or GBP), next to Shopify market and location.
    • Revenue, order value and profit stay in your store currency, using the amounts Shopify records for each order in that currency, never an exchange rate of ours.

    BestConvert is now Benchmyrk

    • New name, same product: BestConvert is now Benchmyrk. Your experiments, results, installs and settings carry over unchanged.
    • Nothing to change on your store. The script tag and Shopify app embed you already installed keep working as they are.

    Offer tests and shipping tests

    • Offer tests (Shopify): give each arm its own checkout offer, such as 10% off, $10 off from $50, bigger discounts for bigger orders, or buy 2 get 1 free, and compare revenue and profit per visitor with the discount already subtracted. Each arm shows a banner, and the launch check makes sure the banner names the same percent and amounts checkout gives.
    • Shipping tests (Shopify): test free shipping from a minimum order, or a percent or amount off shipping, against your normal rates. Shopify checkout gives each shopper their arm’s shipping discount, and you can keep express rates at full price.
    • Offer and shipping test results show, per arm, how many orders used the offer, Benchmyrk’s discount per order, all discounts per order and the shipping charged per order.
    • Pausing an offer or shipping test removes the banner and stops checkout giving the offer. If Benchmyrk’s discount is turned off in Shopify, the test comes off your store within about ten minutes and owners and admins get a notification.

    Feature flags and experiments on your own server

    • Feature flags: serve a value to your own server code (on/off, text, number or JSON) instead of changing the page. Choose how many visitors are in the flag, and optionally keep half of them on the old value to see what the new one changes. Start one from Experiments › Feature flag.
    • Server SDK: developers can read your experiments and flags from their server, edge functions or a Hydrogen storefront, and report what visitors saw. A visitor gets the same variant on the server as in the storefront script. Owners and admins create a token per store in Settings › API tokens › Server SDK; tokens are shown once, can be revoked, and never work in a browser.
    • The SDK package isn’t published to npm yet.

    Template tests and theme previews

    • Template tests on Shopify: compare page templates of your live theme, for example your product template against one with reviews higher up. Shoppers in the test version get the same page with Shopify's ?view= before the original is shown, search engines keep the normal page as canonical, and results work like any other test.
    • Benchmyrk checks your templates at launch, when you publish a theme and every 10 minutes while a test runs. If a template disappears, the test stops on your store and you get a notification.
    • Theme previews: see your unpublished themes with Shopify preview links and a warning when the Benchmyrk app embed is off in one, before you publish it. Theme tests stay preview-only: shoppers are never split across themes.

    Discussions on experiments

    • Discussion tab on every experiment: start a thread on the experiment, its results, one variant or one change from the editor; reply, edit or delete your own comments, and resolve or reopen a thread.
    • Mention a teammate with @ to notify them in the bell and in an hourly email digest (Settings lets each person choose digest or off). Only people who can see the thread are notified.
    • Agencies can add internal notes that their clients never see; threads shared with the client stay visible to them.
    • The experiments list shows open discussion counts, and the API and MCP server can read and add comments.

    Google Analytics 4 and clearer AI model problems

    • Google Analytics 4: send each visitor’s first exposure to an experiment, and optionally the conversions Benchmyrk counts, to your GA4 property from our server (Settings › Integrations, owners and admins). Then compare variants in GA4 with segments and audiences. Sending needs an opt-in to use Benchmyrk’s visitor id as the GA4 client id, until the GA client id is forwarded.
    • AI models: when the AI service refuses the model configured on this server, AI now says “The configured AI model isn’t available right now” instead of blaming the account, and never shows the service’s own wording. When the server’s deep model is refused and its quick model is a different one (both free or both paid), deep audits, experiment drafts and results explanations retry once on the quick model.
    • Settings › Brand & AI lists the AI models on this server with an hourly check (working, unavailable or couldn’t check). Workspace owners get a notification when a model their workspace uses stops working.

    Auto-allocate (multi-armed bandit) tests

    • New experiment mode for promotions and short campaigns: Auto-allocate starts with an even split for the first 24 hours, then sends more visitors to the variant most likely to be best, updated hourly. Every variant keeps at least 10% of visitors and returning visitors keep the variant they first saw.
    • Results for auto-allocate tests show the traffic split over time and each variant’s estimate, chance of being best and expected loss, instead of p-values. A winner is called only when it’s clear however often you check, and Stop and pick winner completes the test with the leader selected.

    Price tests, templates, What’s live and lower running costs

    • Price tests on Shopify: test product prices with the same price shown on the page and charged at checkout, through a Shopify discount that works on every plan. Pausing the test restores normal prices straight away. Results show revenue and profit per visitor, units sold and average price paid.
    • Templates: start from a reviewed test idea (trust badges, free-shipping banner, sticky add to cart, delivery promise and more), fill in your store’s details and get a draft in the editor. Save your own tests as templates for the team.
    • What’s live: a readable page showing exactly what your store runs now, what changed since the last publish and anything held back, with the raw configuration under Advanced.
    • Usage: Settings › Usage & budgets shows each store’s sessions and events this month against its plan. Behind the scenes each tracked event now costs far fewer database operations, and the Shopify pixel sends fewer page views when no test needs them.

    Measurement health, location and segment targeting, analytics

    • Measurement health: the pixel, the theme app embed, Shopify webhooks and the storefront connection are checked when you install, each time you open the app and every hour, and repaired automatically where possible. If page views keep arriving but the pixel goes quiet, you get one notification with the exact fix.
    • Checkout events are counted once per checkout and purchases once per order, however often Shopify’s checkout reports a step.
    • Test event delivery opens your store with a private checklist that ticks off page view, add to cart, checkout and purchase as each event arrives. Your test orders are marked as tests and never count in results.
    • Targeting: Shopify customer segments (for signed-in customers), country, region and city include and leave-out lists, saved locations, Shopify Markets and time of day. The setup wizard has a More conditions section.
    • Results show how each variant developed day by day, with confidence bands, the traffic split and breakdowns by device, new or returning visitor, source and country. A new Analytics page covers traffic, revenue, the shopping funnel and a live view of visitors right now.
    • Security fixes from a review of the new features, including stricter checks on Shopify storefront requests and per-store rate limits.

  3. CUPED and outgoing webhooks

    • Variance reduction (CUPED) for purchase, revenue and profit goals: results can use each visitor’s orders from the 30 days before they entered the test to narrow the confidence intervals. It is off by default, switched on per experiment or as a workspace default, and fixed once the test starts. When too few visitors have history, results stay unadjusted and say why.
    • Outgoing webhooks: send signed events to Slack, a sheet or your own tools when a test launches, pauses, resumes, completes, finds a winner, trips a guardrail, or when a launch approval is requested or decided. Failed deliveries retry for up to 12 hours; Settings › Webhooks shows every delivery and can send a test event.
    • AI features in the hosted app run on OpenRouter free models for now.

    Profit metrics

    • Tests on Shopify stores can be judged on gross profit per visitor: revenue after tax, product cost, payment fees and shipping cost.
    • Turn profit on in Site settings. Unit costs come from Shopify, and each order keeps the product cost it had when it came in, so later cost edits don't rewrite past results.
    • Payment fees and shipping costs are set per site and applied to every order in a test alike.
    • When product costs are known for less than 90% of a test's order value, profit is shown but no winner is called.
    • Not available on script-tag sites yet.

    Connect Claude to your workspace (MCP)

    • Benchmyrk has an MCP server at /mcp. Add it to Claude (or any MCP client) with an API token to list sites, read experiments and results, create draft experiments and variants, request launch approval, and pause or resume tests.
    • Changes made this way go through the same approval, billing and permission checks as the app. JavaScript and redirect changes are never accepted, and experiments can be resumed but not started from MCP.
    • API tokens can now be created read-only. Settings › API tokens shows how to connect.

    Shopify billing, privacy requests and production AI

    • Your plan is billed through Shopify, and Benchmyrk always takes the current plan from Shopify. Stores without an active plan can't start, schedule or resume experiments, and installing the app takes the owner to plan selection.
    • Customer data requests from Shopify now appear in Site settings under Privacy requests, with a JSON export and a Mark answered button. Owners and admins get a notification and an email that leaves out customer details.
    • Tracked page addresses keep only the domain, the path and the five UTM tags; campaign tags that contain an email address are dropped. Session recordings follow the same rule.
    • When Shopify asks us to erase a store's data, the workspace created at install is deleted too if that store was its only site.
    • AI features can run on the Anthropic API in production. Their cost still counts against your AI budget, and if the provider's spending limit is reached the app says AI is paused.
    • AI tasks no longer fail when the app restarts or runs on more than one server.

    Editor, approvals and design polish

    • The on-page editor downloads less up front (224 KB instead of 249 KB compressed); rarely used panels load when you open them.
    • Device frames show only the changes meant for that device, and custom CSS can be set per device.
    • Deep page audits are available in the Chrome extension and on the AI insights page.
    • Layers can name page sections with AI when you ask it to; the names are saved per page.
    • Emails for launch decisions and stage approvals, pending launch approvals in the agency review queue, and Request approval on each journey branch.
    • New workspace setting, Allow AI to create draft variants, enforced on the server for every AI task that drafts or proposes changes.
    • Access requests from Shopify staff pre-fill the invite form only with addresses Shopify has verified.
    • Review & permissions, deployment approval, proposal review, the AI audit panel, Shopify access, Help and the storefront password panel now match the reference designs.

    Fixes

    • Decision-due reminders now reach every live or recently finished experiment. Before, once there were more than 300 across all workspaces, newer experiments could miss them.

  4. Email delivery, Help page and replay privacy

    • Emails are sent through Resend in production.
    • New Help page with a contact form. Error screens show a reference ID you can include so we can find what went wrong.
    • Sign-up no longer reveals whether an email address already has an account.
    • Session recordings show page text by default. Typed values, form fields, account, checkout and order pages, and anything marked data-bc-mask stay masked. Turning off masking of sensitive pages needs confirmation and is recorded in the audit log.
    • Personalization journeys: Complete & remove ends a paused branch's experiment, keeps its results and removes the branch.
    • The Integrations page shows the health of your Shopify webhooks.

    Session recordings, invitations and notifications

    • Session recordings under Behavior › Sessions.
    • Invite teammates by email; they accept from the Invitations page.
    • Email confirmation and password reset.
    • A notification bell for things that need your attention.
    • Exclusion groups keep experiments that run at the same time from reaching the same visitors.
    • Results can be broken down by segment.
    • Decision-due reminders and AI budget alerts.
    • Upload a logo for your workspace.

    Studio, audiences, personalization and the agency portfolio

    • Visual studio for AI-generated page variations.
    • Audiences, Personalization journeys, Behavior heatmaps, the Learning library and QA Lab.
    • New experiment steps: proposal review, launch review, deployment plan and decision.
    • AI activity: every AI task with its status and cost.
    • Agency portfolio: clients, review queue, reports and team.

  5. The new Benchmyrk app

    • Sign up, then add a Shopify store through the app, or any website with one script tag.
    • Shopify app embed and web pixel: install without theme edits and track checkout events.
    • Create experiments, edit variants, share preview links, launch, and read results with Bayesian and classic statistics.
    • AI insights, goals, an install check with live events, site and workspace settings, and API tokens.
    • Pair the Chrome extension with your account.

See what it does on your store.

We'll walk through the product on demo data. Or start the free trial on Shopify.

Book a demoStart free trial