Test several changes at once, in every combination.
Change the headline, the image and the button together. Results show which version of each element wins and whether two of them interact.
| Headline × image | Original photo | Photo on the road |
|---|---|---|
| Original title | Combination 1The control, 25% of visitors | Combination 225% of visitors |
| Benefit-led title | Combination 325% of visitors | Combination 425% of visitors |
Results read each element on its own (which headline wins, averaged over both images) and check whether the two interact.
How a multivariate test is built
- 01
Pick the elements
Two or three elements on the page, such as the headline, the main image and the button.
- 02
Add versions
Two to four versions of each, the first being the page as it is. You edit versions, never combinations.
- 03
Every combination runs
Benchmyrk builds every combination, up to 16, and splits traffic evenly between them.
- 04
Read each element
Results show the winning version of each element and flag elements that interact.
Answers per element, with the interactions checked
Each version is compared with the original version of the same element, averaged over every combination that shows it. A separate check looks at each pair of elements to see whether they work differently together than apart, and says so in plain words.
- All comparisons for a goal are Holm-corrected as one family, and verdicts stay peeking-safe until every combination has its planned visitors.
- The per-combination table is still there, next to the per-element findings.
- Setup shows the traffic the test needs before you launch it.
Good to know
- Every combination is a real variant, so traffic needs grow fast: interactions need roughly four times the traffic of a single change of the same size.
- Two elements can't change the same thing on the page; the launch check blocks overlaps.
- Auto-allocation and CUPED aren't available for multivariate tests.
- Elements and versions are fixed once a combination has visitors.
Questions
Multivariate test or several A/B tests?
A sequence of A/B tests can't show whether two changes help or hurt each other. A multivariate test can, at the cost of more traffic. With modest traffic, test the biggest change first as an A/B test.
How much traffic do I need?
It depends on your conversion rate and the size of change you want to detect. Setup estimates it from the page's recent traffic before you launch, and results say when there isn't enough.
Have more than one idea for the same page?
We'll look at whether your traffic supports a multivariate test, or which A/B test to run first. Or install the Shopify app and try one.