Shopify A/B Test Sample Size (With a Free Template)

Key takeaways

• Your Shopify A/B test sample size is the number of visitors each variant needs before the result means anything. Call the winner too early and you are reading noise.
• Four inputs drive it: your baseline conversion rate, the minimum detectable effect you want to catch, the significance level (usually 95%), and the statistical power (usually 80%).
• A lower baseline conversion rate and a smaller detectable effect both push the required sample up fast. Small changes on low-traffic stores can need tens of thousands of visitors per variant.
• Use a free calculator like Evan Miller’s to get the number, then plan the test in full weeks of traffic so day-of-week patterns don’t skew it.
• Fudge ships your test variant as native theme code in minutes, so the slow part is the traffic, not the build.

Your Shopify A/B test sample size is the single number that decides whether a test can tell you anything. It is how many visitors each version of the page needs to see before the difference between them is real rather than random.

Get it wrong and the whole exercise is theatre. This guide covers why sample size matters, the four inputs that set it, the formula and a worked example, a spreadsheet template you can copy, and the low-traffic reality most Shopify stores run into.

Why you can trust us

We have worked in the Shopify space for over four years and helped hundreds of brands improve their storefronts. We build Fudge, an AI storefront editor with a 5.0 rating on the Shopify App Store, so we sit on the theme layer where these tests actually run and see how often they get called before the data supports it.

Why sample size matters

An A/B test compares two versions of a page and measures which converts better. The catch is that conversion is noisy. Flip a fair coin 20 times and you rarely get exactly ten heads. Small samples swing around the true rate.

Sample size is the guard against that noise. Below the threshold, a variant can look like a 15% winner purely by chance.

Two failure modes come from ignoring it.

Underpowered tests. If you stop with too few visitors, a real improvement can hide inside the noise and read as “no difference.” You conclude the change didn’t work, ship nothing, and leave the uplift on the table.

Peeking and early stopping. The more tempting mistake is checking the dashboard daily and stopping the moment significance flashes green. Every extra peek is another chance for random variation to cross the line, so repeated checking inflates the real false-positive rate far above the 5% you think you are running.¹ The fix is to fix the sample size in advance and wait for it.

This is the difference between running tests and running a testing programme. For the wider picture, see our guide to Shopify conversion testing.

The four inputs to a sample-size calculation

Every calculator asks for the same four things. Understand these and the number stops being a black box.

Baseline conversion rate

This is your starting point, pulled from your own analytics for the exact page and audience you plan to test. A product page tested on mobile traffic has a different baseline than the same page on desktop, so scope it to the test.

Minimum detectable effect

The minimum detectable effect is the smallest lift worth detecting, usually written as a relative percentage. An MDE of 20% on a 3% baseline means you want to reliably spot a move to 3.6%.

This is the input people misjudge most. A smaller MDE means a more sensitive test, and sensitivity is expensive. Halving the effect you want to catch roughly quadruples the visitors you need.

Significance and power

Significance (95%) sets how sure you want to be that a detected win is not random. Power (80%) sets how likely you are to catch a win that genuinely exists. Both are conventions, not laws, but they are the standard defaults and a reasonable starting point for most stores.

The sample-size formula and a worked example

For a test comparing two conversion rates, the standard approximation for visitors needed per variant is:

n = (z_alpha + z_beta)^2 x [ p1(1-p1) + p2(1-p2) ] / (p2 - p1)^2

where:
  p1        = baseline conversion rate
  p2        = baseline x (1 + MDE)
  z_alpha   = 1.96   (two-sided, 95% significance)
  z_beta    = 0.84   (80% power)

Work a concrete case. Baseline conversion rate of 3%, you want to catch a 20% relative lift, at 95% significance and 80% power.

• p1 = 0.03, so p2 = 0.03 x 1.20 = 0.036
• (1.96 + 0.84)^2 = 2.8^2 = 7.84
• p1(1-p1) = 0.0291, and p2(1-p2) = 0.0347, summing to about 0.0638
• (p2 - p1)^2 = 0.006^2 = 0.000036

Putting those together gives 7.84 x 0.0638 / 0.000036, which is about 13,900 visitors per variant, or roughly 27,800 in total.

That is the number the calculator returns. The value of doing it by hand once is seeing exactly what moves it.

How baseline and MDE change the number

The two drivers are your baseline conversion rate and the effect you want to catch. Both can blow the sample size up quickly. The table below runs the same formula at 95% significance and 80% power across a few scenarios.

Two patterns fall out of the math.

Smaller effects cost far more. Dropping the MDE from 20% to 10% on a 3% baseline takes the requirement from about 13,900 to 53,100 per variant. Chasing a tiny lift is the fastest way to design a test you can never finish.

A lower baseline is punishing. At a 1% baseline, even a 20% lift needs about 42,600 per variant, three times the 3% case, because rare events carry more relative noise.

The final column translates the count into calendar time. It assumes 4,000 test-eligible visitors per week, split across both variants, rounded up to full weeks. That column is where most Shopify test plans meet reality.

Build your own sample-size template

You don’t need a fancy tool to plan this. A spreadsheet with the formula above works and lets you sanity-check any calculator’s output.

Set up these cells:

• Baseline CVR (input): your current rate, for example 0.03
• MDE (input): relative uplift, for example 0.20
• p2 (calculated): baseline * (1 + MDE)
• z_alpha (constant): 1.96 for 95% two-sided
• z_beta (constant): 0.84 for 80% power
• n per variant (calculated): (z_alpha + z_beta)^2 * (p1*(1-p1) + p2*(1-p2)) / (p2 - p1)^2

Then add a weekly traffic input and a duration cell that divides 2 * n by your weekly test-eligible visitors. Now you can change the MDE and watch the runtime move before you commit to a test.

Keep a second sheet listing every test: hypothesis, the page, start and end dates, planned sample size, and the result. That log stops you re-running settled questions and makes your programme auditable. It pairs naturally with a broader Shopify CRO plan.

Free A/B test calculators worth using

If you would rather not maintain a spreadsheet, two well-known free tools do the same job.

Evan Miller’s sample size calculator takes your baseline rate and the MDE and returns the sample per variant, using the same significance and power inputs.² It is a long-standing reference in the testing community and a good cross-check against your own sheet.

Optimizely’s sample size calculator is another free, browser-based option that asks for the same inputs and returns a per-variant number. Either tool will land close to the figures in the table above when fed the same assumptions.

Whichever you use, feed it your real baseline and a realistic MDE. A calculator fed optimistic inputs returns an optimistic, and usually unreachable, number.

The Shopify low-traffic reality

Here is the hard truth for smaller stores. The scenario table shows that detecting a modest lift on a typical conversion rate can need tens of thousands of visitors per variant. Plenty of Shopify stores don’t see that in a reasonable window.

That does not mean testing is pointless. It means you have to test smart.

• Test bigger swings. A full page redesign or a new offer moves the needle more than a button-colour tweak, so it needs a smaller sample to detect. Reserve fine-grained tests for high-traffic stores.
• Test higher-traffic pages. Run experiments on the pages the most sessions touch, such as the home page, main collection, and top product page, rather than a low-traffic corner.
• Run full weeks. Always run in whole-week blocks. Weekend and weekday shoppers behave differently, and a test that starts on a Monday and ends on a Thursday bakes in that bias.
• Accept fewer, longer tests. A low-traffic store that runs one clean four-week test beats one that starts and abandons five underpowered ones.
• Lean on qualitative signals too. Heatmaps, session recordings, and surveys don’t need statistical power and can guide you toward the big swings that are worth an A/B test.

For tools that support this on a Shopify budget, see our roundup of the best Shopify apps for CRO.

Where Fudge fits

Sample size sets how long a test must run. The other half of the equation is how long it takes to build the variant in the first place, and that is where most tests stall before they start.

Fudge is an AI storefront editor that writes native theme code. You describe the variant you want to test, a new product-page layout, a different hero, a restructured offer, and it writes the Liquid, CSS, and JavaScript straight into your theme. No developer queue, no widget.

Because the output is real theme code, the variant behaves like the rest of your store during the test, with no third-party script slowing the page and skewing the result. You can spin up the challenger, point your testing tool at it, and spend your effort on the part that actually takes time: waiting for the sample to fill. The same holds for anything else you build with the Shopify store editor.

FAQ