A/B testing with little traffic: what still works
An A/B test needs enough cases to separate chance from effect. At 300 visitors a month that threshold is unreachable — every result is then a coin toss. Five other approaches still deliver usable indications.
In short
- Below roughly 1,000 target actions per variant, A/B results cannot be separated from chance — so on small websites, practically never.
- Five approaches work regardless: watching users, the four-second test, before-and-after across quarters, drop-off points, and the question in the first conversation.
- With little traffic, large changes pay off rather than small ones — a new layout, not a different button colour.
- If you want to A/B test anyway, test on email lists rather than on the website: there the numbers are often sufficient.
Why the arithmetic does not allow it
A test is meant to show whether a change has an effect or whether the difference is chance. How many cases that takes depends on two things: the baseline rate and the size of the expected difference.
An example makes the order of magnitude plain: with a baseline of 3 per cent and a hoped-for rise to 3.6 per cent — a 20 per cent relative improvement — you need several thousand visitors per variant to separate the difference reliably from chance. At 300 visitors a month the test would be informative after years, and by then the offering has changed.
The five approaches that work
1. Watch five people
From the audience, with a task, without explanation. Where they hesitate is where the fault lies. Five people uncover the bulk of the coarse problems.
Effort: 2 hours · Informativeness: high for usability
2. The four-second test
Show the page, close it after four seconds, ask: what was it about, who is it for, what should you do? Anyone who cannot answer that has not understood the top of the page.
Effort: 30 minutes · Informativeness: high for the entry point
3. Before-and-after across quarters
One large change, then measure for three months and compare with the three months before. Imprecise, because other influences are at work — but usable for large changes.
Effort: low · Informativeness: medium
4. Measure drop-off points
Where does scrolling end, where does the form get abandoned? It does not show what would be better — but very precisely where the problem sits.
Effort: one-off setup · Informativeness: high for locating
5. The question in the first conversation
"What made you write to us — and what nearly stopped you?" The second part is the valuable one.
Effort: none · Informativeness: high, if asked regularly
Worth knowing
With little traffic, large changes pay off rather than small ones — and for arithmetic reasons: the larger the actual difference, the fewer cases it takes to detect it.
A different button colour shifts the rate by a few per cent and is in principle undetectable at small numbers. A completely different page layout can double the rate — and that is visible even at 300 visitors. Anyone with little traffic should change more boldly, not more cautiously.
Where A/B tests do work for small companies
| Place | Feasible? | Note |
|---|---|---|
| Subject lines in email sends | yes | roughly usable from a few hundred recipients |
| Ad copy | yes | many impressions, fast feedback |
| Home page | rarely | too few target actions |
| Contact form | no | too few submissions |
| Pricing page | no | too few cases, too much spread |
The most honest way to handle small numbers is not to present the decision as a test result. If a change seems sensible for good reasons — a shorter form, a clearer top of page, a price rather than none — make it and write down what you are basing the expectation on.
After three months you look at whether the inflow figure has moved, and you know: that is an indication, not a proof. That honesty is worth more than a test whose result is a coin toss — because it stops a random number becoming the basis of the next decision.
Help me decide how I can assess a change to our website. Our figures: - Visitors per month on the page concerned: [number] - Target actions per month on this page: [number] - What I want to change: [description] - How large I expect the effect to be: [estimate] Tasks: 1. Tell me whether an A/B test would be informative at our figures. Work with a rule of thumb and show the reasoning. If not, say so clearly. 2. From the five alternatives – watching users, the four-second test, before-and-after across quarters, drop-off points, the question in the first conversation – recommend the two that fit my change. Justify it. 3. Assess whether my planned change is large enough to become visible at all at our figures. If not, propose a bolder version. 4. For the four-second test, phrase the three questions I should ask. 5. Phrase a sentence for our report that says honestly what the result will show and what it will not. Do not invent industry benchmarks.
In closing
At small numbers the classic A/B test is not a tool but a chance machine with a scientific veneer. Use it anyway and you make decisions on the basis of noise.
What works: watching five people, the four-second test, drop-off points, the question in the first conversation — and bold rather than cautious changes, because large differences become visible even with few cases.
Common questions
From what point are A/B tests informative?
As an order of magnitude it takes several thousand visitors per variant when a typical baseline of a few per cent is to be improved by around a fifth. At 300 visitors a month such a test would only be informative after years — and by then the offering has changed.
What do you do instead?
Five approaches: watch five people from the audience using it, run the four-second test on the top of the page, do a before-and-after across quarters for large changes, measure drop-off points, and ask in the first conversation what nearly stopped them.
Why should you change more boldly with little traffic?
Because larger actual differences need fewer cases to be detected. A different button colour is in principle undetectable at small numbers; a completely different page layout can double the rate — and that is visible even at a few hundred visitors.
Where do A/B tests work for small companies too?
On subject lines in email sends, where a few hundred recipients already give rough indications, and on ad copy with many impressions. On the website — home page, contact form, pricing page — the target actions are as a rule not enough.
Can you stop a test when one variant is leading?
No. At small numbers the lead changes regularly; stop at the desired result and you will always find one. That is then not a test but a confirmation of your own assumption — and the random number becomes the basis of the next decision.
Marketing that sets itself up
The Studio Engine beta is live. Claim your spot and help shape it from the start.
Join the beta →