Your website gets few enquiries, and you want to try a new headline, form or opening section. An A/B test seems like a convenient way to choose the better version. With few visitors, however, a difference of one or two enquiries can look impressive as a percentage while being no more than random variation.
Before starting an experiment, assess whether it can deliver a useful answer within an acceptable timeframe. If there is not enough data, you can keep improving the website through other methods, provided you understand what conclusions they support.
An experiment randomly assigns eligible users to the current version or a modified version. Results are assessed against a metric selected in advance.
Showing the old website one week and the new one the next is a different comparison. Advertising, seasonality, audience composition and sales team performance can affect the outcome. The GOV.UK guide to A/B testing describes random allocation and the need to plan an experiment before it starts.
A good hypothesis connects a problem, a change and an expected effect: “Visitors do not understand what happens after an enquiry. We will explain the next step and test the rate of confirmed enquiries.”
“Make it look better” does not specify what decision the test should inform.
Look at the number of users who will actually see the change. Total website traffic does not help if the test runs on a rarely visited page.
Planning requires:
Use these conditions to calculate the sample size for the chosen analysis method. There is no universal rule that “a thousand visitors is enough”. Small improvements are generally harder to detect than large ones; infrequent enquiries also make assessment more difficult.
If collecting the required sample takes longer than the business can keep the experiment's conditions stable, the test may be impractical. Lowering the evidence threshold to get a quick winner increases the risk of mistaking random variation for a result.
Suppose 200 eligible users visit the page. With equal allocation, 100 people see each version. The first receives two enquiries, while the second receives three.
Conversion rates are 2% and 3%. The relative difference is 50%, but it rests on just one additional enquiry. These figures alone do not establish that the second version is better.
This example is illustrative and does not set a minimum sample size. It shows why reports need the number of observations and enquiries, not just percentage growth.
Choose one primary metric and additional safeguards. For example, an increase in form submissions should not be accompanied by more spam or lower enquiry quality.
Check event tracking, random allocation and whether users consistently see their assigned version. Define how to exclude employees and test enquiries without discarding inconvenient results after the experiment ends.
Set duration and stopping rules in advance. For a conventional fixed-sample test, checking daily and stopping at the first favourable result increases the risk of error. If you use sequential analysis, the stopping rules must match that method.
Do not change the offer, advertising traffic and enquiry handling process at the same time without accounting for those changes. If the result is inconclusive, record the uncertainty: a lack of demonstrated advantage does not mean the versions are identical.
Ask people from your target audience to find the relevant service, understand its terms and submit an enquiry. Observe where they hesitate or make mistakes without telling them what to do next.
This helps identify obstacles, but does not measure the percentage increase in future conversion rates.
Find out what information people sought before buying, what they compared and what concerned them. Their answers can produce more meaningful hypotheses than rearranging buttons to suit the team's preferences.
Compare visits, attempts to complete the form, successful submissions and qualification outcomes. Fix broken functionality first: a button that does not work, a form error or an enquiry that never reaches the recipient. Fixing a confirmed malfunction does not require choosing a winner in an A/B test.
Session recordings, where their use is permitted and data protection is configured, can help reveal behaviour on a page. They do not automatically explain why people act as they do; observations need further checking.
Make justified changes in small steps, record the date and monitor enquiries. Account for changes in traffic and the offer. This monitoring is useful for management, but cannot be presented as proof of the effect established by a randomised experiment.
With limited traffic, a useful starting point is a website audit: identify problems, prioritise changes and choose how to assess the outcome. Other reasons for lost enquiries are covered in “Your website gets traffic but no enquiries”.