Orders arrive after a campaign, and the report attributes them to the email. However, some customers might have bought without the message because of habit, search, advertising or a conversation with a manager.
A holdout group helps estimate the incremental effect of a specific email communication. Define the comparison in advance, keep conditions comparable and account for purchases in both groups using the same rules.
Define exactly what you are testing
Identify the intervention: one promotional email, a reminder series or an offer to a particular segment. The holdout group does not receive this specific communication during the agreed period.
Retain necessary service messages, documents and support for every customer. Do not withhold essential information from part of the audience to test marketing.
Separate the question of an email's contribution from the question of the best wording. Comparing two emails measures their difference; a group without the tested campaign measures the effect of adding it to existing communication. The broader logic is covered in the article on checking advertising's contribution.
Allocate eligible customers before sending
First apply participation and exclusion rules: permission to send, segment, current purchase status and data availability. Then randomly assign participants to the groups.
Do not assign customers who “usually do not buy” to the holdout or compare recipients with people who unsubscribed. These groups already differ in behaviour or communication eligibility.
The allocation unit may be a customer, household or B2B company. If one buyer has several contacts, group them so related addresses do not end up in different groups.
Agree other messages and the measurement window
Check whether the holdout could receive the same promotion from another email flow. If you are testing a series, the exclusion must cover the whole agreed workflow, including resends.
Keep other channels comparable or explicitly include the differences in the research question. Sales within an existing channel mix and sales without any marketing contacts are different questions.
Set the observation period, primary metric and return accounting rules in advance. For an abandoned cart sequence, repeated entry into the workflow and orders completed without an email click are particularly important.
Measure results using the original groups
Count all eligible purchases by participants during the agreed period, regardless of email clicks. Comparing only openers or clickers reintroduces behavioural selection and does not measure the same effect.
Preserve the original allocation and primary denominator. If some emails are not delivered, record this as an implementation limitation instead of retrospectively removing those customers to improve the result.
Report participants, buyers, orders and returns. Define in advance whether you count unique buyers or all orders; assess financial effect using contribution margin and incremental expenses alongside revenue.
Review the calculation alongside uncertainty
Illustrative example. Two randomly formed groups of 500 customers generate 40 buyers with the email and 30 without it during the same period. Purchase rates are 8% and 6%, a difference of 2 percentage points. For the 500-recipient group, that corresponds to an estimate of ten additional buyers.
Assume each made one order, contribution after variable costs and the discount was €30, and incremental campaign expenses were €200. The illustrative contribution estimate is 10 × 30 − 200 = €100. The calculation does not account for uncertainty in the difference or establish future campaign profitability.
Assess the range of possible effects given the sample. Do not declare growth proven solely because the difference is positive or generalise one segment's result to the whole list. Insufficient observations may leave no convincing conclusion.
An incremental effect test record
- Tested communication and the decision based on results.
- Eligible participants, exclusions and allocation unit.
- Random assignment and retained original groups.
- Rules for other channels and preventing overlap.
- Period, metric, denominator and return accounting.
- Contribution margin, expenses and uncertainty assessment.
- Stopping conditions and limits of applicability.
For email marketing, prepare purchase history, workflow rules and participant exclusion capabilities. First check that the system supports the chosen experiment and the data can link outcomes to the original groups.