# What is a holdout group? Measuring what personalization adds

> A holdout group is a random slice of visitors who see no personalization. Comparing them with everyone else shows what your changes really added.

Measurement · Published 2026-10-22

A holdout group is a random slice of your visitors who see none of your personalization: no popups, no banners, no changed content. Everyone else gets your rules as usual. Because visitors land in either group by chance, and the groups differ only in whether rules ran for them, the difference in how they convert is what your personalization added.

## Why "conversions went up" isn't proof

Say you add a few rules, and next month your conversion rate is higher. Did the rules do that? Maybe. But a lot else changed in the same month:

- the season, and what people were shopping for;
- your ads, your prices and your emails;
- which pages search engines sent people to.

A before-and-after comparison mixes all of those together.

Comparing the visitors who saw a popup with the visitors who didn't has a trap of its own. Rules are usually aimed at people who are already likely to buy, such as returning visitors or people who came from your ads. The group that saw the popup would have converted more without it.

A holdout avoids both problems. Both groups live through the same month, the same ads and the same season, and nobody chose who went where. The only difference between them is the personalization.

## How a holdout works

1. **Split visitors at random.** Each visitor is put in or out of the holdout by chance, for example 5 in every 100.
2. **Keep the split.** A visitor stays in the same group on every page and every visit, so nobody sees your rules one day and none the next.
3. **Withhold every rule from the holdout.** Everything else stays the same: held-out visitors are still counted, and their goals still record.
4. **Compare the two groups.** Look at the share of each group's visits that reached a goal, side by side.

## Holdout or A/B test?

They answer different questions, and most sites need both.

| | A/B test | Holdout |
| --- | --- | --- |
| The question | Does this version of one rule beat another version, or nothing? | Do all my rules together beat no rules at all? |
| Who sees nothing | The control group of that one rule | The holdout, for every rule |
| Use it to | Pick the better message or offer | Decide whether personalization is worth running |

## How to read the result

The number to look at is the **lift**: how much higher the personalized group's conversion rate is than the holdout's.

For example, say 2.0% of holdout visits reach your goal and 2.3% of personalized visits do. The lift is (2.3 − 2.0) ÷ 2.0, or 15%: visitors with personalization converted 15% more often than visitors without it.

A lift only counts if it's bigger than chance alone would produce. With small groups, a few lucky visits can make either side look better, so a fair result needs:

- **Enough visits in both groups.** GeoEngage waits for at least 100 visits in each group before it shows a lift, and until then says how many more each group needs.
- **A significance test.** GeoEngage calls a difference only when it's statistically significant at the 5% level. It reports a significantly *negative* lift too: rules that hurt are worth knowing about.

## How big should the holdout be?

Big enough to measure, small enough to cost little. Every visitor in the holdout misses your personalization, so the holdout has a real cost, but a small one.

- **5% is a good start** for most sites, and it's GeoEngage's default.
- **The holdout sets the pace.** It's the smaller group, so it gathers visits more slowly. At 10% it gathers them twice as fast as at 5%, and a quiet site may need weeks for a verdict either way.
- **Pick a number you can leave alone.** Changing it redraws both groups, and the measurement starts again.

## Common mistakes

- **Stopping the moment it looks good.** Early leads come and go. Decide how long the test will run before you start.
- **Measuring without goals.** Lift is measured on goals, such as an order or a signup, so set those up first ([Goals and conversions](https://geoengage.app/docs/analytics/goals)).
- **Testing on yourself.** You might be in the holdout. If no rule ever shows for you, that may be why ([Testing your own site](https://geoengage.app/docs/rules/holdouts#testing-your-own-site)).
- **Reading part of a week.** Visitors behave differently on different days, so let the test run for whole weeks.

## Running one in GeoEngage

1. Set up your [goals](https://geoengage.app/docs/analytics/goals) first.
2. Open **Tracking Config → Holdout**, switch on **Holdout group** and set the **Percentage**.
3. Read the result as **Personalization lift** on **Rules & A/B Tests**, or in the Personalization section of **Overview**.

The [holdout docs](https://geoengage.app/docs/rules/holdouts) cover the details: who is held out, how consent mode affects it, and which visits are counted.

To test one rule rather than all of them, run an [A/B test](https://geoengage.app/docs/rules/ab-tests) on it. The [free-shipping message guide](https://geoengage.app/blog/free-shipping-message-by-country) walks through one.

## Common questions

### Doesn't a holdout cost me sales?

A little, if your personalization works: the holdout misses it. That's the price of knowing, and a small holdout keeps the price small. And if the holdout shows your rules don't help, it has saved you from running them.

### Is a holdout the same as a control group?

Yes. It's a control group for all of your personalization at once, where an A/B test has a control group for one rule.

### How long does it take?

It depends on your traffic and how often visitors reach your goal. A busy store can have an answer in days, and a quiet site may need weeks. While it waits, GeoEngage shows how many more visits each group needs.

---

Source: https://geoengage.app/blog/what-is-a-holdout-group