# Holdouts

> Withhold every rule from a small random group of visitors to measure what personalization adds to your conversions overall.

An [A/B test](https://geoengage.app/docs/rules/ab-tests) tells you whether one version of a rule beats another. A holdout answers the bigger question: are visitors who get your rules better off than visitors who get none?

It works by withholding **every** rule from a random slice of visitors, then comparing everyone else against them. Because each visitor is placed at random and the two groups differ only in whether rules ran, the difference between them is what personalization caused.

## Turning it on

Open **Tracking Config → Holdout**, switch on **Holdout group** and set the **Percentage**. It starts at 5%, which suits most sites.

- The percentage can be 1 to 50. Above 50 the test gets no more accurate, and fewer visitors get your rules.
- Changing the percentage redraws both groups and starts the measurement again, so pick a number you can leave alone.
- Switching it off ends the holdout. Switching it back on starts a new one.

> [!WARNING]
> Visitors in the holdout see no personalization at all: no popups, no banners, no changed content, on any page. That's the price of the measurement, so keep the slice small.

## Who is held out

- A visitor is placed in or out of the holdout from their visitor ID, so they get the same answer on every page and every visit.
- If they clear their browser data, they get a new visitor ID and are placed again.
- The holdout is per site. Being held out on one site says nothing about another.
- Held-out visitors are still counted: page views, goals and [recordings](https://geoengage.app/docs/analytics/sessions) are recorded as usual. Only rules are withheld, including rules that wait for the visitor to [scroll or stay on the page](https://geoengage.app/docs/rules/conditions#behavior).
- In [consent mode](https://geoengage.app/docs/widget/consent), nothing is recorded for either group until the visitor accepts.

## Reading the result

The result appears as **Personalization lift** on **Rules & A/B Tests**, and in the Personalization section of **Overview**. It shows:

- The conversion rate of visits **with personalization** against visits in the **holdout**, with how many visits in each group reached a goal and any revenue.
- The lift between them, and whether it's statistically significant.

It's measured on your [goals](https://geoengage.app/docs/analytics/goals), so set those up first. It follows the date range at the top of the dashboard, but counts only visits since the holdout last started or changed: earlier visits belong to a different draw of the two groups. When that cuts the range short, the card says from which date it's counting.

The lift is held back until each group has at least 100 visits. Until then it says how many more visits each group needs before a 10% difference would show up. A difference is called only when it's significant at the 5% level, and a lift that's significantly *negative* is reported too.

> [!TIP]
> A small holdout on a quiet site can take weeks to reach a verdict. The holdout group is the smaller one, so it sets the pace: at 10% it gathers visits twice as fast as at 5%.

## Testing your own site

You might be in the holdout yourself. If no rule ever shows for you, turn on [`data-debug="true"`](https://geoengage.app/docs/widget#debugging) and check the browser console: it says when rules are being withheld. Testing in a private window gives you a new visitor ID, which may land outside it.

---

Source: https://geoengage.app/docs/rules/holdouts