Holdouts

Withhold every rule from a small random group of visitors to measure what personalization adds to your conversions overall.

An A/B test tells you whether one version of a rule beats another. A holdout answers the bigger question: are visitors who get your rules better off than visitors who get none?

It works by withholding every rule from a random slice of visitors, then comparing everyone else against them. Because each visitor is placed at random and the two groups differ only in whether rules ran, the difference between them is what personalization caused.

Turning it on

Open Tracking Config → Holdout, switch on Holdout group and set the Percentage. It starts at 5%, which suits most sites.

The percentage can be 1 to 50. Above 50 the test gets no more accurate, and fewer visitors get your rules.
Changing the percentage redraws both groups and starts the measurement again, so pick a number you can leave alone.
Switching it off ends the holdout. Switching it back on starts a new one.

Who is held out

A visitor is placed in or out of the holdout from their visitor ID, so they get the same answer on every page and every visit.
If they clear their browser data, they get a new visitor ID and are placed again.
The holdout is per site. Being held out on one site says nothing about another.
Held-out visitors are still counted: page views, goals and recordings are recorded as usual. Only rules are withheld, including rules that wait for the visitor to scroll or stay on the page.
In consent mode, nothing is recorded for either group until the visitor accepts.

Reading the result

The result appears as Personalization lift on Rules & A/B Tests, and in the Personalization section of Overview. It shows:

The conversion rate of visits with personalization against visits in the holdout, with how many visits in each group reached a goal and any revenue.
The lift between them, and whether it's statistically significant.

It's measured on your goals, so set those up first. It follows the date range at the top of the dashboard, but counts only visits since the holdout last started or changed: earlier visits belong to a different draw of the two groups. When that cuts the range short, the card says from which date it's counting.

The lift is held back until each group has at least 100 visits. Until then it says how many more visits each group needs before a 10% difference would show up. A difference is called only when it's significant at the 5% level, and a lift that's significantly negative is reported too.

Testing your own site

You might be in the holdout yourself. If no rule ever shows for you, turn on data-debug="true" and check the browser console: it says when rules are being withheld. Testing in a private window gives you a new visitor ID, which may land outside it.