A/B tests
Test versions of a rule against each other, pick the goal that decides the winner, and read whether the difference is real.
An A/B test shows different versions of one rule to different visitors, so you can see which works better. To measure what all your rules add together, use a holdout instead.
A/B testing a rule
In the rule editor, turn on A/B Testing under Advanced Settings.
One version against nothing. Name your variant and set its Traffic Split. That share of the rule's visitors see its action; the rest are the control and see nothing, which is the fairest test of whether the action helps at all.
Several versions. Add variants to the Variants list, each with its own share of traffic (the shares must add up to 100). A variant can have its own action — different copy, a different offer, a different kind of action altogether — or fall back to the rule's. A variant named control always shows nothing.
Success Goal. Pick the goal the test is judged on, such as a purchase or a signup. Without one, the test only counts impressions.
Each visitor is assigned a variant once and keeps it on later visits, so no one sees the rule change from page to page.
Reading the result
Open Rules & A/B Tests in the dashboard. For each variant it shows how many visits saw it and how many completed the goal, and whether the difference is real:
- It waits for at least 100 visits in each variant. Before that, it says there aren't enough visits yet to tell them apart.
- It calls a difference only when it's statistically significant, at the 5% level. A variant that is significantly worse is reported as well: losing is a result too.
Testing more than one rule
Each A/B test judges one rule. To find out whether personalization works across the whole site, compared with visitors who get no rules at all, run a holdout.