A/B test Experience variations
Compare two or more versions of an Experience to find out which one performs better with your users.
Table of Contents
Test two or more versions of an Experience side by side to find out which performs better. Use the Audience Randomizer to split users into groups, then compare Goal conversion or completion rates across variations.
Prerequisites
- Two or more Experiences (your variations)
- A target audience large enough for meaningful results — aim for at least 500 users per group
- A Goal attached to each Experience (optional but recommended)
Set up the test
1. Create your Experience variations
- Build each variation as a separate Experience. To save time, create the first Experience, then clone it from the Experience list and edit the clone in the Builder.
- Set the frequency of each Experience to Show once so each user only sees one version.
2. Create audience Segments
- Go to Segments and click Create a Segment.
- Add a condition using the Audience Randomizer property. Each user is randomly assigned a number from 1–100 on first login.
- For Group A, set the condition to Audience Randomizer is less than 51 (captures users 1–50).
- Clone the Segment and update the clone for Group B: set Audience Randomizer is greater than 50 (captures users 51–100).
- Keep all other audience criteria identical across both Segments.
NOTE: The Audience Randomizer property is a number from 1 to 100 randomly assigned to users upon their first login. As this is a random assignation for each user, it is possible for more users to have an Audience Randomizer less than 50 than there are users with a number greater than 50 (or visa versa), meaning there is no expectation that these two groups will be completely evenly split. This may be a concern if working with fewer users in your account, as the difference in split will statistically decrease as the group sizes increase.
Because this is a static property that does not change once assigned, you may also want to adjust the exact numbers being targeted if you are planning on doing multiple A/B flow variations to help ensure different groups of users are eligible for the A or B variation. For example, the second time you create an A/B flow, you might target the A version to users with an Audience Randomizer greater than 25 and less than 75, with the B flow targeting the remaining users.
3. Apply Segments and Goals
- Open each Experience's Settings and assign the matching segment.
- Confirm all other targeting criteria are identical across Experiences — page targeting, frequency, and priority.
- Add the same Goal to each Experience. For Goal setup details, see Set up Goals.
Test more than two variations
Create one Segment per variation and divide the Audience Randomizer range evenly. For an A/B/C/D test with four Flows, use ranges like 1–25, 26–50, 51–75, and 76–100.
Measure results
Open each Experience's Analytics page and compare Goal conversion rates (or completion rates if you aren't using Goals). Use a significance calculator to confirm your results are statistically meaningful before making changes.
