Submit Article Requests

Do you have a suggestion for an article you would like to see created?
Feel free to submit this form and add your suggestions to our document board.

Please fill out the contact form below and we will reply as soon as possible.

  • Appcues Certifications & Training
  • Contact Us
  • Home
  • Analytics
  • Experience and Event Analytics

Flow variation A/B testing

Updated at March 7th, 2023

Submit Article Requests

Do you have a suggestion for an article you would like to see created?
Feel free to submit this form and add your suggestions to our document board.

Please fill out the contact form with the details about the help content you'd like to see.

  • Installation & Developers
    Installing Appcues Installing Appcues Mobile API & Data Troubleshooting Extras
  • User Experiences
    Web Experiences Building Web Experiences Building Mobile Experiences Customization & Styling Targeting Studio Troubleshooting Use Cases FAQ
  • Mobile
    Installation & Overview Building Mobile Experiences Mobile Analytics Troubleshooting Mobile Use Cases
  • Account Management
    Subscription Users & Data
  • Analytics
    Experience and Event Analytics Data
  • Best Practices
    Use Cases Pro Tips PLG FAQ
  • Integrations
    Integration Documents Use Cases Resources
  • Post-mortem
    System Incidents
+ More

Table of Contents

Setting up a variation test 1. Start by creating two or more Flows. 2. Create Segments for each Flow variation 3. Apply your Segments to each Flow variation. Testing more than two variations Measuring results

Using Appcues to manage your customer experiences allows you to quickly experiment with changes in Flow content in order to optimize your conversion metrics.

With Flow variation testing, you can perform an A/B test to measure the effectiveness of multiple versions of an experience in parallel. For example, are your users more likely to engage with a slideout containing a gif or a static photo? Do your users like a tooltip tour, or would they prefer one modal containing all content? A/B testing will help you find the answer.

Setting up a variation test

1. Start by creating two or more Flows.

Before you set up your A/B test, make sure that your target audience for the Flows is large enough to get statistically significant results. Generally, the larger the group the better, but best practice is to have at least 500 users in each group. There are several online tools you can use to measure your test’s statistical significance.

If you’ll be testing slight variations of the same Flow, you can save time by creating the first Flow, cloning it from your Flows list, and making edits to the cloned version in the Appcues Builder.

For proper variation testing, set the frequency of each Flow to “Show once”.

2. Create Segments for each Flow variation

Navigate to the Segments page and “Create a Segment”, using the “Audience Randomizer” auto-property to define a percentage of users to be included in the Segment.

Each user will be randomly assigned a number from 1-100 upon first login, so targeting a Flow to users with an Audience Randomizer number of less than 51 will include all users assigned numbers 1-50.

Clone your Segment, and update the Segment name and “Audience Randomizer” attribute in the cloned version. Make sure all other audience segmentation criteria are consistent across your test audiences.

3. Apply your Segments to each Flow variation.

Make sure all other Flow targeting criteria are consistent across each test Flow; e.g. ‘show once’ vs ‘show every time’, page targeting, and Flow priority.

Add the Goal you want to measure for conversion to each Flow. For more information on creating and managing Goals, check out this help doc.

Note: it’s always best to measure that your Flows are driving a specific action for the users on your app (example: completed trial signup). However, you could take a more basic approach and simply measure the completion rate of a Flow. In this case, you could not need to add a Goal to your Flows, since completion rate is always tracked.

Testing more than two variations

If you’ll be testing more than two variations, create the same number of audience segments as Flows, and split the Audience Randomizer evenly across each.

For example, if you’re doing an A/B/C/D test, you should have four audience segments that look like this:

Measuring results

Once you’ve set a Goal, it will be tracked on the analytics page of a Flow. Compare the results of your Flows against each other to determine which Flow performed the best.

a/b testing flow comparison

Was this article helpful?

Yes
No
Give feedback about this article

Related Articles

  • Viewing Form & Survey Results
  • Understanding Flow Event Data
  • Export Flow Analytics Data (CSV)
  • Flow Step Breakdown Analytics
Appcues logo

Product

Why Appcues
How it works
Integrations
Security
Pricing
What's new

Use cases

User Onboarding Software
Feature Adoption Software
NPS & Surveys
Announcements
Insights
Mobile Adoption

Company

About
Careers
we're Hiring

Support

Developer Docs
Contact

Resources

The Appcues Blog
Product Adoption Academy
GoodUX
Case studies
Webinar Series
Made with Appcues
Appcues University

Follow us

Facebook iconTwitter icon greyLinkedin iconInstagram icon
© 2022 Appcues. All rights reserved.
SecurityTerms of ServicePrivacy Policy
Expand