A/B testing is one of the primary ways of verifying ideas in mobile apps on a fast scale. The arrangement is typically straightforward: you run a test for a few weeks, gauge its impact, and decide to roll out a feature or iterate further. One of the essential but commonly overlooked aspects of A/B testing is setting the correct exposure points. In this article, we'll cover why correct exposure point setting is important, common pitfalls to avoid, and provide some tips and tricks I've learned along the way. What are exposure points, anyway? An exposure point is the real moment when the user initially encounters or interacts with the feature that you're testing. For example, timing when a user sees a new button, or when they see a redesigned landing page after having clicked on something. Why is it important to choose a good exposure point? If you've ever stared at A/B test results and they simply didn't make sense, bad exposure points might be the culprit. Bad exposure points can lead to: Confusing Data: It won't be clear whether your feature is a hit or miss.
Hidden Bugs: Everything might appear okay on the surface, but serious issues like app crashes might be slipping under the radar.
Missed Opportunities: Your data might falsely show no significant impact, even though the users indeed enjoyed the feature. Confusing Data: It won't be clear whether your feature is a hit or miss. Confusing Data: Confusing Data: Hidden Bugs: Everything might appear okay on the surface, but serious issues like app crashes might be slipping under the radar. Hidden Bugs: Hidden Bugs: Missed Opportunities: Your data might falsely show no significant impact, even though the users indeed enjoyed the feature. Missed Opportunities: Missed Opportunities: Example of a bad exposure point Imagine you're testing an upsell for a new subscription in your app but trigger the exposure point on app launch rather than when users actually view the subscription page. Problem: Maybe only 10% of users ever see the upsell page, making 90% of your test data worthless for this decision. Problem: Problem: Loose vs. Tight Exposure Points Loose Exposure Points: Loose Exposure Points: These occur a bit too early. Users are exposed to the experiment prior to experiencing the tested feature. This premature exposure dilutes the data, and it is hard to find true impact. Tight Exposure Points: Tight Exposure Points: These take place at the exact moment that users experience the tested variant. Data collected with precise exposure points is more accurate and reliable, and easier to analyze. Which is better? It depends on your use case. Tight exposure points are preferable since they provide you with cleaner, more defined data, even at smaller sample sizes. Occasionally, however, tight exposure is not possible. In that situation, you can utilize loose exposure points with the knowledge you'll likely require a larger sample size to achieve significant results. Avoid stacking changes Never mix or pile multiple exposure points together in a single test. Split each change into its own A/B test. While it will take a bit more time, your data accuracy will be far greater, and you'll have more accurate conclusions on each individual feature. Quick real-life example Let's say you are adding a new row type in a table view: Good exposure point: Trigger exposure precisely when the new row is rendered in view.
Bad exposure point: Fire exposure as soon as the table is loaded, even if the new row is not yet visible. Good exposure point: Trigger exposure precisely when the new row is rendered in view. Good exposure point: Good exposure point: Bad exposure point: Fire exposure as soon as the table is loaded, even if the new row is not yet visible. Bad exposure point: Bad exposure point: Final tips Plan ahead: Think through your exposure points at feature planning time.
Iterate rapidly: If initial test results are not as expected, iterate rapidly. Plan ahead: Think through your exposure points at feature planning time. Plan ahead: Plan ahead: Iterate rapidly: If initial test results are not as expected, iterate rapidly. Iterate rapidly: Iterate rapidly: I would love to hear your stories and experiences in determining exposure points for your A/B tests. Share them in the comments!

This story contains new, firsthand information uncovered by the writer.

Mastering Exposure Points for Accurate Mobile A/B Testing

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Writing Great Tech Specs for iOS Features: A Practical Guide

4 Lessons We Learned in 2019 (and How Marketers Can Apply Them in 2020)

A/B Testing For Email Marketing — What Should You Test For?

A/B Testing Our Way to a Better Payment Gateway

A/B Testing - You’re doing it wrong

AB Testing For Digital Products

Writing Great Tech Specs for iOS Features: A Practical Guide

4 Lessons We Learned in 2019 (and How Marketers Can Apply Them in 2020)

A/B Testing For Email Marketing — What Should You Test For?

A/B Testing Our Way to a Better Payment Gateway

A/B Testing - You’re doing it wrong

AB Testing For Digital Products

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps