AI as complement to A/B test design

The practice, the science of split testing is time-honored and remains the standard for both micro and macro optimizations in digital properties, but the upstart we collectively label artificial intelligence, is here to alter the narrative. How will machine learning enhance our ability to deliver the most profitable, customer-delighting experiences?

High Failure Rates in A/B TestingNearly 80% of A/B test variations fail to deliver meaningful, positive results for both users and business outcomes. Cold, hard facts from hundreds of thousands of digital experiments run across the globe. This has led us communally to think fast and act fast with low-cost, high-impact experiments to push that global failure rate down. We’ve gotten clever with where, how and to whom we deliver experiences and as a result, win rates increase, job satisfaction soars and we live to test another day.

But is there is another way to maintain the A/B tester’s hunger for positive change, while reducing the cost of each successful experience delivery?

Enter the machines. With AI in tow, multiple design narratives and dozens of creative assets can live and breathe simultaneously through machine learning optimization over time.

Reduce the Impact of LuckA/B testing relies on solid user research, historical data and strong design, product and technology intuition to form hypotheses that can be tested with a live audience. But in the end, there’s a strong element of luck to finding the next game-changing audience or creative.

With AI as an enhancement, your A/B testing can now center around larger frameworks and meta-hypotheses, while machine learning does the exploratory over time, micro-measuring content delivery and UX optimization. The tester is now free to tackle the larger issues, leaving optimization-over-time to the machine.

More than 1,000,000 total combinations

Understanding the complexityAn example for illustration. My website has 10 creative assets across three major zones on the homepage. With 10 sample points (n) and a modest 4 variations for each (r), there are over 1,000,000 total combinations to choose from. Add in zones with interchangeable UX variations and well, you get the idea.

Rather than throw wave after wave of my best men/women at this optimization problem, I select core metrics, both positive and negative, weighting for desired outcomes (increases in profit and reduced abandonment) and let an evolutionary algorithm determine both the cohorts and placements. The end result is an incredibly rapid explore/exploit approach, with math that would make all but the most stalwart A/B test practitioner cry uncle.

Machine learning also excels at observing micro changes across the value stream to bring the right message (asset) to the right visitor, at the right time. Timely, contextual, relevant.

For starters, we can use AI to model:

• Device type and browser optimization for code and content
• Digital body language, including hover, scroll, mobile button press
• Full flow optimizations with next click optimizations
• Historical consumption from prior visits
• Micro-interaction patterns we may not directly observe

Build Your OwnSeveral off-the-shelf systems have emerged to tap into this 21st century gold mine, including Sentient Ascend, Dynamic Yield, Conductrics, Monetate and others. A vendor not to your liking? Many have built their own with a combination of data collection, a highly available database and recommendation engine (e.g. Python, PostgreSQL, H2O.ai).

With the market moving rapidly toward intelligent experience delivery, it’s time to up your A/B testing game with AI and increase the rate of positive change in your digital experiences.