A New Way to Beta Test

Using feature toggles to beta test in a real-world environment

It’s best practice for products to have some sort of beta, a way to collect customer feedback and test performance before releasing to everyone. In an era of continuous delivery, we are delivering new features and experiences more frequently and with less time to gather thorough customer and performance feedback. With this increased cadence, product teams are having to make betas shorter, forego them all together, or slow down their release cadence to gather adequate customer feedback.

Challenges of Traditional Betas

• Coordinating opt-ins. It sometimes takes weeks or months to gather customer opt-ins to test new betas. You also have to organize the distribution of beta keys (i.e., for early access to games) and reminder emails.
• Organizing focus groups. Getting feedback from focus groups is often time-consuming and expensive, creating a long feedback loop that lengthens the release process.
• Opt-out. If customers opt in to a beta and don’t like the experience, then they will want a simple mechanism to switch back to the production version.
• Granular betas. It is very difficult to do targeted betas based on user attributes or to perform incremental percentage rollouts of new beta features.

Feature Toggles

To overcome these challenges, smart product teams are beginning to run betas with feature flags and toggles. These are mechanisms for granularly controlling software releases, allowing you to control the timing and visibility of a beta release.

Currently, many betas are tied to code releases and are managed by a config file or database. This approach requires engineering time or custom mechanisms to opt-in users.

With feature toggles, you can empower product, marketing, sales, and even customers to opt-in new to a new beta experience.

In this simple example, you can use a toggle to control the visibility of a new beta feature. Ideally, this toggle would be part of a user interface that could be controlled by a non-technical team member. The code itself could be deployed off and then turned on via the toggle.

Moreover, you can also use the toggle to control the percentage of users who get the beta experience. For instance, you could release the new beta experience and have it rolled out to 0% of users. You could gradually increase the rollout percentage from 1% to 5% to 20% and more, collecting customer and performance feedback along the way.

Surfacing this beta control functionality in a user interface is critical for giving non-technical team members access to release controls.

Regional Betas

For a recent example of a targeted rollout, we can look at how Pokémon GO released their product country by country: first to the United States and then abroad.

This is a great use case for feature toggles because you can create targeting rules to determine which users receive the feature first. For example, I could create a toggle that is governed by the rule: “If users live in San Francisco, then serve the new Nearby Pokémon feature.” This allows you to maintain different regional feature sets without having to deploy different versions of the application. It also allowed Pokémon GO to refine their algorithms and assess customer feedback before rolling out the new feature to a wider audience.

Benefits of Beta Testing With Feature Toggles

• Empowered non-technical users. Allow the sales, marketing, product, design, and business teams to turn features on for specific users, collect feedback, and control the business logic. This also substantially cuts down on engineering time.
• Production feedback for your beta tests. Test features in production with limited user segments to collect customer and performance feedback.
• Incremental percentage rollouts. Gradually roll out features to incrementally test performance and mitigate risk. If the feature is bad, toggle it off.
• Real-time opt-in and opt-out. Allow users to opt in and out of beta tests in real time, controlled via the feature toggle. Skylight provides a nice articleon this.

Getting Started With Toggles

Conceptually, a feature toggle is relatively simple. You create a conditional in your code that controls the visibility of a code snippet. There are many open-source libraries that will allow you to get started.