Using Stability as a Barometer of Quality and Quantity

User experiences with web and mobile applications are more important than ever. Users have high expectations for performance and demand consistent updates with new features and content. Thanks to low switching costs and a growing plethora of options available on the market, if users see even the smallest issue with stability, speed, or responsiveness, they’ll explore other options.

Stability: A Barometer of Quality and Quantity

Stability measures application health as well as user experience. It’s calculated in two main ways to provide insight. A percentage of

• Crash-free application sessions (or unhandled exceptions)
• Daily active users who do not experience an error

Our insights help you determine application stability SLAs and SLOs, like how infrastructure and operational teams rely on the “five nines” to measure uptime and availability.

Target stability scores best represent your SLO. This score can be thought of as an aspirational goal to balance fixing bugs with building new features. If your stability score is above target stability, you can focus on building new features. If lower, get the bugs.

On the other hand, you can compare our critical stability score to your SLA. This is the minimum threshold your app should run on to ensure its functionality. If your stability score drops below your critical stability threshold, drop everything and focus on fixing bugs immediately.

To help you compare your application to other leading web and mobile apps, Bugsnag has aggregated data to analyze stability trends and release frequency with industry benchmarks. We welcome you to learn from them as well.

Measuring App Quality Across Industries

When it comes to measuring app quality, stability hits different industries in varied ways. Factors like value of user interactions, and volume. Understanding these factors will help engineering organizations develop strategies for delivering higher quality apps.

How Stability Turns Loyalty Into Revenue

High Stability

Trendsetters: Setting the bar while delivering enriching new experiences

• Media and Entertainment has taken off in the past several years when it comes to streaming media. As many newcomers attempt to grow their users, it remains essential for them to provide an optimal, uninterrupted experience. Meanwhile, to retain their massive subscriber base against increased competition, streaming giants seek to strike a balance between delivering a seamless experience and consistently offering new features to enhance their application – across a range of devices.
• Ecommerce continues to be a driving force even as brick and mortar retail attempts to recover to pre-pandemic levels. As worldwide ecommerce revenue expects to climb past $5.5 trillion in 2022, there’s a lot of money in customers’ carts when they arrive at checkout. Giving customers a smooth experience is paramount, as crashes directly impact revenue.

Medium Stability

Middle of the Pack: Good with the potential to be great

• B2B SaaS, Technology, Finance & Banking, Education, and Telecom all had high stability scores but weren’t quite high enough to meet the standards set by our Trendsetters. B2B SaaS, Technology, Telecom, and Finance & Banking displayed high scores for the second year in a row. Each these industries are involved in significant financial transactions and often are highly regulated. A stable user experience is essential to success, especially when stability translates to higher loyalty and revenue.
• B2C SaaS/Consumer Mobile, Social, Consumer Goods, Travel & Hospitality all have a strong consumer focus. Consumer Goods, Travel & Hospitality and B2C SaaS have financial transactions that must be secure for consumers’ piece of mind and their continued app use.

Low Stability

Run of the Mill: Nothing special and easily replaceable

• Gaming, as in previous editions of our index, continues to lag behind other industries in stability. Despite millions of users, especially among mobile games, these users are less loyal and have no qualms about replacing games with others due to next to none switching costs. To retain these users, games often introduce new features, but this increases  the likelihood of bugs. This explains why Gaming also had the most score variability out of all the included industries.
• Health & Wellness also scored below the industry-wide mean of our index while demonstrating large variability in scores. These apps still have a way to go to achieve a healthy stability score and should focus on fixing bugs before introducing any new features or capabilities.

Web vs Mobile: A Comparison

Mobile apps continue to become more stable while web applications show dramatic improvement.

In our reports from 2020 and 2021, median mobile app stability score in-creased from 99.63% to 99.80%. This year, median stability score for mobile apps continued to trend upwards, increasing to 99.88% in 2022. While mobile apps are becoming more stable, there is still work to be done to achieve the “five nines” of stability.

Web-based applications showed surprising gains in their median stability score. Just two years ago web applications had a median stability score of 99.39%.

In 2020 we mentioned this may be due to monitoring and ad-dressing client-side issues in JavaScript applications usually requires more effort than mobile apps. Evidently, there has been significant strides made in adapting this older discipline to today’s needs, as web apps scored a median stability of 99.94% in 2022’s report. While this is excellent news and shows the success of implementing error handling strategies, there is still room to grow to achieve the “five nines,” like mobile apps.

Different platforms have different errors

Our web applications data includes several front-end development platforms, including Ember, JavaScript, React, and Vue. Each of these platforms is a modern, opinionated JavaScript framework built with consideration for error handling. Some specific causes of unhandled exceptions in web applications may be an event handler bug causing the user interaction to fail, an unhandled promise rejection warning, a bug preventing a page from rendering, and more. Mobile applications data includes several mobile development platforms, among them Android, iOS, React Native, and Unity. For these applications, stability scores are negatively impacted by session-ending events. This means things like crashes and ANRs (Application Not Responding) in Android, React Native, and Unity frameworks as well as OOMs (Out of Memory) in iOS applications.

30 DAY STABILITY SCORE BY PLATFORM

Mobile Platforms Built with Specific Personas in Mind

Android and iOS continue to have high median stability scores. One reason for this consistency is the amount of developers specialized in Android and iOS. Their well of knowledge not only helps them fix errors, it helps them do so efficiently.

While their median stability scores are similar, the range of scores between Android and iOS applications is noticeably different. Many Android applications have a stability lower than the platform median while iOS applications have a much narrower range. One reason for this discrepancy is because Android has a much larger number of devices – a more open development environment. This can make it difficult to test applications and increases exposure to crashes and errors. Whereas dedicated iOS development teams have a limited number of devices to test their application’s stability on.

React Native is a very popular mobile development framework thanks to its ability to simultaneously develop Android and iOS applications. It is especially helpful for engineers that may not be familiar with mobile frameworks but have experience with React for web applications.

Unity is extremely popular in the mobile gaming industry, with many of the most popular games developed on the platform. As primarily a mobile gaming framework, it’s not surprising Unity showed the same trends out of the platforms included in our report. Mobile games often introduce new features rapidly to satisfy their users, which can lead to more errors and instability. Feature flags and experiments to introduce new features to a subsegment of users will help improve app stability while minimizing the amount of bugs (or risk) introduced to your application’s entire user base. As an aside, we recently launched support for Unreal Engine earlier this year, however we were unable to use the data for this year’s report. We expect to see Unreal Engine as a platform listed next year.

The Versatility Behind Modern Web Application Frameworks

JavaScript and React showed similar stability, with JavaScript applications just edging out React applications for the highest stability score. React did have a narrower stability score distribution than JavaScript, which may be a result of JavaScript having a wider range of industries using its framework than React.

Each web-based application has its stability measured across multiple browsers. All customers experiencing errors in Google Chrome, Mozilla Firefox, Safari, or other browsers are included in the overall stability score of the application.

Sometimes errors may be caused by specific browser extensions, which are included in our stability score. However, most engineering organizations do not dedicate time and resources to fix these bugs because they are not from their application’s code. ‍ ‍As a disclaimer, because the volume of Ember and Vue projects was significantly less than JavaScript and React projects, for the purpose of our analysis we grouped Ember and Vue projects with JavaScript projects.

Mobile Apps Release at Almost Twice the Frequency of Web Apps

Web and mobile applications are shifting their release cadences as consumers frequently demand new features and immediate bug resolution. We sifted through our data to analyze the release frequency of the leading applications so you can benchmark them against your release cadence.

One notable example is the trend of faster releases for mobile apps which update weekly. On average, across all industries mobile apps deliver every four to five days. Meanwhile, web applications adhere to the biweekly release frequency, about every ten days. One possible explanation for the discrepancy is mobile apps more frequently test new features by practicing progressive delivery with feature flags, experiments, and A/B tests to retain their users. This is especially true in mobile gaming apps.

Release Cadences Depend on Industry Type

Gaming has the narrowest distribution, averaging about 6.5 releases a month. This is an increase over the weekly frequency we saw in last year’s report. The frequent release schedule may help explain why gaming saw the lowest stability score in our report.

Health & Wellness had the widest range of distribution as well as the second highest release median. Their second-lowest stability median is proof why there needs to be a balance between frequent and infrequent updates, and how measures like target and critical stability scores can help find that balance.

All industries besides Media & Entertainment released a new version of their application on a weekly basis at minimum. It’s becoming clearer developers are more confident in their releases schedules and accelerating new releases through progressive delivery practices like feature flags.

Outside of a few industries, each industry displayed a wide range of release schedules, with some releasing multiple times a day. This may indicate bugs are releasing at large scale and teams need to focus on minimizing risk with new features.

Key Takeaways

• Increase stability to grow customer engagement, adoption, and retain loyal customers.
  • App stability is closely tied to overall customer experience and business outcomes. It is no secret happier users become loyal customers. Delivering a dependable, stable application will set your app apart from its competitors.
• Know when to fix vs build to accelerate release production.
  • The most successful apps closely monitor their app’s stability to determine when to allocate resources to bug fixing for a better user experience or focus on building new features. Freeing up these resources allow for proper allocation to speed up release production while creating exciting new ways to solve customer problems.
• Grow revenue and market share by delivering desired features faster.
  • Release frequency in web and mobile apps is increasing each year as users want more features faster. Balance roadmap agility with app stability to release new features that are bug-free and grow your app’s competitive advantage.