Application Stability Index: How Healthy Are Your Applications?

Stability is the measurement of application health and user experience.

It is usually calculated in two ways to provide the complete picture － as a percentage of application sessions that are crash-free or did not result in an unhandled exception and as a percentage of daily active users who do not experience an error.

How Many Nines of Stability Do You Have?

Providing an error-free experience to your customers is of utmost importance to drive conversion, engagement, and retention.

Although stability is commonly a KPI owned by engineering organizations, it has a significant impact on overall business performance and growth.

To help you understand how your application compares, Bugsnag has compiled data on the session-based stability of leading mobile and web applications from various market segments, including eCommerce, media and entertainment, financial services, logistics, gaming, and more.

This data can also help you determine application stability SLAs and SLOs, which are similar to the “five nines” that infrastructure and operational teams use to measure uptime and availability.

This metrics-driven approach can help development teams make decisions about when to build features vs. fix bugs based on the application’s current stability.

This data can also help you determine application stability SLAs and SLOs, which are similar to the “five nines” that infrastructure and operational teams use to measure uptime and availability.

This metrics-driven approach can help development teams make decisions about when to build features vs. fix bugs based on the application’s current stability.

Mobile Applications

A prime opportunity for mobile-first engineering organizations to measure and improve application stability

The data includes applications from several mobile development platforms, including Android, iOS, React Native, and Unity.

Stability scores are negatively impacted by session-ending events, which include things like crashes as well as ANRs (Application Not Responding) in Android, React Native, and Unity applications and OOMs (Out of Memory) in iOS applications.

Android and iOS native applications tend to have a high median stability because there are very specialized developers working on these applications who have the expertise required to understand and address any stability issues effectively.

Android applications tend to have a slightly lower median stability compared to iOS applications because Android presents a much less constrained development environment.

Increased fragmentation of Android devices makes it more difficult to test applications whereas iOS development teams only need to provide a stable experience on a limited number of devices that Apple releases every year.

More about Android fragmentation

React Native allows Engineering teams to develop for iOS and Android together, however, writing code once to deploy everywhere does present some trade-offs. Oftentimes, React Native is used to open up mobile development to for example JavaScript engineers, who may not be accustomed to the challenges it presents.

Expertise is also needed in Android and iOS to effectively investigate and fix bugs in the native layers of the code. This may explain the wider range and a slightly lower median stability.

More about React Native

Due to the sandboxed environment of Unity applications, bugs that crash the entire application are much less likely to occur.

It is possible that errors within frames are disrupting the gaming experience, but they don’t result in a full on crash, which may be why Unity applications boast the highest median stability and the narrowest range.

Web Applications

Modern web development frameworks may provide more resources to manage errors.

The data includes applications from several front end development platforms, including Angular, Backbone, Ember, React, and Vue.

Causes of unhandled exceptions in web applications can include a bug which prevents the entire page from rendering, an event handler bug which causes the user interaction to fail, an unhandled promise rejection warning, and more.
‍
Web applications may have a lower median stability compared to mobile applications because monitoring and addressing client-side issues in JavaScript applications generally requires more effort than doing so in mobile applications.

Mobile is a newer discipline and its development frameworks were built with a greater emphasis on the importance of managing errors from the get-go, whereas web is an older discipline that had to learn this over time.

Angular, Ember, React, and Vue are modern, opinionated JavaScript frameworks that are built with consideration for error handling. Angular and React were created and sponsored by development teams at Google and Facebook. Engineering organizations working with these platforms have access to the resources and documentation they need to investigate and fix errors that may affect application stability.

Backboneis an older and less opinionated web development framework. Development teams don’t have access to the same coding guidelines, best practices, and considerations for error handling that the other more recent development frameworks offer, which may explain the lower median stability and wider range for Backbone applications.

The stability of each application is monitored across browsers, meaning customers experiencing errors in Internet Explorer, Google Chrome, Mozilla Firefox, and other browsers are included in the overall application stability score. Errors caused by browser extensions are also included; however, most engineering organizations don’t spend resources to investigate and fix these errors since their code is not the culprit.

Engineering Team Sizes

Technical debt can make it more challenging for larger engineering organizations to maintain a higher stability

In order to understand if the size of an engineering team has an impact on application stability, we looked at the stability of applications supported by a few different engineering team sizes.

The size of an engineering team tends to be linear to the age of the organization. Younger organizations prioritize product-market fit and need to release new features quickly. Maintaining a high application stability is less important than capturing market share and increasing competitive advantage. This may explain why the smaller the engineering team, the lower the median stability.

Larger engineering teams are usually working on more mature applications but standardizing on stability can become increasingly challenging. Legacy code, compounding technical debt, and complexity of team structure are just a few of the hurdles that can make maintaining a higher stability more difficult. This may explain why engineering organizations with more than 100 engineers have a lower median stability and a wider range.

Previously published here.