DevOps: I Measure The Metrics A Bit Differently. Here's Why

The four key DevOps metrics are an exciting set of measurements. They’re getting more and more relevant since the book Accelerate has been published. I firmly believe they’re essential for engineering teams seeking effectiveness and efficiency.

The Four key DevOps metrics are:

• Lead Time for Change
• Deploy Frequency
• Change Failure Rate
• Median Time to Restore (MTTR)

You can learn more about measuring it in Paul Duvall’s article or reading the 2019 Accelerate State of DevOps.

In this article, I want to explore why I like them and why we measure it slightly differently at Source Level. I’ll focus on the Lead Time for Change and Deploy Frequency, as I have more experience working with them.

Four Key DevOps Metrics for engineering teams

The main goal of an engineering team is to add value for the customers through technology. Delivering value is abstract, and describing what it means is hard.

Sometimes value translates into a single line of code or even in a configuration change. But at other times, value translates into three months of 8-sized-team work. It happens because value perception is not attached to the work itself. It’s bound to what end-users extract from the running product.

That’s why measuring Story Points is useless. The team attributes a numeric value based on how much effort a given User Story or task requires. So, Story Points measures engineering performance based on their effort. And, as stated, it’s not related to the end-user perception of value.

However, Modern Software Development requires agility, and it doesn’t come from the nature of the work, such as painful features or slippery bugs. Agility comes from a smooth and sustainable process.

Some years ago, using agility and process in the same sentence would be hilarious. But agile’s core is to respond to change appropriately, and delivering more doing less is a consequence, not the goal.

That said, the Four Key DevOps metrics become the best allies of Tech Leads, Engineering Managers, and VPs of Engineering looking for efficiency. These roles require full attention to the big picture, and DevOps metrics give a perfect overview of how the system responds to daily activities.

Lead Time for Change and Deploy Frequency summarize the development pipeline, and any change in the process reflects on them. Managers should have them in their dashboards or any other accessible and accurate place.

Considerations on Lead Time for Change

Lead Time for Change measures how much time it takes for a code change to reach production. It’s crucial to measure up to production because staging or any other non-official environment won’t provide any value.

However, it raises an interesting discussion on feature toggles (or feature flags). Deploying a new feature is not the final step of the value chain if your team uses feature toggles or feature flags. Features must be enabled to add value, and so, I’d consider measuring not only Lead Time for Change but also the Lead Time to Value.

Another aspect I’d like to point is statistics. The recommendation is to calculate the Median Lead Time for Change. It outputs the value at the midpoint of a frequency distribution. In other words, half of the changes are equal or less the median. But how about the other half of the changes? They could take considerably more.

At SourceLevel, we’ve been more likely to use the 75th percentile as the default metric and keeping track of the 95th percentile as well. The 75th percentile is more comprehensive and realistic. In contrast, the 95th percentile gives us an idea of the small portion of work left out of the count and excludes the top 5% most time-consuming changes, as we consider them outliers.

Considerations on Deploy Frequency

Deploy Frequency measures how many times code changes reached production in a period. It can be measure as deploy per hour, per day, per week, per month, and so on.

It’s crucial to adjust the period properly. Saying a team deploys four times in a month sounds like a team deploys once in a week. But it may not be accurate. If the team deploys four times at the end of a month, it means the team has a different pace than deploying once a week.

If it’s usual for your team to have one or zero deploys per day, don’t measure it in days. Start measuring in weeks. Once the team starts deploying more than seven times a week, you can switch for days. Switch to shorter timeframes whenever possible, but always respecting the accuracy of the information.

In short

The Four Key DevOps Metrics are useful and give an overall idea of how the system responds to daily activities. Lead Time for Change and Deploy Frequency are much more price than Story Points and should be accessible and accurate.

Tech Leads, Engineering Managers, and VP of Engineering benefit from these metrics. Although more metrics are necessary to investigate problems, they are excellent indicators of the engineering process healthiness.

(Disclaimer: The author is a co-Founder at SourceLevel)