7 Rules to Track Software Engineering Metrics Correctly

How to Use  -  and NOT Abuse  -  Software Engineering Metrics

Software engineering is one of the top pillars for an organization to become one of the best performers in their industry. That’s why engineering efficiency has become the biggest challenge to enterprises, even above access to capital, according to a Stripe study. Therefore, a lot of investment has been made to create new tools and methodologies to help engineering teams build better software faster.

Effective engineering teams see themselves as complex interdependent systems with a relentless focus on improving their process and output. The best teams keep track of their improvements through a set of chosen indicators, which we call software engineering metrics. The teams that don’t will see their productivity plateau fast and even deteriorate while the organization scales.

The big issue is that throughout organizations, these software engineering metrics have been used the wrong way, creating toxic engineering culture and driving valuable talent out of their organization — without any positive impact on performance. For these reasons, every team needs to be careful about which metrics to track and how to track them.

In this article, we’re going to list the rules you should follow with whatever metrics you’re tracking. Let’s start by providing more context on the topic of software engineering metrics.

What are software engineering metrics?

In software, there are two categories of metrics and we use different names for those:

Software engineering metrics, also known as software development metrics or software delivery performance, every team has a different name for them, it seems. What is important here is that those indicators measure how software is being built and the engineering team productivity. Software or application performance metrics are the metrics of the software delivered, response time of the application, etc. NewRelic is typically one of the main providers of such metrics. You could also think of this in terms of customer satisfaction metrics (Net Promoter Score typically).

In this article, we’re focusing on the first set of metrics. Those are the ones that help an organization scale and that will actually impact the second set, which is one of the end results of the work done by the team.

To understand software engineering metrics, we need to understand the main goals behind tracking and analyzing them:

• Determine the quality and productivity of the current software delivery process, and identify areas of improvement;
• Predict the quality and progress of the software development projects;
• Better manage workloads and priorities between teams and team members.

Overall, engineering metrics will help teams increase the engineering return on investment, by offering an assessment of the impact of every decision on projects, processes and performance goals. Here is more on the topic, including the 22 most used software engineering metrics and how to use them.

Some examples include the following (and some are likely familiar to you):

• Project or Sprint Burndown and Ticket Close Rate
• Code Churn / Refactoring rate / New Work
• Lead Time and Cycle Time
• Deployment and Commit Frequency
• Change Failure Percentage
• Mean Time Between Failures (MTBF) and Mean Time To Recover/Repair (MTTR)

Whatever metrics you monitor, there are a few rules on how to use those metrics efficiently so you have the right impact on your team and organization.

7 rules to track software engineering efficiently

1. Software metrics should be easily understandable.

They should be:

Simple and computableConsistent and unambiguous (objective)Applied with consistent units of measurementIndependent of programming languagesEasy to calibrate and adaptable

This is why software development platforms that automatically measure and track metrics are important.

2. Link software metrics to business priorities.

You can measure almost anything, but you can’t pay attention to everything. The more metrics you track at the same level of importance, the less importance you give them.

It’s important to select the few that are most relevant to your current business priorities. Measure only what matters now. You could have for instance P1 metrics, and P2 ones. Your team could be focusing on improving P1 ones while maintaining P2 ones. It all depends on your current business goals. This implies you could be tracking some metrics temporarily before achieving some goals. For instance:

Reducing the number of bugs reported;Increasing the number of software iterations;Speeding up the completion of tasks.

However, when developing goals, management needs to involve the software development teams in establishing them so everyone can help choose the most relevant metrics and then align behind those. You want adoption from the team for the metrics you track.

A last point, business success metrics drive software improvements, not the other way round.

3. Track trends, not just numbers.

When a software metric target is met, some teams will declare success. But having only a single point of data doesn’t offer much information on how the metrics are trending. It is the trends that will show you what effect any process changes have on progress.

4. Set shorter measurement time frames.

Most sprint retrospectives happen on a weekly or bi-weekly basis in agile teams. So your metrics need to be shorter than these time frames in order for your team to have enough data to iterate at each sprint, but also within each sprint.

5. Stop using software metrics that do not lead to change.

Comparing snowflakes is waste. Be thoughtful that the metrics you track can be acted upon, or the team will grow bored and uninterested by those metrics. Measuring them will just dilute the importance of the other metrics.

If you can’t change a metric that you thought actionable, think again if the issue is in the solution that you’re iterating on, or the metric itself that might need readjusting.

6. Metrics alone cannot tell you the story.

Metrics don’t tell you why the trend is this way or another. Only the team does. Metrics should always be only a discussion starter.

A lot of average managers make the mistake of evaluating contributors based on a set of metrics, without further discussion. Always strive to understand the ‘why’ by discussing with the protagonists, and only then will you know what the metrics actually mean.

7. Don’t use any metrics for individual performances.

Resist the temptation to make any judgement on individual performances based on metrics. Metrics help you make inquiries to understand what really happened and therefore better understand the intricacies of the project and managing a team.

The reality is that management is hard and always contextual. You need to dig deeper to understand the root causes of issues. Sometimes you will find that, indeed, a developer is actually a poor performer, but it’s because you made the effort to know your team and understand how they work together that you can identify if there is one member dragging it down. This is how you become a better manager who works towards better productivity and retention within your team.

As you can see, there are quite a few best practices on how to use software engineering metrics. I may have missed some. In this case, please share your own in the comments. There are still definitely too many organizations using metrics the wrong way, so the more we share best practices, the more it will improve things for all of us.

If you are interested in empowering your team with software engineering data to bolster better collaboration and decision-making, have a look at the software engineering management platform we are building at Anaxi that brings your systems, data, and people together for this purpose.

Originally published at https://anaxi.com on July 30, 2019. This article is the first part of a series of articles. Read more:

• How to Use Software Productivity Metrics The Right Way
• How to Use Data to Improve Your Sprint Retrospectives
• Software Quality: The Top 10 Metrics to Build Confidence