Developing Software Quality Metrics as a Data Scientist - 5 Lessons Learned

This list is a result of my own reflections and advice from my colleagues on developing software quality metrics. Software quality metrics are essential tools in ensuring a product provides the best experience to its users. I had to develop such metrics during my internship as a program manager. Coming into this world as a Ph.D. researcher and data scientist was quite tricky. Here are the tips.

1. Data in software is not normal
2. Software quality metrics need to measure every step
3. Statistics is great, but ...
4. Share your work in progress
5. Think of all the users

1 Data in software is not normal

The first thing I noticed when I started looking at how software quality is measured was the data distribution, i.e. the shape of data. Let's look at how long it takes to send a message from one phone to another, i.e. message latency.

Your intuition might tell you that there is some standard time like 1 second, and then some people get the messages earlier, and then some get them later. This would suggest something akin to a Normal distribution (see the image above). Calculating a mean could seem like a good metric to measure the quality if the latency data were normally distributed.

However, the data in software tend to have longer tails on the right side. Consider the message example. Most of the messages are delivered to the receiver in under 1 second. Still, a few of them can take longer, and the specific times can vary from 2 seconds to 24 hours. Such data could follow Pareto distribution (different from the  80/20 rule or a Pareto Principle) well known in quality control applications.

Can we use this fact in developing software quality metrics?

Thinking in Pareto distribution directs our attention to the tail and to consider all our users, including the unlucky ones. So we focus on the less likely user experiences instead of the middle corresponding to a most likely experience.

This means that we would calculate higher percentiles or quantiles rather than calculating the mean message latency. For instance, the 95th percentile would describe the worst measured experience for 95% of all users. Then we can assess whether that's acceptable.

I would recommend this video for clarification:

https://www.youtube.com/watch?v=IFKQLDmRK0Y

2 Software quality metrics need to measure every step

When it comes to software quality in the context of product management, it's easy to slip into thinking mainly about the front-end. The specific parts of the front-end clearly impact the user experience.

After all, it is what the user can see and interact with. However, the software quality metrics for the back-end shouldn't be an afterthought. Focusing on individual parts of the back-end can lead you to discover opportunities to improve the speed of the processes. As you probably know from your own experience as a human using technology, we hate to wait, so speed improvements are good.

3 Statistics is great, but ...

I know. This sounds weird coming from me after the longest point in this story is about statistics. That should tell you how difficult it was to transition from an academic to a business context and from data science to product management.

When I started developing the software quality metrics during my internship, I tried to find ways of measuring the software quality that would be computationally efficient, yet precise. I couldn't help it.

Then I showed this to my colleague, and as I explained how it would work, he stopped me and asked me to reflect on how long it takes to explain what it is. I had to ask myself, is my proposal better than more standard approaches in all aspects? It was super fun for me to try to develop something new. However, then there was the communication aspect. Will this make sense to the senior management?

It's not to say that you can't use more complex statistics and calculations in business, but even Occam's razor teaches us to prefer simple over complicated. Convoluted algorithms for estimating software quality can be cool to develop, but they are useless when most people don't understand the output.

If someone asks you to paint a portrait, they won't expect a Picasso painting.

4 Share your work in progress

It was amazing to see how easy it was to collaborate with people and how much value they provided to the overall process. In most school systems worldwide, you are often assessed on your individual outputs. There are some collaborative projects here and there, but most of us learn to rely on ourselves.

If you've ever tried to learn something difficult, you know that the best strategy is to just stick with it and try until you get it right. This perseverance can sometimes hinder our progress in getting things done because we might not think of collaboration as a tool.

Surprise, surprise; businesses want to achieve the results as soon as possible. Leveraging the knowledge of your colleagues and collecting feedback on your early ideas is the best way to come up with software quality metrics that are well defined and useful in decision-making.

5 Think of all the users

Software quality metrics can measure different layers of quality. We can point the attention towards the function of the product, user experience but also technical aspects. Software is complex, and very often, the product users or the end customers are not the only users you should think of. If you are working on a back-end service, then the people working on the front-end are your users.

So your software quality metrics should consider your user groups and how far they are from the piece of software you are working on. Consider what is the impact on the specific user groups when your metric shows improved or worsened performance.

Software quality metrics and data science

Product management and data science have some overlaps, and one of the important ones are metrics for software quality. If you are thinking about pivoting from your data science career, be sure to pick the right tools from your data science toolbox.

As a data scientist, you usually focus on coding and statistics, and good communication is a huge plus. When you work as a technical product or program manager, the importance of those skills is reversed, and communication and collaboration become crucial.

If you find yourself developing software quality metrics, I would also recommend reading about the SixSigma critical to quality approach.

Good luck!