The Ultimate Guide to Product Bugs [Part 2]

This is Part 2 of a comprehensive guide to product bugs. Part 1 explored what product bugs are, why they occur, where they come from, and why they do and don’t matter. Part 2 focuses on how to manage product bugs.

Most of us have been through the results of poor management of bugs: hair-on-fire, last-minute late nights to madly scrambling to critical fix bugs blocking a release, or stopping a customer from using a product. But it doesn’t need to be this way.

At its most basic level, sound management of product bugs will help avoid fire-fighting and restore sanity to your team—a baseline for developing and delivering tech products.

Taken to the extreme, management of product bugs can become you and your firm’s competitive advantage. Countless startups attribute early success in taking feedback from customers, fixing it fast and releasing. This builds trust. It’s the best form of customer service. The opposite, taking on bug reports, then doing nothing (or being perceived to be doing nothing) erodes trust. 

No matter where you sit or want to sit on the spectrum of managing product bugs, this guide will help you manage product bugs better. 

How to manage product bugs is broken down into the following sections:

1. Strategy 
2. Prioritisation
3. Triage
4. Prevention
5. Practices 
6. People & Resources
7. Governance, Reporting & Stakeholders
   

1. Strategy

Within your organisation’s and your product’s strategy, you need to work out how product bugs fit. This might seem a bit odd – strategic consideration of product bugs. However, suppose you slightly change the discussion to one about product quality, your competitive advantage, and the type of customer relationships you have or want. In that case, you’ll have almost any executive paying attention.

In times where the key-interaction between many organisations and their customers are digital products, the quality of those products and how those products grow or destroy relationships is worthy of serious consideration. 

In reality, rarely does an executive or strategy team set off on new research or a strategic direction by starting with “what’s our thinking on bugs?”. The organisation and product strategy is usually already determined by the time you’re thinking about bugs.

So your first practical port of call is to work out which elements of the broader strategy will apply to how you will manage bugs.

Some questions to cover here:

1. What is your release cycle? Some companies only release quarterly (often because customers say they don’t want updates more often). Some companies only think about one big release with patches. Others want to release as regularly as possible.
2. How much access do you have to customers? Does your organisation provide you and your team with direct access to customers, or are there gatekeepers? How difficult is it to get through the gatekeepers?
3. How does your culture handle mistakes? Are they something to be learnt from, or something to be avoided?
4. How willing is the organisation to invest in preventing bugs? Are budgets set aside for quality improvement? Are systems in place for automation (e.g. CI/CD)?

If your organisation isn’t where it needs to be, the Harvard Business Review provides a helpful 2×2 grid for grading your organisation’s approach to quality management, and a pathway for improving it.

Source: HBR

Once you’ve established the guard rails for your approach to product bugs, you have a framework to consider and execute on the other aspects of managing bugs.

2. Prioritisation

In many ways, bugs are no different to managing the other work you’re doing on a product. You need to prioritise what you’re going to do, and the best way to do this is by using a product prioritisation framework. 

You just reuse the product prioritisation framework you have or choose one. Then you work your known bugs through it alongside your new features and other activities. 

There is some subtly different treatment to how you work bugs through your framework depending on when the bugs occur in the product and development lifecycle:

• Bugs occurring pre-release: it’s best, if you can, to link the bugs’ prioritisation to the features you have planned for release and assess the impact of the bugs on those features. This will give you a sound prioritisation. However, you’ll likely also need to do a bit of an independent priority assessment of bugs. Additionally, if you haven’t met your original definition of done and the feature must make the release, then you just need to get it done. Don’t lose sight of focusing on value to the customer and business, not just “something isn’t working so we can’t release until it is fixed”. Sometimes this will mean reworking the roadmap (e.g. cutting features), especially when a bug reveals itself that derails what you had planned to release. Because you haven’t released yet, you can delay the release.
• Bugs occurring immediately or soon after release: in this situation, you want to take a similar approach to the point above. However, if something is in the market, you have an urgency that isn’t always there pre-release. You don’t want the bugs to negate the new value you’ve delivered. 
• Bugs occurring sometime after release: usually if a bug hasn’t shown up until sometime after release, you can put it into your backlog/roadmap using the prioritisation framework you’ve chosen, just like any other feature or activity. It is worth pausing to consider why the bug wasn’t found (is anyone using the feature?)?

Prioritisation like this means you’ll be able to keep focused, rather than feel like you’re fighting fires. You’ll also know you’re doing the most valuable thing for the business and the customer. Not all bugs need to be fixed or fixed perfectly.

Prioritising bugs together with features is one of the easiest and most impactful changes that most teams can make. This is because the biggest, and easiest to correct error most teams make, is prioritising their bugs separately to their features. You need to bring them together for a holistic approach.

For example, if part of your admin settings screen isn’t working, but you have a new feature due to increase signups by 20%, you may want to focus on the signups screen. You might have a hard choice (maybe the admin screen is blocking customers from paying you?). Your prioritisation framework will help you make this call.

I suspect one of the reasons bugs trip people up in this way is because they aren’t known in advance, and the details of new bugs cannot be easily predicted. 

However, this doesn’t need to stop you from preemptively prioritising bugs that don’t exist by allowing for them in your roadmap or work somehow. This can be by adding buffers onto items on your roadmap or having specific allowances for “fixing unknown bugs” in your roadmaps/plans. 

Some folks say buffers and allowances at a roadmap and sprint level are non-negotiable.

Failing to plan for failures (bugs), is a plan to fail. 

3. Triage

New bugs need to be triaged. Ideally, this will happen during your standard roadmap grooming. However, triage often happens when teams are under stress from a deluge of bugs. 

Side note: if you’re working with customer support, then the assumption here is that the bug has made it past them, and found its way to the product team. 

The essential steps to triage are:

1. Quick Pass – to qualify, categorise and clarify (if needed). Qualifying is about checking that the bug is genuinely a bug. Categorising is about ensuring you’ve marked it with the appropriate priority, part of the system and any other metadata you need. Clarifying is about asking for or finding more information if needed. This can be performed by one person but could be done in a workshop style meeting. You may remove some bugs here.
2. Prioritise – prioritise the bug using your prioritisation framework as quickly as possible. Sometimes this pushes the bug out of your immediate focus even though the bug might seem like a serious problem. It’s essential to prioritise before researching solutions or estimating. Something that is not a priority doesn’t need to be done, regardless of the effort involved.
3. Investigate – this is where you or a team member looks into what is causing the bug, how it can be reproduced, how it might be solved and how long it will take. Sometimes through investigation, the bug gets fixed.
4. Reprioritise – after investigating bugs, you’ll have more information. You might want to reprioritise.
5. Resolve – fix the bugs or close them, explaining why they don’t need to be fixed or aren’t a bug.
6. Repeat the above steps when new bugs come in, or you resolve bugs. Things shift. 

You might go through each step as its own discrete task, or you may just do them in one go or meeting. Be pragmatic.

4. Prevention

Part of planning for and managing bugs is determining how you will prevent bugs from happening or reduce their impact. At a minimum, your plan for prevention needs to recognise that you cannot completely remove bugs from occurring.

Preventing bugs from happening comes through a combination of practices, or a system, rather than any specific single activity. 

Prevention is also a continuous activity that needs constant revision, consideration and improvement. By regularly analysing where bugs are coming from, why they are occurring, and how they can be prevented, you can work to more generally prevent bugs from occurring in your product. 

5. Practices

The practices a team needs, if it’s to reduce the impact of bugs on products, customers, and the business, are:

I. Continuous Integration / Continuous Deployment (CI/CD)

II. Acceptance Testing and Regular Testing

III. Product Prioritisation

IV. Effort Based Estimation

V. Code Reviews

VI. Doing Definitions

VII. User Testing

I. CI/CD

CI/CD is an essential part of reducing the number and impact of bugs. CI/CD brings with it multiple features that directly, and indirectly deal with product bugs:

Automatically testing some or all of the product every time it is changed. An automated way to deploy a new release An automated way to remove a new releaseReduced dependence on manual steps. 

Each of these combines to mean less room for error, and when an error does get through, a way to solve it quickly.

II. Acceptance Testing and Regular Testing

This practice is about testing the software that has been developed. Testing helps reduce bugs having an impact on the customer. 

The TMMi Foundation provides a maturity model for testing (see table below). The more mature you, your team and organisation are on this model, the better you will be prepared to manage bugs.

V-model link mentioned in Level 3: V-model

There are also some common situations when it comes to the practice of testing that are worth mentioning here:

1. Expecting engineers to produce code without any testing taking place is doomed to fail. People make mistakes. Systems get complicated quickly; things break unexpectedly.
2. Expecting engineers and others on the team to always thoroughly test their own work is problematic. It’s also often a distraction. 
3. Testing must take place throughout development. Every sprint. Not just at the end.
4. Time must be allowed to develop automated tests. Especially if you plan to update and improve the product continuously, or the product is expected to have some longevity. 

III. Product Prioritisation

You’ve already heard about the benefits of prioritisation, but it cannot be emphasised enough. Effective prioritisation helps focus on the right bugs. This is going to mean a better impact.

IV. Effort Based Estimates

Bugs must be estimated (any work needs to be estimated for that matter). Estimations give you an idea of how and when those bugs will be finished. This means you can plan better and communicate effectively with customers/stakeholders.

Bugs must be estimated with effort because the effort is the only way to arrive at a duration that can be communicated to stakeholders in a form they will understand. No customer or senior stakeholder will understand “oh, it’s just 32 story points” or “it’s only three t-shirts big”. They will understand “it will be done in a day” or “it’s quite challenging and will take three weeks”.

If you still want to argue for story points for estimates, then read this first, and if you still don’t agree, I’m ok if you use them. An estimate is better than no estimate.

It’s surprising how often people don’t bother estimating bugs. “They’re bugs, and we just need to fix them, it’s urgent”. When it’s urgent is when you need estimates the most. The pressure is on, and you have expectations to manage. You may even need to relieve pressure. “We usually fix five bugs per day, so don’t need to look at them individually”. Historical numbers don’t help when looking forward because bugs aren’t created equally. You need to understand each bug. 

V. Code Reviews

Peer reviews and independent code reviews are another practice that helps reduce the number and impact of bugs. 

Peer reviews, including pair programming (for short bursts, not all the time), help engineers talk through the changes they’re making or have made. It also helps to allow a second set of eyes to pre-emptively spot any issues. For some good code review practices see Google’s approach to code reviews. 

An occasional independent review helps the team avoid any blind spots, and find ways to improve that they may not have been aware. 

VI. Doing Definitions

For many bugs, you need to put the time in upfront to define the requirements, designs, and solution to the bug(s) you are going to solve. 

Bugs that aren’t trivial (anything other than changing the button text from “Hello” to “Hi”) need some amount of definition. The definition is done before you develop. This means documenting and discussing what the requirements are and verifying this with stakeholders/customers. 

It’s cheaper to fix issues in requirements than after code has been deployed. For some reason, probably due to the urgency bugs often have, people have an urge to always jump in and start fixing bugs. Avoid this tendency. Sometimes people say “but we’re agile.” Agile doesn’t mean avoiding planning and best practices for shipping features. 

VII. User Testing

The final practice to cover that will help you reduce bugs is user testing. From the early concept days to the final release, involving users throughout development will reduce bugs.

6. People & Resources

At some point, you need to get bugs fixed, which will require people and resources (people are not resources!). When it comes to people and resources when managing product bugs, there are a few considerations when working out who will fix the bug(s):

1. Availability – people need time blocked to work on bugs. It sounds like common sense, but people are also expected to work on new features too often. Prioritisation can help here. For some products, you may also need to consider the availability of resources like devices, special hardware and servers.
2. Capability – you need to assess if someone has relevant experience to the bug itself, the system’s components, the technologies used and the context required. You may also decide to use the bug to train someone, as bugs are often a good way to upskill people.
3. Culture – your organisational view on bugs and bug-fixing should be that of learning and applying, rather than blame. ‘Bug’ is not a dirty word. Placing emphasis on your people experiencing bugs to fail, learn, apply and move on, rather than as a finger-pointing culture, can only strengthen your quality management strategy. 
4. Morale – consider how bugs and bug-fixing affects your product development team. This is closely related to the point above on culture. Encouraging positivity, action and learning during bug fixing will benefit your team.
5. Automation – by ensuring your good practices (mentioned in Section 5) and processes are set up, you allow your team to prioritise and fix bugs efficiently. Ensuring your environments are stable and good CI/CD practices are available is important during any product development tasks. It is especially important if the team is time-crunched, stressed about a bug and nearing release time.

7. Governance, Reporting & Stakeholder Management

Finally, when it comes to managing product bugs, you need to think about governance, reporting and stakeholder management. 

1. Who are the stakeholders interested in bugs? What do they each need? Identify who is interested in bugs. Customers? Executives? Sales? Customer Support? Other development teams? Then go through and look at what information each of the stakeholders need.
2. What frequency do you need to communicate with stakeholders? This may vary depending on the situation. For example, just before a major release, you may communicate daily with key-customers and management, but in between major releases you may communicate once per month.
3. What information do you need to send? Work out what information you need to communicate to stakeholders. That is, what reports you need to produce, and what needs to be included.
4. Who will be involved in prioritising bugs? How will they be involved? You may want key-stakeholders to help define a bug, whether it is important and what actions to take. You may also need their help in testing.

Hands-on: How to Manage Product Bugs

Now we’ve covered how to manage product bugs. The next parts in this guide on managing product bugs will look at example reports and working with stakeholders (think customers, customer support, sales, and marketing).

Previously published at https://terem.tech/blog/product-bugs-a-comprehensive-guide-part-2/