The Engineer's Handbook for Effective ML Product Roadmapping

As an ML engineer, you've probably found yourself in a situation where there's no Product Manager on your team or the PM just doesn't have a solid understanding of ML and can’t handle this part of the product planning.

I’ve experienced this multiple times both in small and large companies. While it may seem ineffective for an engineer to handle product work, which obviously can be done better by an expert, I personally love the challenge of expanding my horizons and taking on this critical responsibility.

But before we dive into how to write a roadmap, let’s agree on whata roadmap is. Essentially, a roadmap is a detailed plan that outlines the projects your team will be working on over the next X months. Each project should have a clear description with motivation, a priority level, and a rough estimate of the effort required.

Keep in mind that we're focusing on theproduct roadmap here, not the technical roadmap. So, we won't be covering infrastructure, reliability, or tooling improvements.

By "product" I mean the actual output of your ML team. It could be a direct user-facing application, such as a feed ranking system, or something that serves as input for another team's product, such as credit scoring in a borrower assessment pipeline.

Roadmapping

Now that we have defined our goal on a high level, let’s discuss how to get there.

Start with a Contract

First off, it's important to consider who the roadmap is for.

The target audience can include:

• Your team, so that you stay aligned on the long-term goals;
• Your team’s PM if they don’t have a strong understanding of ML;
• Other teams and their PMs, for example, those who use your products;
• Senior managers, who want to understand the value your team brings.

Depending on the audience, the level of detail required may vary. Senior managers might be interested in a long-term vision, while PMs may want to know about upcoming projects.

It’s very important to define the roadmap stakeholders and agree on the following:

• How are the stakeholders going to use the roadmap?
  • How often?
  • What problems do they aim to solve with it?
• What is the planning horizon they are interested in?
• What is their tolerance to changes in the roadmap (e.g. adding and removing projects or changing their priority)?

Pro tip: Log all decisions in a separate block at the end of the roadmap, so you won't have to revisit the same questions multiple times.

Interview & Brainstorm

What should our team focus on during the next X quarters? How can we improve our product in the most impactful way? These are the questions we need to address at this stage. While doing this, it’s important to keep in mind that our team operates within a larger organization and our plans should be aligned with stakeholders’ objectives.

So let’s interview the stakeholders about their problems and plans to make sure we’re all on the same page.

I would suggest having two meetings with each stakeholder:

1. The goal of the first meeting is to share the context we and they are working in. Ask about their upcoming projects and the problems they are facing now. Discuss their major accomplishments and setbacks in the previous year. Ask them to consider what they would do if they had access to 1000 individuals who could work with data at no cost (this is an approximation of what ML can achieve). Describe the products your team already has and the ideas you have for the future.

   1. To make the meetings more efficient I usually ask stakeholders to send over their roadmap in advance, so I could take a look and ask more targeted questions.

   2. It’s important to take notes of the key points discussed at the meeting and finalize the notes afterward.

2. The goal of the second meeting is to discuss specific ideas on how your team can help solve the stakeholder's problems.

   1. This meeting requires preparation: you can brainstorm some ideas on your own, but it’s usually much better to get the whole team involved. Remember that there are no bad ideas, but some of them may involve higher or lower impact, effort, or risk than others, so it’s helpful to rank them accordingly.

   2. Ask the stakeholder to brainstorm some ideas before the meeting as well, based on the inputs you offered during the first meeting.

   3. Be prepared that not all meetings will result in collaborative projects. Understanding the values of the stakeholder by identifying the projects they don't want to pursue can also be a valuable outcome. Just make sure to document these insights.

Pro tip: It can be tough to fully grasp the context and come up with the best ideas in just a couple of meetings. That's why it's a good idea to schedule monthly check-ins to gradually gain more context over time. The question which leads to a fruitful discussion 100% of the time is "What problems are you currently facing?”. You might not come up with new project ideas each time you ask it, but it will help you gain a deeper understanding of the stakeholder's work context, and ultimately lead to better ideas.

Create the Roadmap

At this stage, we should have a list of ideas — potential projects — that we’ve gathered from the stakeholders and the team. First, we need to prioritize them: this can be done either by the product manager or collaboratively by the whole team.

The body of the roadmap should list the projects and include the following information:

• What is the project about? This is a general description of the project, including the current situation (avoid going into ML specifics, provide high-level descriptions instead);
• What is the value this project brings? State the motivation and expected outcomes;
• Priority of the project;
• The effort required to complete the project;
• Optionally, you may want to include potential challenges that the team could face during the project execution.

Other essential sections that should be included in the roadmap are:

• Information about the team and their work, including examples of products they have developed;
• Principles for roadmap writing, which summarizes the decisions made (see the "Come up with a Contract" section);
• Optionally, a decision log.

Collect Feedback

Once the roadmap is finalized, share it with your team members and the PM, and ask for their feedback. Be prepared to get a lot of suggestions on how to improve it. Don't worry too much about it, though.

Here's how I deal with that:

• Evaluate the suggestions against the contract: remove the ones that don't align with the earlier agreements, or reconsider the agreements;
• Request feedback that the team believes will make a substantial difference in the value of the roadmap (they will need to justify it, of course);
• Make any necessary changes according to the remaining suggestions.

Maintain the Roadmap

The roadmap is not a one-off project and the team should keep it updated. Discuss the schedule of updates with the team. Consider assigning the roadmap update task to a different person each time. This way, everyone has a chance to make an impact and contribute their ideas to the project.

Conclusion

Writing a roadmap is not just about documenting plans; its true value lies in staying aligned with stakeholders and generating ideas collaboratively. Having a clear agreement around the principles and values of the roadmap (the contract) can save you days of repetitive discussions. Follow the suggestions above and you’ll be able to create a valuable artifact for your ML team!