How to Manage Machine Learning Products [ Part II]

Best practices and things I’ve learned along the way.

Part I: Why is managing machine learning (ML) products so hard? And why should you care?

In Part I, I talked about basics and challenges in managing ML products. Developing ML products involves more experiments and iterations.

Therefore, as a PM, you need to give engineers and scientists enough space and flexibility to explore before deciding on the path going forward.

But how do you help your team navigate these uncertainties? How do you go about defining the business problems and metrics while allowing your team to explore?

1. Planning: Start with defining the problem well.

ML is a tool, a means to an end. Don’t build an ML product if the problem you are solving doesn’t require ML. Start with identifying the problem, the customer pain point that is in great demand (business potential) and solvable (technical feasibility). You can do a market sizing exercise to estimate the business potential. Then the next question is: how do we know if ML can help address our user problems. There are numerous ML applications but, to the core, ML is best suited for making decisions or predictions. We can categorize ML applications into a few types.

Detection/Inspection: help users identify where defects or anomalies are, for example, fraud detection in banking or insurance or defect detection in a production line.

Pattern Recognition: help users sift through massive amounts of data. Examples include recommendation, ranking, personalization, classification, predictive maintenance, clustering, and interaction with humans (e.g. natural language processing (NLP) for smart speakers such as Alexa or Google Home).

High Dimension Cognition: help users sift through massive amounts of high dimensional sensory data. Examples include AI-enabled robotics and self-driving vehicles.

You should avoid using machine learning in products if:

• You can solve the problems with simple rules.
• The solution you are building doesn’t need to adapt to new data.
• You can’t get access to the data you need for training ML models.
• Your product requires high accuracy.
• You need full transparency in how your product works.

Once you find the right problem to solve, the next critical task is to clearly define the requirements. Developing ML products is a highly iterative process. It might be tempting to skip proper planning and dive right into seeing what models can do. However, if you do so, you will likely end up wasting lots of time with no concrete results.

2. Define the objective function (outcome) and metrics. Allow more space and flexibility.

For ML products, humans don’t program the rules; machines do. It is more experimental and has to be treated differently than the conventional divide-and-conquer software engineering approach. Often times it’s hard to predict what will work and what won’t work. That’s why it’s important to give engineers and scientists more room and time to explore before deciding on the path going forward.

As a product manager, you can help your team stay focused during such an extensive exploration process by:

(1) Define an objective function: what’s the desired outcome that your model is trying to predict? Or are you trying to identify patterns in data? Are there any “ground truth” that you can compare the outcomes of your models to? For example, if you design a model to predict the weather, you can validate the performance of your model by comparing the forecast to actual weather data.

(2) Define performance metrics: How do you measure the success or failure of your products? It’s not always straightforward to set acceptance criteria. For example, how do you measure the performance of a translation model against a human translator? Sometimes, you will need to see the initial results of the models and then decide on the criteria. But it’s important to take test criteria into consideration early on and constantly test the model until you find the right model that delivers satisfactory results.

(3) Test the models early and frequently from end to end: You can think of ML models as black boxes. You define inputs and outputs that you want your model to generate without necessarily understanding what’s going on in the black box. That’s why it’s important to build end-to-end prototypes and test the models early and frequently whenever possible. Start with simple prototypes that can help you test key functionality and then iterate on it. Avoid starting with a comprehensive end-to-end solution at all costs.

One important thing to note is that model accuracy by itself is usually not a good measure. Instead, consider measuring precision (true positives/all positive predictions) and recall (positive predictions/all true positives). As Wikipedia explains it: Precision is about how many selected items are relevant and Recall is about how many relevant items are selected. There’s no rule of thumb that would apply to all situations. You need to decide on the tradeoffs based on your business cases.

3 Think about your data strategy from day one.

Training ML models often requires lots of high-quality data. Deep Learning outperforms older algorithms when it’s trained with a large amount of data. Hence, it’s extremely important to outline your data acquisition strategy from day one. You can buy data, partner with other companies, gather data from your customers, generate data internally, or hire a third party to generate or label data for you. You need to consider what your competitors do, what your customers and regulators think, as well as corresponding feasibility and cost for each of the strategies. It’s not the responsibility of data scientists to figure out your data strategy. It’s a strategic business decision that product managers, executives, and key stakeholders need to define.

If you are a startup, think twice about the sector you want to enter. Are there industry giants dominating most of the data? For example, you probably don’t want to compete with Amazon on purchase history or Google Maps on location data. Try to find some niche market that no one company dominates the market yet.

Are you able to build a defensible and sustainable data pipeline? How about compliance with your users’ privacy policies? Familiarize yourself with GDPR (General Data Protection Regulation) if your company operates within the EU and other data protection regulations. For example, under GDPR, companies need to make sure that personal data is not only legally gathered but also protected from misuse or exploitation. Therefore, as a PM, you need to take data protection safeguards into consideration from the early stage of product development.

Make sure that you discuss with your ML team to figure out what and how much data you would need. Involve other stakeholders like legal and operation, too. Developing ML products for the physical world such as robotics and self-driving cars poses even more challenges. Make sure that you leverage simulation and pay attention to research areas including sim to real, data augmentation, transfer-learning, and meta-learning for ways to bring down your needs for massive data and accelerate your training process.

4 Think beyond ML.

In most cases, you are actually building more than an ML product. To make it a complete and production-ready product, you need a user interface, software to execute model predictions, and/or hardware components. You won’t have a successful product if you focus too much on building the ML model and overlook user experiences, for instance. You need a multifunctional team including not only ML engineers and scientists but also data engineers, software engineers, UX UI specialists, and/or hardware engineers. You also need to work with backend engineers to ensure that the infrastructure is there to support the ML team.

Try to minimize the interdependencies and clashes between different functions or teams. As previously mentioned, the nature of ML makes it radically different from conventional software programming. For example, while daily stand-up meetings might be useful to keep the software engineering team productive, it might not be the best practice for the ML team. That’s why ML likely requires not only technical but also organizational changes. As a PM, you can help other teams understand why and how building ML products is different and help address potential conflicts. You should also try to communicate uncertainties inherent to ML to other stakeholders in your organizations.

Communication is critical not only with internal teams but also with customers. The performance of ML products improves over time. It’s likely that customers will not get the perfect results in the beginning. Is this acceptable for your users? How to mitigate the risks for your users and guarantee an acceptable baseline performance? How to design your product to optimize user experiences?

5. Make the case for investing in ML.

If you want to make the case for investing in ML features, consider the following:

Enhance user experiences or product functionality: Can ML be used to personalize or customize your products so it’s easier for your users to find the most relevant results, for example? Or can ML be applied to increase the accuracy of your forecasts or predictions? Consider potential applications of such for both internal users and external users (customers).

Automate process or repetitive tasks: Is there any process that employees in your company or your customers need to go through repeatedly that can be automated? By automating repetitive tasks, you can save time, cost, and resources but also potentially create better user experiences. If the process is too complicated, is it possible to automate part of the process or help humans to do things more efficiently? Gmail’s Smart Compose is a good example: instead of having users typing the same words or sentences like “best regards” every time, Gmail can now finish your sentences automatically.

Open new business opportunities: Are there any new opportunities or business problems that were unsolvable but now seem solvable with ML? For example, the piece picking process is highly manual in warehouses because it’s not feasible to program robot arms to recognize and handle millions of products. But now with ML, robots can learn to recognize a wide variety of objects with minimal supervision from humans. And that opens up tremendous opportunities for AI-enabled robots in warehouses.6.

Where good PM instincts go bad for ML products

Sometimes best practices for managing software products don’t necessarily apply to ML products. Here are a few things I usually remind myself of:

• Recognize the differences between developing ML and software products. There’s no one size fits all process. Adjust your sprint process, planning, or organization whenever needed.
• Instead of detailing all requirements on your PRD, focus on defining objective functions and key performance criteria, allowing the team to explore and experiment.
• Rather than asking your ML team for deterministic results at the beginning of the development process, work with the team to develop and test end-to-end prototypes early and often.
• ML is only one of the tools. Don’t use ML if you don’t have to.

Summary — here’s what I want you to remember about this series of articles:

• ML is best suited for making decisions or predictions.
• Managing ML products is more challenging than managing normal software products because it involves more uncertainties and requires not only technical but also organizational changes.
• Clearly define the problem, scope out requirements, set the metrics, and give engineers and scientists enough space and flexibility to explore before deciding on the path going forward.
• Think about your data strategy from day one.
• Building ML products is interdisciplinary. Think beyond ML.

Connect Deeper: If you enjoy reading this article, please subscribe to my personal blog here!

Bastiane Huang is a Product Manager at OSARO, a San Francisco based startup building software-defined robotics. She has worked for Amazon in its Alexa group and with Harvard Business Review as well as the university’s Future of Work Initiative. She writes about ML, robotics, and product management. Follow her here.