Your Kickass Machine Learning Algorithm May Ruin Your Product

While the excitement around Machine Learning is not unfounded, not all of the progress necessarily translates into improved products. This article lists a few examples of the most common product cases where ML is likely to be a poor fit and suggests a process to avoid such failures.

‘Machine Learning can transform your business’ is a phrase we’ve been hearing more and more of these days and it makes sense for every business to want to invest their time and effort in it.

Over the last year I’ve had the fortune of working on products that have machine learning (hereafter referred to as ‘ML’) algorithms at their core. I’ve interacted with other product managers, data scientists and engineers who all seem to be fascinated by the possibilities that ever-improving algorithms are bringing to the fore.

While this excitement is not unfounded, not all of this progress necessarily translates into improved products. This article lists a few examples of the most common product cases where ML is likely to be a poor fit.

Source: memegenerator.net

Before we dive right into it, it is important to distinguish an ML algorithm from an ML product. Needless to say, the algorithm is just a part of the product, and both are measured on completely different scales. A good ML algorithm may be identified by mathematical measures such as high precision and recall. Good products on the other hand are commonly measured by their ability to profitably deliver a value proposition to the user.

Common product cases where ML fails

1. Non-sticky products & Non-power users

The phrase, ‘A machine learning model is only as good as the data behind it’, has been overused and rightly so, since it also applies to all machine learning products.

While every product would be happy for its users to engage with it 24x7, depending on the nature of the product, this may simply not be possible. Even for products that could be sticky, there are often user-segments who use the service as a one-off. Such domains and use-cases are extremely ill-suited for any ML applications.

Take the example of a healthcare app that allows users to book doctors’ appointments. While patients with chronic medical conditions may use this app frequently, the average guy who falls sick a couple of times a year, would have barely any data stored with the app.

Over-eager clustering algorithms may find that people who visited a cardiologist also visited a heart surgeon. However, a ‘recommended for you’ pop-up showing you a heart surgeon the next time your stomach wasn’t doing well, perhaps just because you previously booked a cardiologist appointment for your grandma would certainly give you a massive heart-ache.

2. Ads are a major revenue source

Let’s take the classic example of a traditional online news website. The typical user flow would look something like this:

Traditional Flow — No Recommendations

• User visits the website homepage from google or by typing the URL into the address bar
• Navigates to a top-level section page
• Chooses the relevant sub-section page
• Clicks on the relevant article link to view it

Let’s suppose, the website came up with an incredible ML algorithm to recommend personalized articles to their users and added a section linking to them on their homepage. The common flow might then look like:

New Flow — Recommended Articles on Homepage

• User visits the website homepage
• Clicks on the relevant article link to view it

That’s it. Sounds pretty awesome, right? By cutting two steps in the user-flow, the user experience has been improved greatly, however the corresponding business consequences will more likely than not be quite catastrophic. Consider the following:

• The primary revenue source for most online news websites is advertisements displayed on the website to the users
• The category and sub-category pages are two high-traffic pages which get a lot of page views themselves and often lead the user to click through to more than one article
• Removing these two pages from the flow will see page views and ad impressions decrease leading to significant revenue-loss

3. Usage is an end-user KPI

‘The more the merrier’ is a common fallacy in human thinking and at times tends to conflict with the core value proposition of improved algorithms. Volume of manual actions or even the amount of time spent on doing something can be a KPI for success for certain users.

Source: memeshappen.com

Almost any enterprise SaaS tool could be a good example of this. For now, we can take the example of a recruitment website used by companies to source candidates.

The end users are employees in the recruitment function who typically conduct searches for candidates, view CVs/profiles and contact them. The ultimate success measure is the number of successful hires made. But hiring is a lengthy process, and feedback on its outcome comes with a lag. Hence, as an immediate proxy, employers tend to monitor the performance of their employees by measuring the volume of activity such as CV searches and CV/profile views on the website.

Source: memegenerator.net

This also extends to their opinion of the overall effectiveness of the recruitment website itself. The website with the best text learning algorithms, which enables recruiters to find the right candidates in the matter of few clicks may not always find itself most popular with employers since it generates lesser volumes of searches or views. This might see the website be junked in favour of a peer in this highly competitive space.

This list of common examples where ML fails could easily be made much longer but I suppose you get the general idea by now. The real question then is, how should you actually approach machine learning products? I’ve made an attempt to answer this question at a high-level below.

The way to approach ML Products

The key step that most people tend to miss is that we need to think about ML in terms of business outcomes rather than fancy algorithms or applications of data. Below is a 8-step approach to achieving your desired business outcome with ML.

1. Understand ML use-cases: If you are new to machine learning, inform yourself of the business problems that ML can actually enable you to solve, you may find this link useful as a starting point.
2. Define a clear business objective: Understand the specific ML use-case that applies to your business and would deliver value if you could enhance or automate it. Spend time to clearly define the problem statement you are trying to solve.
3. Assess data availability: You may want to do a lot of things with ML but if you don’t have the right type of data or enough of it available, you will end up failing despite your best efforts. This may be the right time to take a step-back and actually start collecting the right data as per your needs.
4. KPIs, KPIs and more KPIs: Identify the KPIs that you will track to determine the success or failure of your product. Include any relevant engagement metrics such as page views or time on site since they may get affected because of your product. Don’t forget any critical business KPIs that are relevant since a miss here could make or break your product.
5. MVP-first, beta-first approach: Data from actual users is subjective and much more valuable than the outcome predicted by your analysis. Define an MVP that you can roll-out with low effort and learn from. This, by definition rules out spending 6 months training and re-training your model. Roll-out as a beta first to power users who you don’t risk losing completely if the output were to be bad.
6. Get qualitative feedback: Once in beta, leave ample ways for your users to give you feedback on what they see. Proactively reach out on phone calls and emails to gauge how happy people are with your product.
7. Analyse with an unbiased mind: Evaluate the performance of your KPIs, combine with qualitative insights from interviews, segmenting users as necessary. Decide whether a full-rollout makes sense at this stage without being biased about the effort already put into the task.
8. Roll-out and monitor: If all the above boxes were ticked positively, go ahead with a full-rollout. Keep monitoring KPIs even after launch to ensure business risks are minimal and caught in time.

There are some great posts out there which expand on this approach and I would highly recommend that you also check out this post before starting any work on ML for your product.