Companies that have pioneered the application of AI at scale did so using their own in-house ML platforms (uber, LinkedIn, Facebook, Airbnb). Many vendors are now making these capabilities available to purchase off-the-shelf products. There’s also a range of open-source tools addressing MLOps. The rush to the space has created a new problem - . There are now hundreds of tools and at least 40 platforms available: too much choice This is a very difficult landscape to navigate. But organizations have to figure it out because there is an imperative to get value from ML. Let’s understand the big challenges and then we’ll introduce some new free material that aims to address these problems. Challenge #1: Overwhelming Choice Chip Huyen gathered data on the . Chip found . Evaluations don’t have to consider all of these tools provided we can narrow them down to the ones that are most relevant to us... but that’s not easy to do. ML tool scene in 2020 284 tools and the number keeps on growing Challenge #2: Blurred Categories Typically we get a picture of which software does what by putting products in categories. There are attempts to do this with the , the , and . But often ML software does more than one thing and could fit into any of several categories. Platforms are especially hard to categorise as they explicitly aim to do more than one thing. LFAI Landscape diagram MAD landscape GitHub lists Because software that does several things is hard to categorize, ML platforms all tend to end up in a single category of ‘platform’. This obscures what emphasis each platform has and also loses all of the . nuances of how different platforms do similar things in different ways Challenge #3: Shifting Landscape ML categories are hard to keep track of in part because . Feature stores, for example, haven’t been around very long but are now a significant category. This affects platforms too as platforms are introducing big new features and changing their emphasis (in part in response to what new tools appear). new categories keep appearing Challenge #4: Complex Problems ML is complex. It’s also a big field. so that means understanding regression, classification, nlp, image processing, reinforcement learning, explainability, AutoML and a lot more. Platforms do a wide range of things Challenge #5: Range of Roles and Stakeholders Not only are the problems various and complex, there’s also a . Data Scientists, ML Engineers, Data Engineers, SRE, Product Managers, Architects, Application Developers, Platform Developers, End Users etc. Different roles have different points of interaction with the software and different needs from it. range of roles involved in the ML lifecycle Challenge #6: Build vs Buy and other controversies There’s a lot of discussion of build vs buy trade-offs in the ML space. How much control do organisations need over their stack? How much does that control cost? Build vs buy is often presented as an either-or but it is more of a spectrum. This is just one of the (consider how controversial AutoML is). confusing controversies in the ML space How do we get on top of all this? We’ve launched two new resources to help. For understanding the landscape and dealing with the trade-offs we’ve launched a Guide to Evaluating MLOps Platforms: https://www.thoughtworks.com/what-we-do/data-and-ai/cd4ml/guide-to-evaluating-mlops-platforms This is available for free without any sign-up. It addresses and . how to distinguish MLOps platforms how to structure an evaluation to suit the needs of your organisation We also need to apply this knowledge and see how to compare platforms against each other. For this we’ve released an open-source comparison matrix: https://github.com/thoughtworks/mlops-platforms The matrix is structured to highlight how vendors do things in their own ways and also point to more detail in the product documentation. We’ve also included in the repository a series of profiles that describe the product directions of popular platforms concisely and in a marketing-free way. We hope you find this material helpful and welcome contributions in GitHub. Feel free to ask any questions either on or to me directly on , . github twitter linkedin Title image by chenspec on pixabay.

AirBnB

Facebook

ThoughtWorks

Twitter

Uber

Writes about data, how to manage it, how to do AI with it, how to make data-driven products. Principal Data Consultant at Thoughtworks and 2020 HackerNoon Contributor of the Year - Engineering.

2021 - HackerNoon Contributor of the Year - DATA

2022 - HackerNoon Contributor of the Year - Agile

2022 - HackerNoon Contributor of the Year - Agile Software Development

2022 - HackerNoon Contributor of the Year - Digital Transformation

2022 - HackerNoon Contributor of the Year - Innovation

2022 - HackerNoon Contributor of the Year - Kubernetes

2022 - HackerNoon Contributor of the Year - Machine Learning

2022 - HackerNoon Contributor of the Year - Monitoring

2022 - HackerNoon Contributor of the Year - Product Management

2021 - HackerNoon Contributor of the Year - SCRUM

Nominated for 2022 - HackerNoon Contributor of the Year - Innovation

Nominated for 2022 - HackerNoon Contributor of the Year - Scrum

Nominated for 2022 - HackerNoon Contributor of the Year - Agile

Nominated for 2022 - HackerNoon Contributor of the Year - Digital Transformation

Nominated for 2022 - HackerNoon Contributor of the Year - Kubernetes

Nominated for 2022 - HackerNoon Contributor of the Year - Machine Learning

Nominated for 2022 - HackerNoon Contributor of the Year - Agile Software Development

Nominated for 2022 - HackerNoon Contributor of the Year - Product Management

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

How to Evaluate MLOps Platforms

How to Evaluate MLOps Platforms

Too Long; Didn't Read

People Mentioned

Companies Mentioned

Ryan Dawson

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

How to Evaluate MLOps Platforms

Too Long; Didn't Read

People Mentioned

Companies Mentioned

Ryan Dawson

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES