Using MLflow and Modzy for Automated Model Deployment

As the world of artificial intelligence (AI) and machine learning (ML) continues to grow, the demand for leveraging AI capabilities is slowly becoming overshadowed by the issues that organizations face with deploying and operationalizing AI capabilities.

The combination of strong data science skillsets and ML development tools allows organizations to build optimal, efficient, and reproducible ML pipelines that produce state-of-the-art AI technologies.

On the deployment side, there are tools and platforms that exist to support the operationalization and management of these technologies at scale. Unfortunately, these ML development and deployment tools often suffer compatibility issues and cannot interoperate, resulting in a fragmented ML lifecycle.

What if there was a way to integrate these two stages of the ML lifecycle and decrease the average time from model development to deployment (which according to Gartner, takes on average, nine months) to only a few hours? Here at Modzy, we created an end-to-end integration that uses MLflow (a popular tool for ML training, tracking, and logging) to train an ML model and then automates the deployment process to the Modzy platform, thus creating an automated model deployment pipeline.

Figure 1. Due to technological and organizational limitations, many companies support model development and model deployment as independent functions (A). This new MLflow-Modzy integration unifies these two stages to give Data Scientists full access to the complete model lifecycle (B)

Modzy MLFlow Integration – Key Components

MLflow

Data scientists around the world leverage MLflow to enhance and streamline their experience during the ML model development process. As an example, consider a data scientist who wishes to log performance metrics and monitor them in real-time during training experiments.

Specifically, this data scientist might want to train an image classification model and log the accuracy on a validation dataset while testing different combinations of hyperparameters. MLflow makes viewing and logging these metrics and parameters very easy with its user interface (UI), providing tables, graphs, and experiment details as training executes in real-time.

Upon conclusion of this experiment, the data scientist will choose the set of hyperparameters that achieve the highest accuracy to tackle the challenge of moving this powerful model out of the lab and into production.

Modzy

Modzy makes the deployment and scaling of AI capabilities across an organization not only possible, but also streamlined and simple for data scientists. Once an ML model is trained, tested, and fine-tuned, Modzy provides a template to package these models in a Modzy-compatible Docker container.

This process allows data scientists to save their model’s Docker image to a registry and upload the model to the Modzy platform, thereby unlocking endless potential for accessing and using AI capabilities through its powerful application programming interfaces (APIs) and software developer kits (SDKs).

The Modzy team realized the significant value an end-to-end automated pipeline would bring to data scientists who already leverage MLflow or other common development tools (e.g., SageMaker, H20.ai, Neptune, WandB, Comet.ml) and produced an integration that:

• Trains a model, optimizes its hyperparameters, selects the best-performing model, and saves its model artifacts—all with full access to the native MLflow platform features (UI, logging, etc.)
• Sends these model artifacts to a model converter that plugs them into a ready-made repository (repo) and builds a docker image that conforms to the Modzy API specification, preparing the newly minted model to perform inference
• Deploys the model image and static metadata for the model to Modzy’s platform in order to make it available for use via Modzy’s marketplace

Figure 2. Schematic showing the logical flow and components of the new MLflow integration, from a local developer to a machine learning model on the Modzy platform

Model Trainer

MLflow allows users to run as many training experiments as they wish—all while tracking the performance of different sets of hyperparameters. This Model Trainer portion of this integration gives the model developer the opportunity to test several hyperparameter sets during the training phase to find an optimal, best-performing model.

Model Converter

The output of a model training experiment is called a model artifact. Model artifacts can vary depending on the model, use case, or framework leveraged in training.

Most commonly, model artifacts consist of model weights, labels, or any other configuration files required for the model to perform inference. In this integration, the model converter feeds the model artifacts (raw output of the Model Trainer) into a pre-built Github repo structured in a Modzy-compatible manner.

Using this repo’s Dockerfile, the model converter then builds the Docker container image—which contains the model’s code and artifacts—and pushes this container image to a Docker registry.

Model Importer

After the model converter converts the model repository to a valid Docker container image, the model importer takes two inputs before uploading the model to Modzy’s platform:

• The container image
• Tar archive that contains model metadata

The model metadata archive contains all marketplace information for the model, including technical model details, performance metrics, training information, images, and real-life usage examples.

Value of Automated Model Deployment

Ultimately, this integration takes a model from its experimentation phase and yields a production-ready model accessible via Modzy’s APIs and SDKs, expediting the time it takes to make this model available across an organization.

The open architecture nature of Modzy’s platform makes this development-to-deployment automated pipeline adaptable to other ML training tools. Our solution opens the door to many new MLOps pipelines and decreases the amount of work required for data scientists and software developers to deploy models into production at an enterprise scale.