A software platform for organizations and developers to responsibly deploy, monitor, and get value from AI - at scale.
Containers and Microservices have become the most popular way for new software deployments, and for application re-factoring, such as twelve factor apps.
At Modzy, we use containers and Kubernetes for our entire platform. This blog covers choices, lessons learned, and how Modzy uses Kubernetes—and what that means for our customers.
The choice isn’t a huge factor when it comes to container orchestrators. Kubernetes, Hashicorp Nomad, Docker Swarm, and Marathon/Mesos are the primary tools in use today.
In addition to cloud provider managed offerings such as Elastic Kubernetes Service (EKS) from Amazon and Azure Kubernetes Service (AKS) from Microsoft Azure, companies such as Rancher and Red Hat/IBM also have their own Kubernetes offerings. These companies contribute heavily to the open-source community, which is a bonus.
So, why would you use Kubernetes vs. another solution? There are a number of factors we considered.
First, Kubernetes is the defacto orchestrator, with the largest open source community out of all the orchestrators. It is also the most supported in terms of vendors, features, and contributors. Kubernetes also scales very well, with good documentation, including potential issues that a user might encounter based on scale.
While Nomad and Mesos also scale well, they lack Kubernetes’ open source contribution and documentation. Additionally, out of all the orchestration tools, Kubernetes has the widest variety of options for deployment methods, tools, cloud provider support, tutorials, docs, guides, etc.
While deploying Kubernetes has its own challenges, the community support, managed offerings, additional tools, and vendor choices provided for the deployment process are much more robust than those of other orchestration tools. Deployment spans a wide range of industry verticals and customer use cases, and the managed Kubernetes offering offloads daily maintenance from our DevOps and SRE teams.
Kubernetes offers flexible deployment options, with many managed distributions for software deployment. Kubernetes offers support for everything from large data centers, all the way down to Raspberry Pis.
Kubernetes is open source, by default—meaning no licensing required. From a security perspective, Kubernetes has robust open-source tooling, as well as the option to add additional security via vendor tools. Tools like Vault can be used to secure credentials, secrets, and more. Open source products such as Falco do runtime security, allowing you to detect potential threats in real time.
In addition to the orchestration features, Kubernetes offers a number of application features and integrations.
We’ve learned a lot about Kubernetes in AI and MLOps in the last year at Modzy. We realized that AI and MLOps require good documentation for deployment purposes. And, scaling is different between a Web frontend and an AI platform. Model deployments require additional thinking around scale, resource use, and security.
Other best practices we follow for consideration:
What are your thoughts on scaling AI? Is it something you’ve done? We’d love to chat further and share our lessons learned with you.
Create your free account to unlock your custom reading experience.