Kubeflow is a modern solution to design, build and orchestrate Machine Learning pipelines using the latest and most popular frameworks. Out of the box, Kubeflow ships with MinIO inside to store all of its pipelines, artifacts and logs, however that MinIO is limited to a single PVC and thus cannot benefit from all the features a distributed MinIO brings to the table such as , unlimited storage via - and so much more. Active-Active Replication Tiering In this blog post we are going to configure Kubeflow to use a large MinIO Tenant on the same Kubernetes cluster, but of course, this configuration applies to Kubeflow and MinIO being on different clusters as well. For your reference, please see our earlier blog post, , and the site. Machine Learning Pipelines with Kubeflow and MinIO on Azure Kubeflow While we go from soup to nuts in this blog post, if you already have a Kubeflow setup and a MiniO setup, you can skip straight to the section of this blog post to see what needs to be configured. Configure Kubeflow Setting up the MinIO Operator Let's start by installing the MinIO Operator and creating a tenant that Kubeflow will use. My favorite way to install MinIO Operator is via but we also have Helm Charts available, and we are also available on the , and . kubectl apply -k AWS Marketplace Google Cloud Marketplace Azure Marketplace kubectl apply -k github.com/minio/operator/ This will install the latest and greatest MinIO Operator, now we just need to log into the Operator UI and create a tenant. For this step we'll get a to login, but   this UI can also be secured with AD/LDAP or OIDC. service account JWT token kubectl -n minio-operator  get secret $(kubectl -n minio-operator get serviceaccount console-sa -o jsonpath="{.secrets[0].name}") -o jsonpath="{.data.token}" | base64 --decode && echo "" Now let's port forward the UI and login. kubectl -n minio-operator port-forward svc/console 9090 Now open a browser, go to and login with the JWT token we got on the previous step. http://localhost:9090 After logging in, click on and set up a 1TiB tenant. Create Tenant Enter the name of the new tenant and the namespace for it. If the namespace doesn't exist you have the option to create the namespace. Now let's size the tenant. I'll be setting up a that has 4 drives on each node, in this case, because we’re on Kubernetes, node or server translates to pods and drives per server translates to PVCs per pod. 4 node cluster I'm also starting with 1TiB of capacity but you can always . expand the capacity of the tenant Let's go to and create a basic user that will be used by Kubeflow. If you choose to configure an that uses OpenID or Active Directory/LDAP, you can just go ahead and create a service account after you log in to the tenant. Identity Provider external identity provider Lastly, we'll disable just to keep this blog post from getting too long, but if you want to have TLS enabled on your tenant, you'll need a certificate configured on the tenant that Kubeflow trusts. TLS And that's it, just hit and the tenant will be created in a few minutes. Create That's it, now you have a distributed, high performance, hyper scale object storage that can be expanded endlessly. From here, let's configure Kubeflow to use this MinIO deployment. Setting up Kubeflow In this section, we'll set up Kubeflow from scratch on Kubernetes. This works for on-premise deployments, development environments or any public cloud, although cloud providers frequently  offer a pre-configured version of Kubeflow. We'll be using the repository. Bear in mind there are some strict requirements for this to work, for example, so make sure you’re using  a Kubernetes cluster that meets this requirement. kubeflow/manifest the highest version of Kubernetes supported by Kubeflow 1.5.0 (at the time of writing) is 1.21 One additional requirement is to have Kustomize version , and that's it. 3.2.0 Let's start by cloning the kubeflow/manifest repository git clone https://github.com/kubeflow/manifests Then change directories the manifest folder and run the following command: cd manifests
while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done This command will take a few minutes to install all the resources needed by Kubeflow. If anything fails to apply, the command will continue attempting to apply it until it succeeds entirely. After a few minutes, you can confirm all the pods in the namespace are up and running: kubeflow kubectl -n kubeflow get pods Now we will configure Kubeflow to use our new MinIO. Configure Kubeflow The following section is the core of connecting Kubeflow and MinIO. Please note that the resources that need to be modified in this section are also what you'd tweak if you were starting with an existing Kubeflow deployment. We are going to edit a variety of Config Maps, Secrets and Deployments on the namespace first, and then on any existing user namespaces. kubeflow All of these steps assume MinIO is running in the namespace and running on port If you were running the tenant with TLS you'd use port 443. ns-1 80. Tenant URL: minio.ns-1.svc.cluster.local Tenant Port: 80 Edit Configmaps pipeline-install-config Edit the config map and add the following fields to : pipeline-install-config .data minioServiceHost: minio.ns-1.svc.cluster.local
minioServicePort: "80" Edit command: kubectl -n kubeflow edit cm pipeline-install-config workflow-controller-configmap Edit the configmap and configure the field inside the section to point to your tenant workflow-controller-configmap endpoint s3 s3:
      endpoint: "minio.ns-1.svc.cluster.local:80" Use this command to edit the configmap: kubectl -n kubeflow edit cm workflow-controller-configmap ml-pipeline-ui-configmap Edit the configmap and replace the json content of viewer-pod-template.json with the following json: ml-pipeline-ui-configmap {
  "spec": {
    "containers": [
      {
        "env": [
          {
            "name": "AWS_ACCESS_KEY_ID",
            "valueFrom": {
              "secretKeyRef": {
                "name": "mlpipeline-minio-artifact",
                "key": "accesskey"
              }
            }
          },
          {
            "name": "AWS_SECRET_ACCESS_KEY",
            "valueFrom": {
              "secretKeyRef": {
                "name": "mlpipeline-minio-artifact",
                "key": "secretkey"
              }
            }
          },
          {
            "name": "AWS_REGION",
            "valueFrom": {
              "configMapKeyRef": {
                "name": "pipeline-install-config",
                "key": "minioServiceRegion"
              }
            }
          }
        ]
      }
    ]
  }
} Use this command to edit the configmap: kubectl -n kubeflow edit cm ml-pipeline-ui-configmap Make sure the indentation structure of the json matches the existing format. Edit Secrets We will update the secret that holds the credentials to MinIO, however these are meant to be encoded, so you can encode them with shell: base64 echo -n "kubeflow" | base64 
echo -n "kubeflow123" | base64 mlpipeline-minio-artifact Edit the secret and set these values in the field mlpipeline-minio-artifact .data data:
  accesskey: a3ViZWZsb3c=
  secretkey: a3ViZWZsb3cxMjM= Use this command to edit the configmap: kubectl -n kubeflow edit secret mlpipeline-minio-artifact Edit Deployments We will now edit the deployments last to cause a pod restart and to get everything ready. ml-pipeline-ui Edit the deployment and add the following environment variables: ml-pipeline-ui - name: AWS_ACCESS_KEY_ID
  valueFrom:
    secretKeyRef:
      name: mlpipeline-minio-artifact
      key: accesskey
- name: AWS_SECRET_ACCESS_KEY
  valueFrom:
    secretKeyRef:
      name: mlpipeline-minio-artifact
      key: secretkey
- name: MINIO_NAMESPACE

- name: MINIO_HOST
  value: minio.ns-1.svc.cluster.local
- name: MINIO_PORT
  value: "80" make sure to edit the MINIO_NAMESPACE environment variable to be empty, this is critical as that environment variable is already present in the deployment. Note: Use the following command to edit the configmap: kubectl -n kubeflow edit deployment ml-pipeline-ui ml-pipeline Edit the deployment and add the following environment variables: ml-pipeline - name: OBJECTSTORECONFIG_HOST
  valueFrom:
    configMapKeyRef:
      name: pipeline-install-config
      key: minioServiceHost
- name: OBJECTSTORECONFIG_PORT
  value: "80" Use the following command to edit the deployment: kubectl -n kubeflow edit deployment ml-pipeline Configure Every User Namespace This is also very important, for every user namespace, patch the deployment in that namespace and the artifact secret. For example, in my case my namespace is since we used the example manifest. ml-pipeline-ui-artifact kubeflow-user-example-com Edit the secret and set these values in the field: mlpipeline-minio-artifact .data data:
  accesskey: a3ViZWZsb3c=
  secretkey: a3ViZWZsb3cxMjM= Edit the and add the following environment variables ml-pipeline-ui-artifact - name: MINIO_NAMESPACE
- name: MINIO_HOST
  value: minio.ns-1.svc.cluster.local
- name: MINIO_PORT
  value: "80" Use the following command to edit the artifact: kubectl -n kubeflow-user-example-com edit secret mlpipeline-minio-artifact

kubectl -n kubeflow-user-example-com edit deployment ml-pipeline-ui-artifact At this point Kubeflow is properly configured to use your tenant. There’s one last step and then we are good to test our deployment. Migrate All Data from Kubeflow's Internal MinIO to the New Tenant Now that we have configured everything, we just need to make sure the data Kubeflow is expecting to be in its buckets is actually there. Let’s copy that data over and then shutdown the internal MinIO that we’re replacing. To achieve this we will use MinIO Client (mc), a CLI tool for managing MinIO. We'll do all these operations from a pod running inside Kubernetes, but you can do this via port-forwarding and using mc from your own machine if you choose to do so. Let's run a pod with an Ubuntu shell: kubectl -n kubeflow run my-shell  -i --tty --image ubuntu -- bash This shell runs on a pod running inside our Kubernetes cluster in the Kubeflow namespace. Now we will: Install wget Download mc Make executable mc Add an alias to the current MinIO Add an alias to the new MinIO Copy all the data To accomplish this we run the following commands: apt update && apt install -y wget
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
mv mc /usr/local/bin/
mc config host add kubeflow http://minio-service.kubeflow.svc.cluster.local:9000 minio minio123
mc config host add tenant http://minio.ns-1.svc.cluster.local kubeflow kubeflow123
mc mirror kubeflow tenant Finally, turn off the internal MinIO as it is no longer required. kubectl -n kubeflow scale deploy minio --replicas=0 All right! We are done moving to the full MinIO deployment. Validate that Kubeflow is Using the new MinIO Next we’ll validate  the setup and run some pipelines. If you go to MinIO Operator, you can see the tenant now has data: Click the tenant, and then click in the top right of the browser window to open MinIO Console in order to browse that tenant. Console From this view, you can see the bucket. Click browse to see its contents. mlpipeline You'll see the existing demo pipelines have been copied over. Now let's go into Kubeflow and run some pipelines, you can use port forwarding  to expose the Kubeflow central dashboard: kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80 Then in your browser go to . http://localhost:8080 Login with the default credentials for this example setup: Email Address: user@example.com 12341234 Password: Then go to the Pipelines menu in the left menu bar. We’re going to run the most basic pipeline, "[Tutorial] DSL - Control structures": Click on the pipeline’s name. From here, click Create Experiment on the top right. This will create a new experiment since it's the first time it is running, but in subsequently you can re-use this experiment. And click on Start: After the run is complete, explore the pipeline to verify that it ran successfully. Kubeflow and MinIO for Multi-Cloud Machine Learning This blog post taught you how to replace the MinIO that ships with Kubeflow with the MinIO Operator. You’re now prepared to take your Kubeflow use to the next level and back it with Kubernetes-native high performance and highly scalable MinIO object storage. When it comes to Machine Learning pipelines and infrastructure, use MinIO's to deploy tenants backed by super fast NVMe drives as your hot tier for fast training and model serving, and  also set up a warm tier backed up by SSDs or HDDs for your aging datasets. MinIO does this transparently  without disrupting your applications. Tiering  is configured on a per-bucket basis or even for a single prefix within a bucket, providing  granular control over  which data gets moved to a slower tier. Lifecycle Management With MinIO's , you can configure  buckets serving production machine learning models to be replicated instantly across multiple sites for disaster recovery and fast failover. Active-Active Replication I truly hope this blog post helped you discover how easy it is to set up MinIO object storage on Kubernetes and to consume it with Kubeflow. If you have any questions, please join our and ask! Slack community Also published . here

This writer has a vested interest be it monetary, business, or otherwise, with 1 or more of the products or companies mentioned within.

Installing and Configuring Kubeflow with MinIO Operator

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A Closer Look Into the MinIO Enterprise Object Store Firewall

10 Best Datasets for Time Series Analysis

12 Best Pre-Installed R Datasets Commonly Used for Statistical Analysis

Import JSON To Google Sheets - 3 Best Ways To Do It

How To Import External Data Into Google Sheets Without Copy/Paste

A Closer Look Into the MinIO Enterprise Object Store Firewall

10 Best Datasets for Time Series Analysis

12 Best Pre-Installed R Datasets Commonly Used for Statistical Analysis

Import JSON To Google Sheets - 3 Best Ways To Do It

How To Import External Data Into Google Sheets Without Copy/Paste

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps