The container-native workflow engine for Kubernetes is a tremendously popular open source option for teams looking to run workflows, manage clusters, orchestrate parallel jobs, and more. Argo Pairing Kubernetes and Argo, multiple dev teams can run workloads on centrally-managed clusters, sharing compute resources to increase efficiency and reduce overhead. However, keeping track of each team’s spending—and spending efficiency—in the Kubernetes shared resources environment has become an all-too-familiar challenge. Teams routinely struggle with budgeting projections and waste reduction efforts without visibility into historical resource usage data and the ability to break down cost responsibilities by team. That visibility is crucial to addressing minor inefficiencies before they grow and lead to significant waste as workflows scale. [A completed Argo workflow, its costs completely unknown] Kubecost, a Kubernetes cost monitoring tool with a , enables dev teams to accurately measure Argo workflow costs in real-time, allocate costs to teams, recognize resource efficiencies, and suggest opportunities for optimization. Enter Kubecost, Argo’s spend-conscious companion free community version The following guide demonstrates how teams managing Kubernetes workloads with Argo Workflows can deploy Kubecost to analyze and control their workflow spending. Those interested in beginning to leverage Argo Workflows should follow this on GitHub.   Kubecost itself can be installed using , and should be given its own namespace: simple Quick Start guide Helm Once installed, Kubecost will collect cost data on running Argo Workflows. For this guide, let’s define an Argo Workflow by creating a CRD. We’ll make the workflow mimic high resource usage by having the Docker container make resource-consuming Pods. The workflow will perform its single “stress” stage, running the stress-ng image, along with inputs Argo will handle in the launched workflow.
 
To create the CRD in the Argo namespace, save the following yaml to a file and run . Create an example workflow WorkflowTemplate stress-ng WorkflowTemplate argo template create $filename -n argo The workflow requests 1200m CPU and 40M memory, while the running stress-ng container uses 1 CPU and 30M memory. That gap represents wasteful inefficiency, as the over-provisioned container uses more cluster compute resources than it requires to operate. Because Argo Workflow jobs all feature the metadata label , viewing each workflow’s costs and efficiency metrics within Kubecost is simple. In the Kubecost UI, go to the Cost Allocation page and change the “Aggregate By” field to “Label: workflows_argoproj_io_workflow.” This will display the following per-workflow cost and efficiency data: Measure Kubernetes costs with Kubecost workflows.argoproj.io/workflow Each time a workflow runs, that run receives a unique name based on the template and an added identifier. In the below example, the stress-ng workflow’s first run is named stress-ng-q9wdz. Filtering to this specific workflow run, Kubecost displays that its cost efficiency was just 57%. Clicking the workflow name to open the Details view shows the degree to which CPU and RAM overprovisioning caused this inefficiency: only 55% of requested CPU and 60% of requested RAM actually went to use! Clusters with Kubecost installed are available for inspection using the kubectl-cost CLI tool as well as the UI. Use Krew to install kubectl-cost: In the CLI, workflow costs are available with the command Use the kubectl-cost CLI for quick cost inspection kubectl krew install cost kubectl cost label --label workflows.argoproj.io/workflow : Boost the precision of cost insights The Kubecost tool uses the major cloud providers’ publicly-listed prices by default, but integrating will yield even more precise cost analysis. Doing so can also account for any custom discounts, reserved instances, or other pricing differences that might be relevant to your environment. actual cloud provider billing data

Read My Stories

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

How to Optimize Kubernetes and Argo Workflow Spending

Too Long; Didn't Read

Company Mentioned

Kate Weber

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

How to Optimize Kubernetes and Argo Workflow Spending

Too Long; Didn't Read

Company Mentioned

Kate Weber

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES