Introduction One of the major advantages of using Kubernetes for container orchestration is that it makes it really easy to scale our application horizontally and account for increased load. Natively, horizontal pod autoscaling can scale the deployment based on CPU and Memory usage but in more complex scenarios we would want to account for other metrics before making scaling decisions. Welcome . Prometheus is the standard tool for monitoring deployed workloads and the Kubernetes cluster itself. Prometheus adapter helps us to leverage the metrics collected by Prometheus and use them to make scaling decisions. These metrics are exposed by an API service and can be readily used by our object. Prometheus Adapter Horizontal Pod Autoscaling Deployment Architecture Overview We will be using Prometheus adapter to pull custom metrics from our Prometheus installation and then let the use it to scale the pods up or down. Horizontal Pod Autoscaler (HPA) Prerequisite Basic knowledge about horizontal pod autoscalingPrometheus deployed in-cluster or accessible using an endpoint. We will be using a Highly Available deployment. Prometheus-Thanos Deploying the Sample Application Let’s first deploy a sample app over which we will be testing our Prometheus metrics autoscaling. We can use the manifest below to do it apiVersion: v1
kind: Namespace
metadata:
  name: nginx
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  namespace: nginx
  name: nginx-deployment
spec:
  replicas: 1
  template:
    metadata:
      annotations:
        prometheus.io/path: prometheus.io/scrape: prometheus.io/port: labels:
        app: nginx-server
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - nginx-server
              topologyKey: kubernetes.io/hostname
      containers:
      - name: nginx-demo
        image: vaibhavthakur/nginx-vts:1.0
        imagePullPolicy: Always
        resources:
          limits:
            cpu: 2500m
          requests:
            cpu: 2000m
        ports:
        - containerPort: 80
          name: http
---
apiVersion: v1
kind: Service
metadata:
  namespace: nginx
  name: nginx-service
spec:
  ports:
  - port: 80
    targetPort: 80
    name: http
  selector:
    app: nginx-server : LoadBalancer "/status/format/prometheus" "true" "80" type This will create a namespace named and deploy a sample Nginx application in it. The application can be accessed using the service and also exposes at the endpoint over port . For the sake of our setup we have created a DNS entry for the ExternalIP which maps to nginx nginx vts metrics /status/format/prometheus 80 nginx.gotham.com root$ kubectl get deploy 
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   1/1     1            1           43d

root$ kubectl get pods 
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-65d8df7488-c578v   1/1     Running   0          9h

root$ kubectl get svc
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
nginx-service   ClusterIP   10.63.253.154   35.232.67.34      80/TCP    43d

root$ kubectl describe deploy nginx-deployment
Name:                   nginx-deployment
Namespace:              nginx
CreationTimestamp:      Tue, 08 Oct 2019 11:47:36 -0700
Labels:                 app=nginx-server
Annotations:            deployment.kubernetes.io/revision: 1
                        kubectl.kubernetes.io/last-applied-configuration:
                          { : , : , :{ :{}, : , : }, :...
Selector:               app=nginx-server
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:       app=nginx-server
  Annotations:  prometheus.io/path: /status/format/prometheus
                prometheus.io/port: 80
                prometheus.io/scrape: Containers:
   nginx-demo:
    Image:      vaibhavthakur/nginx-vts:v1.0
    Port:       80/TCP
    Host Port:  0/TCP
    Limits:
      cpu:  250m
    Requests:
      cpu:        200m
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   nginx-deployment-65d8df7488 (1/1 replicas created)
Events:          <none>


root$ curl nginx.gotham.com
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href= >nginx.org</a>.<br/>
Commercial support is available at
<a href= >nginx.com</a>.</p>

<p><em>Thank you using nginx.</em></p>
</body>
</html> "apiVersion" "extensions/v1beta1" "kind" "Deployment" "metadata" "annotations" "name" "nginx-deployment" "namespace" "nginx" "spec" true "http://nginx.org/" "http://nginx.com/" for These are all the metrics currently exposed by the application $ curl nginx.gotham.com/status/format/prometheus nginx_vts_info{hostname= ,version= } 1 nginx_vts_start_time_seconds 1574283147.043 nginx_vts_main_connections{status= } 215
nginx_vts_main_connections{status= } 4
nginx_vts_main_connections{status= } 215
nginx_vts_main_connections{status= } 0
nginx_vts_main_connections{status= } 15577
nginx_vts_main_connections{status= } 3
nginx_vts_main_connections{status= } 1 nginx_vts_main_shm_usage_bytes{shared= } 1048575
nginx_vts_main_shm_usage_bytes{shared= } 3510
nginx_vts_main_shm_usage_bytes{shared= } 1 nginx_vts_server_bytes_total{host= ,direction= } 3303449
nginx_vts_server_bytes_total{host= ,direction= } 61641572
nginx_vts_server_requests_total{host= ,code= } 0
nginx_vts_server_requests_total{host= ,code= } 15574
nginx_vts_server_requests_total{host= ,code= } 0
nginx_vts_server_requests_total{host= ,code= } 2
nginx_vts_server_requests_total{host= ,code= } 0
nginx_vts_server_requests_total{host= ,code= } 15576
nginx_vts_server_request_seconds_total{host= } 0.000
nginx_vts_server_request_seconds{host= } 0.000
nginx_vts_server_cache_total{host= ,status= } 0
nginx_vts_server_cache_total{host= ,status= } 0
nginx_vts_server_cache_total{host= ,status= } 0
nginx_vts_server_cache_total{host= ,status= } 0
nginx_vts_server_cache_total{host= ,status= } 0
nginx_vts_server_cache_total{host= ,status= } 0
nginx_vts_server_cache_total{host= ,status= } 0
nginx_vts_server_cache_total{host= ,status= } 0
nginx_vts_server_bytes_total{host= ,direction= } 3303449
nginx_vts_server_bytes_total{host= ,direction= } 61641572
nginx_vts_server_requests_total{host= ,code= } 0
nginx_vts_server_requests_total{host= ,code= } 15574
nginx_vts_server_requests_total{host= ,code= } 0
nginx_vts_server_requests_total{host= ,code= } 2
nginx_vts_server_requests_total{host= ,code= } 0
nginx_vts_server_requests_total{host= ,code= } 15576
nginx_vts_server_request_seconds_total{host= } 0.000
nginx_vts_server_request_seconds{host= } 0.000
nginx_vts_server_cache_total{host= ,status= } 0
nginx_vts_server_cache_total{host= ,status= } 0
nginx_vts_server_cache_total{host= ,status= } 0
nginx_vts_server_cache_total{host= ,status= } 0
nginx_vts_server_cache_total{host= ,status= } 0
nginx_vts_server_cache_total{host= ,status= } 0
nginx_vts_server_cache_total{host= ,status= } 0
nginx_vts_server_cache_total{host= ,status= } 0 # HELP nginx_vts_info Nginx info # TYPE nginx_vts_info gauge "nginx-deployment-65d8df7488-c578v" "1.13.12" # HELP nginx_vts_start_time_seconds Nginx start time # TYPE nginx_vts_start_time_seconds gauge # HELP nginx_vts_main_connections Nginx connections # TYPE nginx_vts_main_connections gauge "accepted" "active" "handled" "reading" "requests" "waiting" "writing" # HELP nginx_vts_main_shm_usage_bytes Shared memory [ngx_http_vhost_traffic_status] info # TYPE nginx_vts_main_shm_usage_bytes gauge "max_size" "used_size" "used_node" # HELP nginx_vts_server_bytes_total The request/response bytes # TYPE nginx_vts_server_bytes_total counter # HELP nginx_vts_server_requests_total The requests counter # TYPE nginx_vts_server_requests_total counter # HELP nginx_vts_server_request_seconds_total The request processing time in seconds # TYPE nginx_vts_server_request_seconds_total counter # HELP nginx_vts_server_request_seconds The average of request processing times in seconds # TYPE nginx_vts_server_request_seconds gauge # HELP nginx_vts_server_request_duration_seconds The histogram of request processing time # TYPE nginx_vts_server_request_duration_seconds histogram # HELP nginx_vts_server_cache_total The requests cache counter # TYPE nginx_vts_server_cache_total counter "_" "in" "_" "out" "_" "1xx" "_" "2xx" "_" "3xx" "_" "4xx" "_" "5xx" "_" "total" "_" "_" "_" "miss" "_" "bypass" "_" "expired" "_" "stale" "_" "updating" "_" "revalidated" "_" "hit" "_" "scarce" "*" "in" "*" "out" "*" "1xx" "*" "2xx" "*" "3xx" "*" "4xx" "*" "5xx" "*" "total" "*" "*" "*" "miss" "*" "bypass" "*" "expired" "*" "stale" "*" "updating" "*" "revalidated" "*" "hit" "*" "scarce" Among these we are particularly interested in . We will be using the value of this metric to determine whether or not to scale our Nginx deployment. nginx_vts_server_requests_total Create Prometheus Adapter ConfigMap Use the manifest below to create the Prometheus Adapter Configmap apiVersion: v1
kind: ConfigMap
metadata:
  name: adapter-config
  namespace: monitoring
data:
  config.yaml: |
    rules:
    - seriesQuery: resources:
        overrides:
          kubernetes_namespace:
            resource: namespace
          kubernetes_pod_name:
            resource: pod
      name:
        matches: as: metricsQuery: (sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)) 'nginx_vts_server_requests_total' "^(.*)_total" " _per_second" ${1} This config map only specifies a single metric. However, we can always add more metrics. Create Prometheus Adapter Deployment Use the following manifest to deploy Prometheus Adapter apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: custom-metrics-apiserver
  name: custom-metrics-apiserver
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: custom-metrics-apiserver
  template:
    metadata:
      labels:
        app: custom-metrics-apiserver
      name: custom-metrics-apiserver
    spec:
      serviceAccountName: monitoring
      containers:
      - name: custom-metrics-apiserver
        image: quay.io/coreos/k8s-prometheus-adapter-amd64:v0.4.1
        args:
        - /adapter
        - --secure-port=6443
        - --tls-cert-file=/var/run/serving-cert/serving.crt
        - --tls-private-key-file=/var/run/serving-cert/serving.key
        - --logtostderr= - --prometheus-url=http://thanos-querier.monitoring:9090/
        - --metrics-relist-interval=30s
        - --v=10
        - --config=/etc/adapter/config.yaml
        ports:
        - containerPort: 6443
        volumeMounts:
        - mountPath: /var/run/serving-cert
          name: volume-serving-cert
          readOnly: - mountPath: /etc/adapter/
          name: config
          readOnly: volumes:
      - name: volume-serving-cert
        secret:
          secretName: cm-adapter-serving-certs
      - name: config
        configMap:
          name: adapter-config true true true This will create our deployment which will spawn the Prometheus adapter pod to pull metrics from Prometheus. It should be noted that we have set the argument . This is because we have deployed a Prometheus-Thanos cluster in the monitoring namespace in the same Kubernetes cluster as Prometheus adapter. You can change this argument to point to your Prometheus deployment. --prometheus-url= http://thanos-querier.monitoring:9090/ If you notice the logs of this container you can see that it is fetching the metric defined in the config file I1122 00:26:53.228394       1 api.go:74] GET http://thanos-querier.monitoring:9090/api/v1/series?match%5B%5D=nginx_vts_server_requests_total&start=1574381213.217 200 OK
I1122 00:26:53.234234       1 api.go:93] Response Body: { : , :[{ : , : , : , : , : , : , : , : , : , : },{ : , : , : , : , : , : , : , : , : , : } "status" "success" "data" "__name__" "nginx_vts_server_requests_total" "app" "nginx-server" "cluster" "prometheus-ha" "code" "1xx" "host" "*" "instance" "10.60.64.39:80" "job" "kubernetes-pods" "kubernetes_namespace" "nginx" "kubernetes_pod_name" "nginx-deployment-65d8df7488-sbp95" "pod_template_hash" "65d8df7488" "__name__" "nginx_vts_server_requests_total" "app" "nginx-server" "cluster" "prometheus-ha" "code" "1xx" "host" "*" "instance" "10.60.64.8:80" "job" "kubernetes-pods" "kubernetes_namespace" "nginx" "kubernetes_pod_name" "nginx-deployment-65d8df7488-mwzxg" "pod_template_hash" "65d8df7488" Creating Prometheus Adapter API Service The manifest below will create an API service so that our Prometheus adapter is accessible by Kubernetes API and thus metrics can be fetched by our Horizontal Pod Autoscaler. apiVersion: v1
kind: Service
metadata:
  name: custom-metrics-apiserver
  namespace: monitoring
spec:
  ports:
  - port: 443
    targetPort: 6443
  selector:
    app: custom-metrics-apiserver
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.custom.metrics.k8s.io
spec:
  service:
    name: custom-metrics-apiserver
    namespace: monitoring
  group: custom.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: groupPriorityMinimum: 100
  versionPriority: 100 true Testing the Set-Up Let’s check what all custom metrics are available root$ kubectl get --raw | jq .

{ : , : , : , : [
    { : , : , : , : , : [ ]
    },
    { : , : , : , : , : [ ]
    }
  ]
} "/apis/custom.metrics.k8s.io/v1beta1" "kind" "APIResourceList" "apiVersion" "v1" "groupVersion" "custom.metrics.k8s.io/v1beta1" "resources" "name" "pods/nginx_vts_server_requests_per_second" "singularName" "" "namespaced" true "kind" "MetricValueList" "verbs" "get" "name" "namespaces/nginx_vts_server_requests_per_second" "singularName" "" "namespaced" false "kind" "MetricValueList" "verbs" "get" We can see that metric is available. Now, let’s check the current value of this metric nginx_vts_server_requests_per_second root$ kubectl get --raw | jq .

{ : , : , : { : }, : [
    { : { : , : , : , : }, : , : , : }
  ]
} "/apis/custom.metrics.k8s.io/v1beta1/namespaces/nginx/pods/*/nginx_vts_server_requests_per_second" "kind" "MetricValueList" "apiVersion" "custom.metrics.k8s.io/v1beta1" "metadata" "selfLink" "/apis/custom.metrics.k8s.io/v1beta1/namespaces/nginx/pods/%2A/nginx_vts_server_requests_per_second" "items" "describedObject" "kind" "Pod" "namespace" "nginx" "name" "nginx-deployment-65d8df7488-v575j" "apiVersion" "/v1" "metricName" "nginx_vts_server_requests_per_second" "timestamp" "2019-11-19T18:38:21Z" "value" "1236m" Create an HPA which will utilize these metrics. We can use the manifest below to do it. apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-custom-hpa
  namespace: nginx
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: nginx-deployment
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - : Pods
    pods:
      metricName: nginx_vts_server_requests_per_second
      targetAverageValue: 4000m type Once you have applied this manifest, you can check the current status of HPA as follows: root$ kubectl describe hpa
Name:               nginx-custom-hpa
Namespace:          nginx
Labels:             <none>
Annotations:        autoscaling.alpha.kubernetes.io/metrics:
                      [{ : , :{ : , : }}]
                    kubectl.kubernetes.io/last-applied-configuration:
                      { : , : , :{ :{}, : , : "type" "Pods" "pods" "metricName" "nginx_vts_server_requests_per_second" "targetAverageValue" "4" "apiVersion" "autoscaling/v2beta1" "kind" "HorizontalPodAutoscaler" "metadata" "annotations" "name" "nginx-custom-hpa" "namespace" "n...
CreationTimestamp:  Thu, 21 Nov 2019 11:11:05 -0800
Reference:          Deployment/nginx-deployment
Min replicas:       2
Max replicas:       10
Deployment pods:    0 current / 0 desired
Events:             <none> Now, let's generate some load on our service. We will be using a utility called for this. Vegeta In a separate terminal run the following command | vegeta attack -rate=5 -duration=0 | vegeta report echo "GET http://nginx.gotham.com/" Meanwhile monitor the nginx pods and horizontal pod autoscaler and you should see something like this root$ kubectl get -w pods
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-65d8df7488-mwzxg   1/1     Running   0          9h
nginx-deployment-65d8df7488-sbp95   1/1     Running   0          4m9s
NAME                                AGE
nginx-deployment-65d8df7488-pwjzm   0s
nginx-deployment-65d8df7488-pwjzm   0s
nginx-deployment-65d8df7488-pwjzm   0s
nginx-deployment-65d8df7488-pwjzm   2s
nginx-deployment-65d8df7488-pwjzm   4s
nginx-deployment-65d8df7488-jvbvp   0s
nginx-deployment-65d8df7488-jvbvp   0s
nginx-deployment-65d8df7488-jvbvp   1s
nginx-deployment-65d8df7488-jvbvp   4s
nginx-deployment-65d8df7488-jvbvp   7s
nginx-deployment-65d8df7488-skjkm   0s
nginx-deployment-65d8df7488-skjkm   0s
nginx-deployment-65d8df7488-jh5vw   0s
nginx-deployment-65d8df7488-skjkm   0s
nginx-deployment-65d8df7488-jh5vw   0s
nginx-deployment-65d8df7488-jh5vw   1s
nginx-deployment-65d8df7488-skjkm   2s
nginx-deployment-65d8df7488-jh5vw   2s
nginx-deployment-65d8df7488-skjkm   3s
nginx-deployment-65d8df7488-jh5vw   4s

root$ kubectl get hpa
NAME               REFERENCE                     TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
nginx-custom-hpa   Deployment/nginx-deployment   5223m/4   2         10        3          5m5s It can be clearly seen that the HPA scaled up our pods as per the requirement, and when we interrupted the Vegeta command, we got the vegeta report. It clearly shows that all our requests were served by the application. root$ | vegeta attack -rate=5 -duration=0 | vegeta report
^CRequests      [total, rate, throughput]  224, 5.02, 5.02
Duration      [total, attack, ]      44.663806863s, 44.601823883s, 61.98298ms
Latencies     [mean, 50, 95, 99, max]    63.3879ms, 60.867241ms, 79.414139ms, 111.981619ms, 229.310088ms
Bytes In      [total, mean]              137088, 612.00
Bytes Out     [total, mean]              0, 0.00
Success       [ratio]                    100.00%
Status Codes  [code:count]               200:224  
Error Set: echo "GET http://nginx.gotham.com/" wait Conclusion This set-up demonstrates how we can use the Prometheus adapter to autoscale deployments based on some custom metrics. For the sake of simplicity we have only fetched one metric from our Prometheus server. However, the adapter Configmap can be extended to fetch some or all the available metrics and use them for autoscaling. If the Prometheus installation is outside of our Kubernetes cluster, we just need to make sure that the query end-point is accessible from the cluster and update it in the adapter deployment manifest. With more complex scenarios, multiple metrics can be fetched and used in-combination to make scaling decisions. Feel free to reach out should you have any questions around the set-up and we would be happy to assist you. . Previously published on https://appfleet.com/blog/prometheus-metrics-based-autoscaling-in-kubernetes/

How To Use Prometheus Adapter to Autoscale Custom Metrics Deployments

Too Long; Didn't Read

Company Mentioned

@sudip-sengupta

RELATED STORIES