Posted in Software Engineering

Kubernetes HPA, Custom metrics, and prometheus

The early version of Horizontal Pod Autoscaler (HPA) was limited in features and it only supported scaling deployments based on CPU metrics. The most recent Kubernetes releases included support for Memory, multiple metrics and in the latest version Custom Metrics.

In this article I’ll cover how to store custom metrics using prometheus and use that to scale using Horizontal Pod Autoscaler.

Prerequisite

HELM

Install Helm version 3 (https://helm.sh/). Following are steps to install helm

  • From Homebrew (macOS)

Members of the Kubernetes community have contributed a Helm formula build to Homebrew. This formula is generally up to date.

brew install helm

(Note: There is also a formula for emacs-helm, which is a different project.)

  • From Chocolatey (Windows)

Members of the Kubernetes community have contributed a Helm package build to Chocolatey. This package is generally up to date.

choco install kubernetes-helm
  • From Snap (Linux)

The Snapcrafters community maintains the Snap version of the Helm package:

sudo snap install helm --classic
  • From Script

Helm now has an installer script that will automatically grab the latest version of Helm and install it locally.

You can fetch that script, and then execute it locally. It’s well documented so that you can read through it and understand what it is doing before you run it.

$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh

Yes, you can curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash that if you want to live on the edge.

Kubernetes Metrics Server

Run the following command, this will install metrics-server to kube-system namespace

helm install metrics-server stable/metrics-server -n kube-system

Prometheus

Run the following command, this will install prometheus to kube-system namespace

helm install prometheus stable/prometheus  -n kube-system

Prometheus Adapter

Prometheus Adapter Custom purposes is to communicate using Metrics API. Custom metrics are used in Kubernetes by Horizontal Pod Autoscalers to scale workloads based upon your own metric pulled from an external metrics provider like Prometheus. This chart complements the metrics-server chart that provides resource only metrics.

helm install prometheus-adapter --set rbac.create=true,prometheus.url=http://prometheus-server, prometheus.port=80 stable/prometheus-adapter -n kube-system

After installing prometheus adapter. Wait 2-3 minutes, then use the following command to check if the prometheus adapter is working.

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq

if the installation is working, it displays a bunch of information.

Steps

Http Request Custom Metrics

After ensuring that the installation is working, now we will set up a custom metrics rule to collect HTTP requests.

kubectl edit cm prometheus-adapter -n kube-system

Add HTTP request as the following

apiVersion: v1
data:
  config.yaml: |
    rules:
    - seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
      resources:
        overrides:
          kubernetes_namespace: {resource: "namespace"}
          kubernetes_pod_name: {resource: "pod"}
      name:
        matches: "^(.*)_total"
        as: "${1}_per_second"
      metricsQuery: 'sum(rate({}[2m])) by ()'

Pod Deployment

Now deploy a prometheus instrumented pod (https://prometheus.io/docs/practices/instrumentation/), following is an example

Create a file podinfo-dep.yml as the following:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: podinfo
  namespace: temp
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: podinfo
      annotations:
        prometheus.io/scrape: 'true'
    spec:
      containers:
      - name: podinfod
        image: stefanprodan/podinfo:0.0.1
        imagePullPolicy: Always
        command:
          - ./podinfo
          - -port=9898
          - -logtostderr=true
          - -v=2
        ports:
        - containerPort: 9898
          protocol: TCP
        resources:
          requests:
            memory: "32Mi"
            cpu: "1m"
          limits:
            memory: "256Mi"
            cpu: "100m"
---
apiVersion: v1
kind: Service
metadata:
  name: podinfo
  namespace: temp
  labels:
    app: podinfo
spec:
  type: NodePort
  ports:
    - port: 9898
      targetPort: 9898
      nodePort: 31198
      protocol: TCP
  selector:
    app: podinfo

run kubernetes apply to deploy to the kubernetes cluster, as the following:

kubectl apply -f podinfo-dep.yml

Wait 2-3 minutes, to collect the metrics. Then run the following command:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/temp/pods/*/http_requests_per_second" | jq .

Expected output, will similarly look like this:

 {
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/temp/pods/%2A/http_requests_per_second"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "temp",
        "name": "podinfo-58b68656c9-4nx52",
        "apiVersion": "/v1"
      },
      "metricName": "http_requests_per_second",
      "timestamp": "2020-04-21T03:35:53Z",
      "value": "850m",
      "selector": null
    }
  ]
}

HPA Configuration

Now we have everything ready. We can configure an HPA autoscaler for our deployment using the custom metrics.

Create hpa.yml as the following:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: podinfo
  namespace: temp
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: podinfo
  minReplicas: 1
  maxReplicas: 6
  metrics:
  - type: Pods
    pods:
      metricName: http_requests_per_second 
      targetAverageValue: 2000m

Deploy the files to kubernetes using the following command:

kubectl apply -f hpa.yml

Well done, You are doing great ! all the configuration is completed, now the testing time.

Testing

Assuming you deploy using temp namespace you can run the following command to generate traffic

 kubectl run apache-bench -i --tty --rm --image=httpd -- ab -n 500000 -c 1000 http://podinfo.temp.svc.cluster.local:9898//

open another console and run the following command:

kubectl get hpa -n temp -w                                                                                                                                       NAME      REFERENCE            TARGETS   MINPODS   MAXPODS   REPLICAS   AGE

output will be something like this:

podinfo   Deployment/podinfo   850m/2    3         6         3          

After a while the output will change to the following:

podinfo   Deployment/podinfo    65755m/2    3         6         6          

If you are successfully run without any problem, It is an indication that HPA is working to scale out the pods using custom metrics.
Congratulations !!!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s