Kubernetes Autoscaling (HPA) with Custom Metrics

To scale components based on custom metrics we need to have two components:

One that collects metrics from our applications and stores them to Kloudfuse database.
The second one that extends the Kubernetes Custom Metrics API with the metrics supplied by a collector, the k8s-prometheus-adapter. This is an implementation of the custom metrics API that attempts to support arbitrary metrics.

Consider we have an application specific metric named some_metric ingested into Kfuse which we want to modify and Autoscale our pods on.

First we would need to install Prometheus adapter with proper configuration to emit a custom metric some_custom_metric based on a query for some_metric

Create prometheus-values.yaml file for Prometheus Adapter configuring the proper endpoint, authentication and custom metric rules :-

prometheus:
  # Replace with your endpoint for Kloudfuse
  url: <<https://your.hostname.kloudfuse>>
  port: 443
  path: ""
  
# Rules to generate `some_custom_metric` as an external metric
rules:
  default: false
  external:
  - seriesQuery: '{__name__=~"some_metric"}'
    resources:
      template: <<.Resource>>
    name:
      as: "some_custom_metric"
    metricsQuery: avg (some_metric)
    
# Base64 Encoded Auth string in the format <<username:password>>
extraArguments:
  - --prometheus-header=Authorization=Basic YWRtaW46cGFzc3dvcmQ=

Deploy Prometheus adapter on your cluster

helm install --name RELEASE_NAME prometheus-community/prometheus-adapter -f prometheus-values.yaml

Verify the adapter properly emits your custom metric within the K8s external metrics api

kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 | jq .

#Should show some output like this 
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "some_custom_metric",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

For additional Prometheus-adapter configurations please refer to their Helm documentation here

Once the custom metric is available to use within k8s metrics api, you can configure HPA to use the metric like this:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: <<some_name>>
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: <<some_name>>
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: some_custom_metric
      target:
        type: AverageValue
        averageValue: 100

For additional information on configuring HPA with Custom metrics refer to k8s documentation here