Keda Autoscaling: A K8s-Based Event Driven Autoscaler

This blog discusses scaling Kubernetes deployments based on Apache Kafka topic lag with KEDA.

What is Apache Kafka?

Kafka is a distributed message streaming platform that uses a publish and subscribe mechanism to stream the records or messages.

What is Keda ?

KEDA is a Kubernetes-based Event Driven Autoscaler. KEDA can be installed into any Kubernetes cluster and can work alongside standard Kubernetes components like the Horizontal Pod Autoscaler (HPA). When we install KEDA in any Kubernetes cluster, two containers run. The first one is keda-operator, and the second one is keda-operator-metrics-apiserver. Here, the Role of keda-operator is to scale Deployments, Statefulsets or Rollouts from zero or to zero on no events. KEDA acts as a Kubernetes metrics server that exposes rich event data like queue length or stream lag to the HPA to drive scale-out. The metric serving is the primary role of the second container i.e. keda-operator-metrics-apiserver.

Installing Keda (Using Helm)

1. Add Helm repo

helm repo add kedacore https://kedacore.github.io/charts

2. Update Helm repo

helm repo update

3. Install keda Helm chart

Using Helm2 -

helm install kedacore/keda --namespace keda --version 1.4.2 --name keda

Using Helm3 -

kubectl create namespace keda
helm install keda kedacore/keda --version 1.4.2 --namespace keda

Example:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-scaledobject
  namespace: demo3
spec:
  scaleTargetRef:
    apiVersion: argoproj.io/v1alpha1
    name: keda-test-demo3
    kind: Rollout
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka-demo3.demo3:9092
        topic: test
        consumerGroup: console-consumer-76109
        lagThreshold: "300"
        offsetResetPolicy: latest
  maxReplicaCount: 10
  minReplicaCount: 1

In .spec.triggers section, we provide the information that KEDA uses to trigger the autoscaling. Here are some of the parameters for your reference which can be used for triggering autoscaling.