In this blog, we will discuss about scaling kubernetes deployments based on Apache kafka topic lag with KEDA.

What is Apache Kafka ?

Kafka is a distributed message streaming platform that uses a publish and subscribe mechanism to stream the records or messages.


What is Keda ?

KEDA is a Kubernetes-based Event Driven Autoscaler. KEDA can be installed into any Kubernetes cluster and can work alongside standard Kubernetes components like the Horizontal Pod Autoscaler (HPA). When we install KEDA in any kubernetes cluster, two containers run. First one is keda-operator and second one is keda-operator-metrics-apiserver. Here, Role of keda-operator is to scale Deployments, Statefulsets or Rollouts from zero or to zero on no events. KEDA acts as a Kubernetes metrics server that exposes rich event data like queue length or stream lag to the HPA to drive scale out.The metric serving is the primary role of the second container i.e. keda-operator-metrics-apiserver.

Installing Keda (Using Helm)

  1. Add Helm repo
helm repo add kedacore https://kedacore.github.io/charts

2. Update Helm repo

helm repo update

3. Install keda Helm chart

         Helm2

helm install kedacore/keda --namespace keda --version 1.4.2 --name keda

        Helm3

kubectl create namespace keda
helm install keda kedacore/keda --version 1.4.2 --namespace keda

Example:

   apiVersion: keda.sh/v1alpha1
     kind: ScaledObject
     metadata:
         name: kafka-scaledobject
         namespace: demo3
     spec:
         scaleTargetRef:
             apiVersion: argoproj.io/v1alpha1
             name: keda-test-demo3
             kind: Rollout
         triggers:
             - type: kafka
               metadata:
                   bootstrapServers: kafka-demo3.demo3:9092
                   topic: test
                   consumerGroup: console-consumer-76109
                   lagThreshold: "300"
                   offsetResetPolicy: latest
         maxReplicaCount: 10
         minReplicaCount: 1

          In .spec.triggers section, we provide the informations that KEDA uses to trigger the autoscaling.

Parameters Descriptions
.spec.triggers.type Type of the metric used for scaling.
.spec.triggers.metadata Additional information about the metric used for scaling.
.spec.triggers.metadata.bootstrapServers :9092 .
.spec.triggers.metadata.topic Name of topic from which messages are to be consumed.
.spec.triggers.metadata.consumerGroup Name of Consumer-group.
.spec.triggers.metadata.lagThreshold Target value of lag to trigger autoscaling.