Kafka on Kubernetes, minimal configuration

0-kafka-on-k8s.md

This gist describes how to create a Kafka cluster on Kubernetes with minimal effort.

Dislike to incubator/kafka helm chart, this approach uses wurstmeister/kafka Docker image or its GraalVM equivalent, dongjinleekr/kafka.

Note: This configuration is intended for dev or testing purpose; it may be used in production environment, but I can't give any guarantees in that respect.

Prerequisites

Install Zookeeper with the following helm command:

# Add helm incubator repository
helm repo add incubator https://charts.helm.sh/incubator

# Install Zookeeper cluster named 'test-zookeeper'
helm install test-zookeeper incubator/zookeeper

To test whether the Zookeeper cluster runs correctly, run ls from the Zookeeper cli client.

kubectl -n default exec -it test-zookeeper-0 -- bin/zkCli.sh -server test-zookeeper-0.test-zookeeper-headless:2181,test-zookeeper-1.test-zookeeper-headless:2181 ls /

How to run

The following command creates a kafka cluster named 'test-kafka', on top of 'test-zookeeper' Zookeeper cluster.

kubectl apply -f kafka.yaml

To install Kafka client pod or Kafkacat pod, run:

# install kafka client pod
kubectl apply -f kafka-client.yaml

# install kafkacat pod
kubectl apply -f kafkacat.yaml

To test whether the Kafka cluster runs correctly, run:

# list topics with kafka client
kubectl -n default exec -it kafka-client -- kafka-topics --bootstrap-server test-kafka-0.test-kafka-headless:9092,test-kafka-1.test-kafka-headless:9092 --list --topic

# list topics with kafkacat
kubectl -n default exec -it kafkacat -- kafkacat -b test-kafka-0.test-kafka-headless:9092,test-kafka-1.test-kafka-headless:9092,test-kafka-2.test-kafka-headless:9092,test-kafka-3.test-kafka-headless:9092 -L

kafka-client.yaml

apiVersion: v1
kind: Pod
metadata:
  name: kafka-client
  namespace: default
spec:
  containers:
    - name: kafka-client
      # alternative: wurstmeister/kafka
      image: dongjinleekr/kafka:2.13-2.7.0
      command:
        - sh
        - -c
        - "exec tail -f /dev/null"

kafka.yaml

apiVersion: v1
kind: Service
metadata:
  name: test-kafka-headless
  labels:
    app: test-kafka-headless
spec:
  selector:
    app: test-kafka
  ports:
  - port: 9092
    targetPort: 9092
    name: kafka
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test-kafka
  labels:
    app: test-kafka
spec:
  selector:
    matchLabels:
      app: test-kafka
  serviceName: test-kafka-headless
  replicas: 4
  template:
    metadata:
      name: test-kafka
      labels:
        app: test-kafka
    spec:
      initContainers:
      containers:
      - name: test-kafka
        # alternative: dongjinleekr/kafka
        image: wurstmeister/kafka
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9092
          name: kafka
        env:
        - name: BROKER_ID_COMMAND
          value: "[[ `hostname` =~ -([0-9]+) ]] && echo ${BASH_REMATCH[1]}"
        - name: HOSTNAME_COMMAND
          value: hostname
        - name: KAFKA_LISTENERS
          value: PLAINTEXT://:9092
        - name: KAFKA_ADVERTISED_LISTENERS
          value: PLAINTEXT://_{HOSTNAME_COMMAND}.test-kafka-headless:9092
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: test-zookeeper-0.test-zookeeper-headless:2181,test-zookeeper-1.test-zookeeper-headless:2181/test-kafka
        - name: KAFKA_LOG_DIRS
          value: /kafka/kafka-logs
        - name: KAFKA_CLEANUP_POLICY
          value: "compact"
        volumeMounts:
        - name: kafka-storage
          mountPath: /kafka
  volumeClaimTemplates:
  - metadata:
      name: kafka-storage
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 50Mi
 

kafkacat.yaml

apiVersion: v1
kind: Pod
metadata:
  name: kafkacat
  namespace: default
spec:
  containers:
    - name: kafkacat
      image: confluentinc/cp-kafkacat:latest
      command:
        - sh
        - -c
        - "exec tail -f /dev/null"