Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Kafka on Kubernetes, minimal configuration

This gist describes how to create a Kafka cluster on Kubernetes with minimal effort.

Dislike to incubator/kafka helm chart, this approach uses wurstmeister/kafka Docker image or its GraalVM equivalent, dongjinleekr/kafka.

Note: This configuration is intended for dev or testing purpose; it may be used in production environment, but I can't give any guarantees in that respect.

Prerequisites

Install Zookeeper with the following helm command:

# Add helm incubator repository
helm repo add incubator https://charts.helm.sh/incubator

# Install Zookeeper cluster named 'test-zookeeper'
helm install test-zookeeper incubator/zookeeper

To test whether the Zookeeper cluster runs correctly, run ls from the Zookeeper cli client.

kubectl -n default exec -it test-zookeeper-0 -- bin/zkCli.sh -server test-zookeeper-0.test-zookeeper-headless:2181,test-zookeeper-1.test-zookeeper-headless:2181 ls /

How to run

The following command creates a kafka cluster named 'test-kafka', on top of 'test-zookeeper' Zookeeper cluster.

kubectl apply -f kafka.yaml

To install Kafka client pod or Kafkacat pod, run:

# install kafka client pod
kubectl apply -f kafka-client.yaml

# install kafkacat pod
kubectl apply -f kafkacat.yaml

To test whether the Kafka cluster runs correctly, run:

# list topics with kafka client
kubectl -n default exec -it kafka-client -- kafka-topics --bootstrap-server test-kafka-0.test-kafka-headless:9092,test-kafka-1.test-kafka-headless:9092 --list --topic

# list topics with kafkacat
kubectl -n default exec -it kafkacat -- kafkacat -b test-kafka-0.test-kafka-headless:9092,test-kafka-1.test-kafka-headless:9092,test-kafka-2.test-kafka-headless:9092,test-kafka-3.test-kafka-headless:9092 -L
apiVersion: v1
kind: Pod
metadata:
name: kafka-client
namespace: default
spec:
containers:
- name: kafka-client
# alternative: wurstmeister/kafka
image: dongjinleekr/kafka:2.13-2.7.0
command:
- sh
- -c
- "exec tail -f /dev/null"
apiVersion: v1
kind: Service
metadata:
name: test-kafka-headless
labels:
app: test-kafka-headless
spec:
selector:
app: test-kafka
ports:
- port: 9092
targetPort: 9092
name: kafka
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: test-kafka
labels:
app: test-kafka
spec:
selector:
matchLabels:
app: test-kafka
serviceName: test-kafka-headless
replicas: 4
template:
metadata:
name: test-kafka
labels:
app: test-kafka
spec:
initContainers:
containers:
- name: test-kafka
# alternative: dongjinleekr/kafka
image: wurstmeister/kafka
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9092
name: kafka
env:
- name: BROKER_ID_COMMAND
value: "[[ `hostname` =~ -([0-9]+) ]] && echo ${BASH_REMATCH[1]}"
- name: HOSTNAME_COMMAND
value: hostname
- name: KAFKA_LISTENERS
value: PLAINTEXT://:9092
- name: KAFKA_ADVERTISED_LISTENERS
value: PLAINTEXT://_{HOSTNAME_COMMAND}.test-kafka-headless:9092
- name: KAFKA_ZOOKEEPER_CONNECT
value: test-zookeeper-0.test-zookeeper-headless:2181,test-zookeeper-1.test-zookeeper-headless:2181/test-kafka
- name: KAFKA_LOG_DIRS
value: /kafka/kafka-logs
- name: KAFKA_CLEANUP_POLICY
value: "compact"
volumeMounts:
- name: kafka-storage
mountPath: /kafka
volumeClaimTemplates:
- metadata:
name: kafka-storage
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 50Mi
apiVersion: v1
kind: Pod
metadata:
name: kafkacat
namespace: default
spec:
containers:
- name: kafkacat
image: confluentinc/cp-kafkacat:latest
command:
- sh
- -c
- "exec tail -f /dev/null"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment