tuxfight3r

KafkaCat Configuration for AWS MSK

Set the below environment variable with the following values

NOTE: Kafkacat is renamed to kcat recently and the config variable should be KCAT_CONFIG for version 1.7 onwards.

# you can export the variable or present the config with -F parameter for kafkacat
export KAFKACAT_CONFIG=/home/tools/persistent/kcat/kafkacat_config

Contents of kafkacat configuration

tuxfight3r

#Simple script to test all the pods liveness/readiness
#creates portforward and curls the endpoint for status
#clears the portforward at the end of the script
#uses bash v3 based implementation.
#Test only the pods which are up.

#!/bin/bash


aggregate_feed="tfeed"

tuxfight3r

# retrieve the default kubernetes service account token
 kubectl get secret $(kubectl get serviceaccounts default -o jsonpath='{.secrets[0].name}') -o json|jq -r .data.token |base64 -D 
 
# Decode it using Python (pip install PyJWT)
import jwt
var="encoded.jwt.token"
jwt.decode(var, options={"verify_signature":False})

tuxfight3r

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-cluster-issuer
spec:
  selfSigned: {}

tuxfight3r

#!/bin/bash

HEIGHT=10
WIDTH=40
CHOICE_HEIGHT=4
BACKTITLE="Cluster Options"
TITLE="Select Cluster"
MENU="Choose one of the following options:"

ENV="${ENV:-dev}"

tuxfight3r

Kafka Topics

List existing topics

bin/kafka-topics.sh --zookeeper localhost:2181 --list

Describe a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic mytopic

Purge a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000

... wait a minute ...

tuxfight3r

## Get private key fingerprint for AWS generated key ( will match the id in AWS Console)
$ openssl pkcs8 -in test_key.pem -inform PEM -outform DER -topk8 -nocrypt | openssl sha1 -c
(stdin)= f6:38:9c:53:9c:64:a8:b3:40:23:9f:6c:ed:0e:3d:bf:bf:16:bb:1c

## Get fingerpint from Openssh public key only md5 / sha1
$ ssh-keygen -f test_key.pub -e -m PKCS8 | openssl pkey -pubin -outform DER | openssl md5 -c
(stdin)= 55:b6:90:db:34:29:da:60:68:c9:08:24:f1:9f:c8:4b

(or)

tuxfight3r

use kubectl to patch configmap via json patch operations (add/remove/replace/test)

## Add a new key to the existing configmap
$ kubectl patch cm demo-app-config --type json --patch '[{ "op": "add", "path": "/data/TEST_VALUE", "value": "test_key" }]'
configmap/demo-app-config patched

$ kubectl get cm demo-app-config -o json | jq .data.TEST_VALUE
"test_key"

tuxfight3r

Kafka Cheat Sheet

Display Topic Information

$ kafka-topics.sh --describe --zookeeper localhost:2181 --topic beacon
Topic:beacon	PartitionCount:6	ReplicationFactor:1	Configs:
	Topic: beacon	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: beacon	Partition: 1	Leader: 1	Replicas: 1	Isr: 1

tuxfight3r

Utilities you'll care about

All these are already installed on epyc.

• kafkacat (conda install -c conda-forge kafkacat)

• kt (grab it from https://github.com/fgeller/kt/releases)

• kafka-* (come with kafka, if you yum install if from Confluent's repo, or via Docker if you're so inclined). Warning -- JVM based and dreadfully slow.

• jq (conda install -c conda-forge jq or use your favorite package manager)