tuxfight3r

Consume kafka messages from a specific offset

There are 2 ways of consuming kafka messages from a given topic at a specific offset.

Consume the messages from any offset by creating consumer group

get_group_offset(){
  # describe a given group and sort by partition
/home/tools/confluent/bin/kafka-consumer-groups --bootstrap-server ${KAFKA_BROKER} --command-config ${CONFIG} --timeout 15000 --group $1 --describe | sort -k2,3
}

tuxfight3r

#to be able to view the java jmx internals of an application running in kubernetes as a pod

# set the below environment value in the pod definition
_JAVA_OPTIONS="-Dcom.sun.management.jmxremote
  -Dcom.sun.management.jmxremote.authenticate=false
  -Dcom.sun.management.jmxremote.ssl=false
  -Dcom.sun.management.jmxremote.local.only=false
  -Dcom.sun.management.jmxremote.port=5000
  -Dcom.sun.management.jmxremote.rmi.port=5000
  -Djava.rmi.server.hostname=127.0.0.1"

tuxfight3r

#!/bin/bash
date1="2023-02-08 10:50:33"
date2="2023-02-08 14:10:33"
date1_seconds=$(date -d "$date1" +"%s")
date2_seconds=$(date -d "$date2" +"%s")
duration=$(( $date2_seconds - $date1_seconds ))
echo "Time Elapsed: $(($duration/3600)) hours $(($duration %3600 / 60)) minutes and $(($duration % 60)) seconds."

tuxfight3r

For environments where there is no full fledged printf. The below function does a pretty print of numbers with thousand separator

function

#converts given number to pretty format
function pretty_num(){
     awk '{ len=length($0); res=""; for (i=0;i<=len;i++) { res=substr($0,len-i+1,1) res; if (i > 0 && i < len && i % 3 == 0) { res = "," res } }; print res }'|sed -e's/-,\(.*\)/-\1/g'
}

output

tuxfight3r

#!/usr/bin/env bash
# Allows you to connect to remote endpoints via port forward
set -e

TEMP_POD_NAME=db-jump-server
LOCAL_PORT=3307
REMOTE_HOST=prod.abc123.region-1.rds.amazonaws.com
REMOTE_PORT=3306

function cleanup {

tuxfight3r

KafkaCat Configuration for AWS MSK

Set the below environment variable with the following values

NOTE: Kafkacat is renamed to kcat recently and the config variable should be KCAT_CONFIG for version 1.7 onwards.

# you can export the variable or present the config with -F parameter for kafkacat
export KAFKACAT_CONFIG=/home/tools/persistent/kcat/kafkacat_config

Contents of kafkacat configuration

tuxfight3r

#Simple script to test all the pods liveness/readiness
#creates portforward and curls the endpoint for status
#clears the portforward at the end of the script
#uses bash v3 based implementation.
#Test only the pods which are up.

#!/bin/bash


aggregate_feed="tfeed"

tuxfight3r

# retrieve the default kubernetes service account token
 kubectl get secret $(kubectl get serviceaccounts default -o jsonpath='{.secrets[0].name}') -o json|jq -r .data.token |base64 -D 
 
# Decode it using Python (pip install PyJWT)
import jwt
var="encoded.jwt.token"
jwt.decode(var, options={"verify_signature":False})

tuxfight3r

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-cluster-issuer
spec:
  selfSigned: {}

tuxfight3r

#!/bin/bash

HEIGHT=10
WIDTH=40
CHOICE_HEIGHT=4
BACKTITLE="Cluster Options"
TITLE="Select Cluster"
MENU="Choose one of the following options:"

ENV="${ENV:-dev}"