zaffys

Setup Kubernetes cluster in minutes using Talos

1. Get talosctl

Download and install talosctl binary

wget https://github.com/talos-systems/talos/releases/download/v0.8.1/talosctl-linux-amd64

zaffys

Architecture Design Decision: Proactive Memory-Based Graceful Shutdown for Kafka Streams Pods in OpenShift

Problem Statement

Pods running in an OpenShift Kubernetes environment process complex events using Kafka Streams and KTables, with state stores backed by RocksDB. Over time, these state stores consume increasing memory until the container exceeds its memory limit, crashes, and causes data loss. To mitigate this, we need a mechanism to detect impending memory exhaustion, trigger a graceful shutdown, and allow the pod to restart without data loss.

Key Requirements

1. Proactive Memory Monitoring: Detect when memory usage approaches the container's limit.
2. Graceful Shutdown: Safely terminate Kafka Streams operations to commit state and offsets.
3. State Recovery: Ensure the pod resumes processing from the last committed state after restart.
4. Automation: Minimize manual intervention and integrate with OpenShift/Kubernetes lifecycle management.

zaffys

#!/bin/bash

# ------------------- Logging Function -------------------
log() {
  echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a logs/get-deployment-resources-$(date +"%Y%m%d%H%M%S").log
}

# Ensure required directories exist
mkdir -p logs data