Skip to content

Instantly share code, notes, and snippets.

View dpalomar's full-sized avatar

David Palomar dpalomar

View GitHub Profile
# Calculate the latest and earliest offsets, and provide the total # of messages in the topic (the difference):
#!/bin/bash
if [[ $# -eq 0 ]] ; then
echo 'provide the name of a topic to get totals against'
exit 0
fi
latest=`bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic $1 --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'`
earliest=`bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic $1 --time -2 --offsets 1 | awk -F ":" '{sum2 += $3} END {print sum2}'`
total=`expr $latest - $earliest`
@dpalomar
dpalomar / kafka-cheat-sheet.md
Created May 7, 2021 10:11 — forked from ursuad/kafka-cheat-sheet.md
Quick command reference for Apache Kafka

Kafka Topics

List existing topics

bin/kafka-topics.sh --zookeeper localhost:2181 --list

Describe a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic mytopic

Purge a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000

... wait a minute ...

@dpalomar
dpalomar / kafka-consumer-by-topic.sh
Created May 7, 2021 08:04 — forked from padilo/kafka-consumer-by-topic.sh
Script to get from Kakfa the list of topics consumed by each consumer group
#!/bin/bash
if [ $# -ne 2 ]; then
>&2 echo "usage: $0 <kafka_binary_folder> <bootstrap.server>"
>&2 echo
>&2 echo "It outputs the list of topics consumed by consumergroup as csv."
>&2 echo "With the following structure:"
>&2 echo " <consumer>,<topic>"
exit 1
fi
@dpalomar
dpalomar / benchmark-commands.txt
Created May 6, 2021 18:29 — forked from functicons/benchmark-commands.txt
Kafka Benchmark Commands
Producer
Setup
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test-rep-one --partitions 6 --replication-factor 1
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test --partitions 6 --replication-factor 3
Single thread, no replication
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test7 50000000 100 -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 batch.size=8196
@dpalomar
dpalomar / Kafka commands.md
Created March 21, 2019 17:19 — forked from vkroz/Kafka commands.md
Kafka frequent commands

Kafka frequent commands

Assuming that the following environment variables are set:

  • KAFKA_HOME where Kafka is installed on local machine (e.g. /opt/kafka)
  • ZK_HOSTS identifies running zookeeper ensemble, e.g. ZK_HOSTS=192.168.0.99:2181
  • KAFKA_BROKERS identifies running Kafka brokers, e.g. KAFKA_BROKERS=192.168.0.99:9092

Server

Start Zookepper and Kafka servers

@dpalomar
dpalomar / openshift-cheatsheet.md
Created January 28, 2019 10:18 — forked from rafaeltuelho/openshift-cheatsheet.md
My Openshift Cheatsheet

My Openshift Cheatsheet

Examine the cluster quota defined for the environment:

$ oc describe AppliedClusterResourceQuota

Install pkgs using yum in a Dockerfile

@dpalomar
dpalomar / howto-manually-add-trust-cert-to-rubygems.md
Created January 19, 2017 20:03
Workaround RubyGems' SSL errors on Ruby for Windows (RubyInstaller)

SSL upgrades on rubygems.org and RubyInstaller versions

UPDATE 2016-12-01: Please refer to the official guide instead of this process.

UPDATE 2014-12-21: RubyGems 1.8.30, 2.0.15 and 2.2.3 have been released. It requires manual installation, please see instructions below.


Hello,

@dpalomar
dpalomar / resource_alloc_docker.md
Created December 27, 2016 10:02 — forked from afolarin/resource_alloc_docker.md
Resource Allocation in Docker

#Container Resource Allocation Options in docker-run You have various options for controlling resources (cpu, memory, disk) in docker. These are principally via the docker-run command options.

##Dynamic CPU Allocation -c, --cpu-shares=0
CPU shares (relative weight, specify some numeric value which is used to allocate relative cpu share)

##Reserved CPU Allocation

@dpalomar
dpalomar / TrueColour.md
Created November 15, 2016 12:22 — forked from XVilka/TrueColour.md
True Colour (16 million colours) support in various terminal applications and terminals

Colours in terminal

It's a common confusion about terminal colours... Actually we have this:

  • plain ascii
  • ansi escape codes (16 colour codes with bold/italic and background)
  • 256 colour palette (216 colours + 16 ansi + 24 gray) (colors are 24bit)
  • 24bit true colour ("888" colours (aka 16 milion))
printf "\x1b[${bg};2;${red};${green};${blue}m\n"
@dpalomar
dpalomar / Operaciones-Git
Last active January 29, 2019 09:39 — forked from jelcaf/Operaciones-Git
Git Tips - Mini-trucos de Git para facilitarme la tarea
#############################################
# Push de la rama actual
git push origin $rama_actual
#############################################
# Volver a un commit anterior, descartando los cambios
git reset --HARD $SHA1
#############################################
# Ver y descargar Ramas remotas