dpalomar

# Calculate the latest and earliest offsets, and provide the total # of messages in the topic (the difference):

#!/bin/bash
if [[ $# -eq 0 ]] ; then
    echo 'provide the name of a topic to get totals against'
    exit 0
fi
latest=`bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic $1 --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'`
earliest=`bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic $1 --time -2 --offsets 1 | awk -F ":" '{sum2 += $3} END {print sum2}'`
total=`expr $latest - $earliest`

dpalomar

Kafka Topics

List existing topics

bin/kafka-topics.sh --zookeeper localhost:2181 --list

Describe a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic mytopic

Purge a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000

... wait a minute ...

dpalomar

#!/bin/bash

if [ $# -ne 2 ]; then
    >&2 echo "usage: $0 <kafka_binary_folder> <bootstrap.server>"
    >&2 echo
    >&2 echo "It outputs the list of topics consumed by consumergroup as csv."
    >&2 echo "With the following structure:"
    >&2 echo "    <consumer>,<topic>"
    exit 1
fi

dpalomar

Producer

Setup
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test-rep-one --partitions 6 --replication-factor 1
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test --partitions 6 --replication-factor 3

Single thread, no replication

bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test7 50000000 100 -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 batch.size=8196

dpalomar

Kafka frequent commands

Assuming that the following environment variables are set:

• KAFKA_HOME where Kafka is installed on local machine (e.g. /opt/kafka)
• ZK_HOSTS identifies running zookeeper ensemble, e.g. ZK_HOSTS=192.168.0.99:2181
• KAFKA_BROKERS identifies running Kafka brokers, e.g. KAFKA_BROKERS=192.168.0.99:9092

Server

Start Zookepper and Kafka servers

dpalomar

My Openshift Cheatsheet

Examine the cluster quota defined for the environment:

$ oc describe AppliedClusterResourceQuota

Install pkgs using yum in a Dockerfile

dpalomar

SSL upgrades on rubygems.org and RubyInstaller versions

UPDATE 2016-12-01: Please refer to the official guide instead of this process.

UPDATE 2014-12-21: RubyGems 1.8.30, 2.0.15 and 2.2.3 have been released. It requires manual installation, please see instructions below.

---

Hello,

dpalomar

#Container Resource Allocation Options in docker-run You have various options for controlling resources (cpu, memory, disk) in docker. These are principally via the docker-run command options.

##Dynamic CPU Allocation -c, --cpu-shares=0
CPU shares (relative weight, specify some numeric value which is used to allocate relative cpu share)

##Reserved CPU Allocation

dpalomar

Colours in terminal

It's a common confusion about terminal colours... Actually we have this:

• plain ascii
• ansi escape codes (16 colour codes with bold/italic and background)
• 256 colour palette (216 colours + 16 ansi + 24 gray) (colors are 24bit)
• 24bit true colour ("888" colours (aka 16 milion))

printf "\x1b[${bg};2;${red};${green};${blue}m\n"

dpalomar

# base requirements
sudo apt-get update
sudo apt-get install -y software-properties-common
sudo apt-get install -y python-setuptools python-dev libffi-dev libssl-dev git sshpass tree
sudo easy_install pip
sudo -H pip install cryptography

# sudo -H pip install ansible==2.1.1.0 if you don't need virtualenv

# virtualenv