tuxfight3r

Kafka Topics

List existing topics

bin/kafka-topics.sh --zookeeper localhost:2181 --list

Describe a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic mytopic

Purge a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000

... wait a minute ...

tuxfight3r

## Get private key fingerprint for AWS generated key ( will match the id in AWS Console)
$ openssl pkcs8 -in test_key.pem -inform PEM -outform DER -topk8 -nocrypt | openssl sha1 -c
(stdin)= f6:38:9c:53:9c:64:a8:b3:40:23:9f:6c:ed:0e:3d:bf:bf:16:bb:1c

## Get fingerpint from Openssh public key only md5 / sha1
$ ssh-keygen -f test_key.pub -e -m PKCS8 | openssl pkey -pubin -outform DER | openssl md5 -c
(stdin)= 55:b6:90:db:34:29:da:60:68:c9:08:24:f1:9f:c8:4b

(or)

tuxfight3r

use kubectl to patch configmap via json patch operations (add/remove/replace/test)

## Add a new key to the existing configmap
$ kubectl patch cm demo-app-config --type json --patch '[{ "op": "add", "path": "/data/TEST_VALUE", "value": "test_key" }]'
configmap/demo-app-config patched

$ kubectl get cm demo-app-config -o json | jq .data.TEST_VALUE
"test_key"

tuxfight3r

Kafka Cheat Sheet

Display Topic Information

$ kafka-topics.sh --describe --zookeeper localhost:2181 --topic beacon
Topic:beacon	PartitionCount:6	ReplicationFactor:1	Configs:
	Topic: beacon	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: beacon	Partition: 1	Leader: 1	Replicas: 1	Isr: 1

tuxfight3r

Utilities you'll care about

All these are already installed on epyc.

• kafkacat (conda install -c conda-forge kafkacat)

• kt (grab it from https://github.com/fgeller/kt/releases)

• kafka-* (come with kafka, if you yum install if from Confluent's repo, or via Docker if you're so inclined). Warning -- JVM based and dreadfully slow.

• jq (conda install -c conda-forge jq or use your favorite package manager)

tuxfight3r

#code: https://github.com/miguno/avro-cli-examples
# get schema
java -jar ../avro-tools-1.10.1.jar getschema twitter.avro

# get meta
java -jar ../avro-tools-1.10.1.jar getmeta twitter.avro


#create avro
java -jar ../avro-tools-1.10.1.jar fromjson --schema-file twitter.avsc twitter.json > twitter.avro

tuxfight3r

# generate a key if you don't have one already
gpg --full-gen-key

# list the keys in long format and copy the key id after sec rsa4096/<KEY>
gpg --list-secret-keys --keyid-format LONG
 
# export the public key and post into your git profile (ui)
gpg --armor --export 127CAA89CEA6BE66
 

tuxfight3r

Common Options

-#, --progress-bar Make curl display a simple progress bar instead of the more informational standard meter.

-b, --cookie <name=data> Supply cookie with request. If no =, then specifies the cookie file to use (see -c).

-c, --cookie-jar <file name> File to save response cookies to.

tuxfight3r

Virtual Hosts on nginx (CSC309)

When hosting our web applications, we often have one public IP address (i.e., an IP address visible to the outside world) using which we want to host multiple web apps. For example, one may wants to host three different web apps respectively for example1.com, example2.com, and example1.com/images on the same machine using a single IP address.

How can we do that? Well, the good news is Internet browsers

tuxfight3r

# UPDATED 17 February 2019
# Redirect all HTTP traffic to HTTPS
server {
   listen 80;
   listen [::]:80; 	
   server_name www.domain.com domain.com;
   return 301 https://$host$request_uri;
}

# SSL configuration