Nomad Kafka Stuff

gistfile1.txt

## start kafka-zookeeper cluster
$ nomad job run -no-color kafka-zookeeper.nomad
$ nomad job status -no-color kafka-zookeeper # verify allocations (containers) are running

## wait 30 seconds for kafka-zookeeper to discover and start kafka-brokers
$ nomad job run -no-color kafka-broker.nomad
$ nomad job status -no-color kafka-broker # verify allocations (containers) are running

## verify zookeeper cluster is up
run the following against each node to verify it is either a leader or follower, pull ip ports from kafka-zookeper-client service in consul
$ echo stat | nc  <kafka-zookeeper-client-1 ip> <kafka-zookeeper-client-1 client_port> | grep Mode
Mode: follower
$ echo stat | nc  <kafka-zookeeper-client-2 ip> <kafka-zookeeper-client-2 client_port> | grep Mode
Mode: follower
$ echo stat | nc  <kafka-zookeeper-client-2 ip> <kafka-zookeeper-client-2 client_port> | grep Mode
Mode: leader

## verify kafka cluster
for all these commands pull kafka-zookeeper-client-1 ip and port from consul

### download kafka zookeeper tools onto OS X
$ brew install kafka

### verify number of brokers
$ zookeeper-shell <zookeeper-client ip>:<zookeeper client port> ls /brokers/ids
Connecting to 10.102.44.92:22475
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[1005, 1004, 1003] # => id of each broker

### add topic
$ kafka-topics --bootstrap-server <kafka-client ip>:<kafka-client port> --create  --replication-factor 1 --partitions 1 --topic <topic name>
```

### list topics
$ kafka-topics --bootstrap-server <kafka-client ip>:<kafka-client port> --list

### remove topic
$ kafka-topics --boostrap-server <kafka-client ip>:<kafka-client port> --delete --topic test

### use producer to send messages to a topic
cat test/web.log | kafka-console-producer --broker-list <kafka-client ip>:<kafka-client port>,... --topic test

### use consumer to receive message from topic
kafka-console-consumer --bootstrap-server <kafka-dc-client ip>:<kafka-dc-client port> --topic test --from-beginning

## Docker images
Each of the zookeeper and kafka images required some custom entry point scripts to help set configs and in the case of the zookeeper cluster auto-discover the node ips and create the zookeeper dynamic cfg. These custom images are in the docker directory.

## mTLS configuration for Kafka/Zookeeper
The truststore and keystore are pulled from the Vault PKI and converted into the JKS format within the image's docker-entrypoint.sh

1. truststore => vault pki root + intermediate certificate
2. keystore => vault certificate w/ shared common name across cluster and ip_sans set to the nodes ip address

The passwords used for truststore and keystore JKS formats are pulled from vault.

## Zookeeper quorum
Zookeeper does not support dynamic ensemble (reaching quorum), so zookeeper consul services need to be up before the actual zookeeper services can be started in order to build the dynamic.cfg file. This is accomplished by a service check which just verifies that the zoo.cfg file exists along w/ a customized docker-entrypoint script in the kafka-zookeeper image which will sleep for 20 secs than make a consul-template call to construct the zoo.cfg.dynamic before starting the zookeeper process.