Checking out and running your projects locally is usually sufficient to for your development needs. However, sometimes it can happen that this is not practical or you just don't have the time to set up everything up locally. Tunnelling into a live environment gives you additional options to access Kafka and debug your issue. For instance, it allows things like having access to you bash tools, history, a local clojure repl, etc
# Assuming ~/src
git clone git@github.com:fjsousa/bash-sugar.git
cd bash-sugar
# Create env config:
echo export BROKER_LIST=kafka-4:9092 >> uat.env
echo export SCHEMA_REGISTRY_URL="https://schema-registry.fc-uat.us" >> uat.env
echo export ZOOKEEPER_ADDRESS="localhost:2181" >> uat.env
echo export TUNNEL_ENV=uat >> uat.env
source uat.env
# Configure virtual interfaces
source kafka-utils.sh
# Configure ssh/config:
# Edit `Host` in bash-sugar/ssh-config and IdentityFile
echo Include ~/src/bash-sugar/ssh-config >> ~/.ssh/config
source kafka-tunnel.sh
# Create the tunnel with
create-kafka-tunnel
Add this to your /etc/hosts
:
192.168.4.1 kafka-3
192.168.5.1 kafka-4
192.168.6.1 kafka-5
And consume data with:
avro-consume loan-part-repayment-paid-puma-4
or
`kafkacat -b kafka-3:9092 -C -t raw-legacy-product-unit-unkeyed-1 -f "[%p] %t %s \n"
Having a tunnel open can have tricky side efects so you should clean after yourself:
destroy-kafka-tunnel
Your local Kafka clients will have to resolve the brokers in a given environment. Usually they'll be named kafka-
followed by a number. We'll have to figure these out and edit /etc/hosts
and etc/config
:
First, ssh
into UAT and check which brokers you'll need to forward:
$ consul members | grep kafka
kafka-3 172.35.8.211:8301 alive client 0.8.3 3 us-east-1 <default>
kafka-4 172.35.9.106:8301 alive client 0.8.3 3 us-east-1 <default>
kafka-5 172.35.10.196:8301 alive client 0.8.3 3 us-east-1 <default>
Here we have kafka-3
, kafka-4
and kafka-5
, which we'll have to reach from our local machine. The problem is that the brokers are
listening in the same port (9092), so you'll have to use different network addresses. One way to do this is using aliases with ifconfig
:
sudo ifconfig en0 alias 192.168.4.1 broadcast 192.168.4.255
sudo ifconfig en0 alias 192.168.5.1 broadcast 192.168.5.255
sudo ifconfig en0 alias 192.168.6.1 broadcast 192.168.6.255
(Network wizardry: how to figure out broadcast addresses)
Add this to your /etc/hosts
to resolve the kafka brokers in uat to your virtual interfaces:
192.168.4.1 kafka-3
192.168.5.1 kafka-4
192.168.6.1 kafka-5
And this to your ~/.ssh/config
to tunnel the broker trafic into uat
Host uat-tunnel
User flavio.sousa
HostName bastion.fc-uat.us
IdentityFile ~/.ssh/fc
LocalForward 2181 zookeeper.service.consul:2181
LocalForward 192.168.4.1:9092 kafka-3:9092
LocalForward 192.168.5.1:9092 kafka-4:9092
LocalForward 192.168.6.1:9092 kafka-5:9092
ControlPath /tmp/ssh-%r@%h:%p
This is how your interface configuration should look like:
$ ifconfig en0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ether a0:99:9b:0d:95:a1
inet 192.168.4.1 netmask 0xffffff00 broadcast 192.168.4.255
inet 192.168.5.1 netmask 0xffffff00 broadcast 192.168.5.255
inet 192.168.6.1 netmask 0xffffff00 broadcast 192.168.6.255
inet6 fe80::1488:a680:f789:362f%en0 prefixlen 64 secured scopeid 0x4
inet 172.16.8.137 netmask 0xfffffc00 broadcast 172.16.11.255
nd6 options=201<PERFORMNUD,DAD>
media: autoselect
status: active
Nice! I think we could make a Clojure fixture out of this.