Skip to content

Instantly share code, notes, and snippets.

@toderesa97
Last active September 28, 2020 20:11
Show Gist options
  • Save toderesa97/b0ed6ff550110ebb00e12daf69e87f03 to your computer and use it in GitHub Desktop.
Save toderesa97/b0ed6ff550110ebb00e12daf69e87f03 to your computer and use it in GitHub Desktop.
Apache Kafka pub/sub basic example in MSK
#!/bin/bash
./kafka_2.12-2.3.1/bin/kafka-topics.sh --create --topic foobar --bootstrap-server b-1.demo-cluster-1.8hn95l.c3.kafka.eu-west-1.amazonaws.com:9092 --partitions 4 --replication-factor 2
from kafka import KafkaConsumer
consumer = KafkaConsumer('foobar', bootstrap_servers=['b-1.demo-cluster-1.8hn95l.c3.kafka.eu-west-1.amazonaws.com:9094','b-2.demo-cluster-1.8hn95l.c3.kafka.eu-west-1.amazonaws.com:9094'], security_protocol="SSL")
for msg in consumer:
print(msg)
from kafka import KafkaProducer
from datetime import datetime
import time
producer = KafkaProducer(bootstrap_servers=['b-1.demo-cluster-1.8hn95l.c3.kafka.eu-west-1.amazonaws.com:9094','b-2.demo-cluster-1.8hn95l.c3.kafka.eu-west-1.amazonaws.com:9094'], security_protocol="SSL")
while True:
now = datetime.now()
msg = str.encode("random event at {}".format(now.strftime("%m/%d/%Y, %H:%M:%S")))
producer.send('foobar', msg)
print("Sent message at {}".format(now.strftime("%m/%d/%Y, %H:%M:%S")))
time.sleep(3)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment