- Kafka - publish-subscribe messaging system
- All incoming data is placed in Kafka
- All outgoing data is read from Kafka
- Messages organized into topics
- Messages are sent to / read from (published to / subscribed from) topics
- Producers publish/push/emit messages to a topic
- Consumers subscribes to / pulls off messages from a topic
- Topics are divided into partitions
- Each message within partition has an id called offset
- Message can be uniquely identified by the tuple
(topic, partition, offset) - Consumers can be set to read from specific partitions, as a group, all consuemers read from all partitions for a particular topic
- Kafka Streams: https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/
- Kafka in a nutshell: https://sookocheff.com/post/kafka/kafka-in-a-nutshell/
- Sarama - Go Client library for Kafka: https://github.com/Shopify/sarama
- Kafka-Docker -
docker-composebased setup with topic creation on startup: https://github.com/wurstmeister/kafka-docker - Spotify/Kafka - Kafka with Zookeeper: https://github.com/spotify/docker-kafka