Welcome to the kafka workshop @LINE Taiwan!
Today session will help you get started with doing some basic stream processing using Kafka Streams and KSQL, and how you can manage materialzed views using Kafka Connect.
In what follows, we will give you a detail step-by-step guide on building your first stream processing application. The application considers 2 data sources:
- a Kafka topic with stores
- a Kafka topic with photos (contain a reference to the store they belong to, i.e. foreign key)
The streaming app's task is to join the stores with the photos, and output a new topic with an enriched store, that contains stores with their photos.
The output topic will then be persisted via Kafka Connect to:
- MySQL
- Elasticsearch
Every time a store or photo record is added to the topics, the output stream will be updated, and consequently the databases will reflect this changes.
There are 2 versions:
- KSQL version (declarative-style)
- Kafka Streams DSL version using Kotlin (functional-style)
You can work on either one, but we strongly recommed trying both.
To learn more about reorienting your database architecture around streams and materialized views, we recommend you to watch this Martin Kleppman talk (transcript):
- Docker
- Gradle (for the Kotlin part)
The docker-compose.yml file contains:
- zookeeper
- broker
- schema-registry
- connect
- ksql-server
- ksql-cli
- elasticsearch
- kibana
- mysql
- adminer
Apart from Kafka components, we also include Elasticsearch, Kibana and MySQL and Adminer.
Start all the services by running:
> docker-compose up> docker exec -it ksql-cli ksql http://ksql-server:8088