Reproduced from https://thecodinginterface.com/blog/kafka-source-sink-with-apache-flink-table-api/
PyFlink is compatible with Python>=3.5<3.9
Process:
- Produce events and send to Kafka topic
- Set up streaming service via PyFlink DataStream API
- Read from Kafka source via PyFlink TABLE API
- Process data
- Write to Kafka sink via PyFlink TABLE API
python3 -m venv venv
source venv/bin/activate
docker-compose up -d
http --download https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka_2.11/1.13.0/flink-sql-connector-kafka_2.11-1.13.0.jar
docker exec -it broker kafka-topics --create \
--bootstrap-server localhost:9092 \
--topic sales-usd
docker exec -it broker kafka-topics --create \
--bootstrap-server localhost:9092 \
--topic sales-euros
python sales_producer.py
python sales_processor.py
docker exec -it broker kafka-console-consumer --from-beginning \
--bootstrap-server localhost:9092 \
--topic sales-euros
docker-compose down -v