Abdelrahman-Adnan

Part 6: Troubleshooting, Best Practices, and Learning Path 🚦

Common issues and quick fixes

Problem	Likely reason	Fix
No rows appear in sink	Watermark not advancing	Set env.set_parallelism(1) for single-partition topic
Consumer reads nothing	Started at latest	Use auto_offset_reset='earliest'
Duplicated counts	Topic contains old messages from prior runs	Delete/recreate topic and rerun producer
Flink job runs but no output	Wrong timestamp parsing	Verify TO_TIMESTAMP format matches payload

Abdelrahman-Adnan

Part 5: End-to-End Flink Jobs (Questions 4-6 Pattern) 🛠️

Common job skeleton

All three Flink jobs follow the same structure:

1. Create source table from Kafka (green-trips)
2. Create sink table in PostgreSQL
3. Execute INSERT INTO ... SELECT ... with window aggregation
4. Query sink table in Postgres to validate result

Abdelrahman-Adnan

Part 4: PyFlink Basics — Event Time, Watermarks, Windows ⏱️

Why Flink after Kafka?

Kafka stores/streams events. Flink computes continuous transformations over those events.

Kafka answers: "Can we move events reliably?" Flink answers: "Can we compute useful stateful analytics in real time?"

Event time vs processing time

Abdelrahman-Adnan

Part 3: Producer and Consumer Pipeline in Python 🐍

Step A: Create the topic

docker exec -it workshop-redpanda-1 rpk topic create green-trips

Step B: Build the producer

Abdelrahman-Adnan

Part 2: Kafka and Redpanda Architecture for Beginners 🧱

Redpanda vs Kafka (practical view)

In this module, Redpanda is used as a Kafka-compatible broker. That means Kafka client libraries work without code changes.

Why this is useful:

• Faster local setup
• Familiar Kafka protocol

Abdelrahman-Adnan

Module 7 Summary - Stream Processing with Redpanda, Kafka, and PyFlink

#DataEngineeringZoomcamp #Streaming #Kafka #Redpanda #PyFlink #DataEngineering

---

Part 1: Stream Processing Fundamentals 🌊

What is stream processing?

Abdelrahman-Adnan

Part 6: Spark Internals & Monitoring 🔍

How Spark Executes Your Code

When you trigger an action (like .count() or .show()), Spark goes through several stages:

Your Code
    │

Abdelrahman-Adnan

Part 5: Spark SQL & Joins 🔗

Spark SQL: Write SQL Instead of Python

If you're more comfortable with SQL, Spark lets you run full SQL queries. First, register your DataFrame as a temporary "view" (like a temporary table):

# Register DataFrame as a SQL view
df.createOrReplaceTempView("trips")

Abdelrahman-Adnan

Part 4: Transformations & Actions — How Spark Executes 🧠

Lazy Evaluation: Spark's Secret Weapon

One of the most important things to understand about Spark is lazy evaluation. Spark doesn't actually DO anything until it absolutely has to:

Your code:                          What Spark does:

df = spark.read.parquet(...) → "I'll remember to read this" (no action yet)

Abdelrahman-Adnan

Part 3: Working with DataFrames & Parquet 📊

What is a DataFrame?

If you've used pandas, you already know what a DataFrame is — it's a table with rows and columns. Spark DataFrames work the same way, but distributed across multiple machines:

Pandas DataFrame (lives on ONE machine):
┌─────────────┬──────────┬──────────┐
│ pickup_time │ distance │   fare   │