Google Summer of Code 2023 with Knative under CNCF

Google Summer of Code 2023.md

Google Summer of Code 2023 with Knative under CNCF🔗

Hello everyone. I am Debasish Biswas.

During this summer, as part of Google Summer of Code, I contributed to Knative Eventing, an open-source project that comes under the Cloud Native Computing Foundation (CNCF). I am going to share my GSoC experience and project details.

Project Overview:

What is Knative Eventing?

Knative eventing is a system designed to meet a frequent demand for cloud-native development. It offers composable primitives that enable late-binding between event sources and event consumers. The Knative Eventing system's Eventing Kafka Broker is the part that offers a mechanism to consume and output events using Apache Kafka as the underlying messaging infrastructure.

Project:

In the previous scenario, the Knative Eventing Kafka Broker's data-plane communication with Apache Kafka for consuming and producing records is done only via the Vert.x-Kafka-client library, which is a wrapper for communications with Apache Kafka inside the Vert.x threading model. I aimed to implement the Knative Kafka Broker data-plane communication with the native Apache-kafka-client library working on Java 20 and evaluate OpenJDK 20's Project Loom and leverage its virtual threads.

Actual issue: Dataplane migration for Apache Kafka communications: From Vert.x to Project Loom #2928

Contribution Details:

Community Bonding Period:

During my community bonding period, along with learning and trying to fix other issues, I and my mentor finalized the overall implementation details and the architecture we will follow. We decided that we would create two modules for each dispatcher and receiver module. One of them will have Vert.x implementation and the other will have Virtual Thread implementation, both of them will be implemented through a common Reactive Interface that will be used by the base modules' common code.
Modules:

---├── receiver          (Base receiver module with the ReactiveProducer interface)
   ├── receiver-vertx    (Vertx implementation for Kafka Producer communication)
   ├── receiver-loom     (Loom implementation for Kafka Producer communication)
   ├── dispatcher        (Base dispatcher module with ReactiveConsumer)
   ├── dispatcher-vertx  (Vertx implementation for Kafka Consumer communication)
   ├── dispatcher-loom   (Loom implementation for Kafka Consumer communication)

Before Mid-term Evaluation:

1. Create New modules for Vert.x implementation of receiver and dispatcher.
2. Migrate the base dispatcher and receiver to use the reactive interface with Vert.x implementation.
3. Remove all the vertx kafka client library dependencies from the base module.
4. Update the unit test to use Java 20.

Pull Requests:

• Adding modules that will have the Vert.x implementations of the kafka communication #3106
• ReactiveKafkaProducer Interface and it's Vert.x Implementation #3131
• Migrate base receiver module to use ReactiveKafkaProducer instead of KafkaProducer #3142
• ReactiveKafkaConsumer Interface and it's Vert.x Implementation #3159
• Migrate current dispatcher to use ReactiveKafkaConsumer instead of Vertx KafkaConsumer. #3177
• Upgrade workflow's java version to 20 #3186
• Remove the use of KafkaProducer from the dispatcher and use ReactiveKafkaProducer instead. #3206

After Mid-term Evaluation:

1. Create the Modules and test all the configuration works with Java 20 with the preview feature enabled.
2. Upgrade the E2E test infra to use Java20.
3. Implement both receiver-loom and dispatcher-loom modules utilizing virtual threads.
4. Written unit test ensuring the Implementation works properly.
5. Upgrade the Dataplane test to run for both Vert.x and Loom module implementation.
6. Add and Modify script to deploy and do CI with these new modules.
7. Added customer Tracing for the Consumer.
8. Added Installation Docs for the new implementation.

Pull Requests:

• Remove ConsumerRebalanceListener from ConsumerVerticleContext #3207
• Add Receiver loom based module Implementation #3211
• Add Receiver loom based module Implementation #3215
• Add Dispatcher loom based module Implementation #3257
• Infra Update Java Version to 20 #119
• Consumer tracer to replace the use of Vert.x Kafka lib in base dispatcher module.
• Added Scripts to deploy the Loom modules
• Fixing: NullPointerException occurring and it's discarding the records while tracing for Loom deployment
• Added Installation Docs for Loom Dispatcher and Receiver Implementations

Conclusion:

In conclusion, my Google Summer of Code journey with Knative Eventing has been incredibly rewarding. Now the data-plane communication with the kafka broker could be done via the Native Apache Kafka client library. This experience has not only enhanced my technical skills but also fostered a deeper appreciation for collaborative open-source development. I'm excited about the positive impact my work will have on Knative Eventing and beyond, and I look forward to continuing my journey in the world of open-source.

In the end, I would like to thank my mentors, Ali Ok, Christoph Stäbler, Matthias Wessendorf, and especially Pierangelo Di Pilato who have supported and shared their knowledge with me throughout the whole time.