This is summary of my GSoC project with Jboss by Red hat.

GSoC_2018_work_product.md

GSOC 2018 Work product submission.

This is summary of my GSoC project with Jboss by Red hat. In this gist i have included some of the major features i implemented during the programe.

• Project Repo : https://github.com/strimzi/amqp-kafka-bridge

• My fork : https://github.com/shubhamvashisht/amqp-kafka-bridge

Strimzi - Bridging HTTP to Apache Kafka

Strimzi is a project about running Apache Kafka on platforms like kubernetes and openshift. It contains two main modules.

• Strimzi-Kafka-operator
• AMQP-Kafka-bridge.

The project idea for GSoC was to add HTTP support to the bridge so that it can listen to http clients and bridge them to Kafka.

• As GSoC is divided into many phases i will breakdown my work done on the project in various phases as well.

The Architecture

The first challege was to design an architecture for HTTP and kafka interactions. At that time bridge supported AMQP and Both protocols have many key differences which definately reflected on the architecture. Some of the key differences were :

• Uni-direction communication (request-response)

• Records could not be stored inside bridge and pushed to client.

• Automatic poll would not be very efficient ( used on-demand polling)

• Handling delivery reports

Architecture : https://docs.google.com/document/d/1eLVzzYaVL6eABs64iNFkfy7lcz2Or3act5LquPIGaE0/edit?usp=sharing

I divided the project into four parts.

• HTTP server Implementation

• Producer API : Producing Records to kafka.

• Consumer API : Consuming Records from kafka.

• Testing

HTTP Server

I used Vert.xHttpServer library to create a Http Server. Here the major part was to distinguish between incoming requests and map them to various corresponding operations on kafka. such as produce,creation of consumer, subscription , consuming, deleting consumer instances etc.

• initial commit for Http server implementation : https://github.com/strimzi/amqp-kafka-bridge/pull/100/commits/84a3e6ee7d6372c10586c1f15dba899b3fddbe21

Producer API

Producer part is strongly related KafkaProducers which produces messages to kafka server. some key features:

• The creation of producer is not handled by the client unlike consumers.

• There exists only one KafkaProducer per connection.

• The clients makes a HTTP POST request to the producer endpoint defined in order to produce records.

Producer PR : https://github.com/strimzi/amqp-kafka-bridge/pull/100/commits/0d5ae1eb119b9eb4f4e14dafc1455b3e7cc9e6a6 In this commit, I implemented the following features/commits :

• route producer requests to HttpSourceEndpoint

• added Json Message converter and producing records

• added response from producer and test for producing simple messages

• fixed types in Message Converter

• send record metadata in delivery response

• added request type indentify utility and documentation about distinguishing requests

• added and fixed logs

• added missing license header to files

• fixed codestyle related errors

• fixed rejected delivery response

• fixed producer tests

• reformat RequestIdentifier

• added doc for producer API

Consumer API

The consumer API is about fetching records from kafka. HTTP clients can use consumer api in order to create consumers, manage subscriptions, manage offsets and consuming records. Some key features :

• Create multiple consumers

• Subsribe to multiple topics(one per request)

• consume records.

• Commit offsets

• Delete consumer instances

All requests to consumer API uses JSON format for body/data. Http responses also uses JSON format.

Consumer PR : https://github.com/strimzi/amqp-kafka-bridge/pull/100/commits/958f5aa0aea1f4b8ec27c5ec61dc42c962165cc8 In this commit i implemented following features.

• added consumer creation : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/065c202a29587b422b00fa0d911b539363bec7d3

• added subscription : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/ffda0129879c4007c8b30557b91ca7b268a0d0bc

• Changed "adhoc" calls to sink.() to single sink.handle() : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/c9b285862bf8a2b59071870c69dd152f8bd600a7

• added consuming records : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/b800d19b52e6c2fc9eda5f97532dca22e31d0799

• Fixed endpoint map for producer : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/5f28cc5f86bff8f09fc7c278b64f1dda25d24e8c

• added consumer deletion upon client request : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/ec73a6faed54831b3c28c2a81f35d3bb02108001

• added commiting offsets : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/827e1648505eb0d0aa6570dd5ab7f6177f84856e

• fixed commit offset handler : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/d2d2f5d9b5361c4d38c27e65e1bd4457b8ed257d

• added doc for consumer API : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/2085ea58d882abb94d81a7ec4904bd342cdf33e8

• fixed NPE while deleting consumer : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/a88d32655f1d4f1f5f31a29950299992dfa06d9d

• changed baseUri String with BRIDGE_HOST and BRIDGE_PORT : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/d22f5340bc11ab3da7336c214c609bfdd5607ac7

• fixed all the responses : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/d40e5bc507b58219b7b054d2d5e16cc270230a9d

• fixed and handled cases when conusme,commit fails : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/1254b0841df8a756c3582890c5d383210d21b1ca

TESTING

For testing i used vert.x unit and JUnit.

Consumer testing

• added test for consumer and fixed partitionAssignHandler : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/1e1bf1bed0b7dfa7619ca63c9cde3845c7f07caa

• • added consumer test for receiving message from partition and offset : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/e7b776b7440f092f394b34bc8362e8f1a2108658

• • added consumer deletion test : https://github.com/shubhamvashisht/amqp-kafka-bridge/commit/bc058b194815fa786d0a6728f246a3c42a774c2a

Producer Testing

• added tests for sending messages

• added test for sending periodic messages

Final work product

Details about the API's and usage of project can be found here in these doc.

• Producer API DOC : https://github.com/strimzi/amqp-kafka-bridge/blob/master/doc/HTTP_Producer_API.md

• Consumer API DOC : https://github.com/strimzi/amqp-kafka-bridge/blob/master/doc/HTTP_Consumer_API.md

Blog posts about the GSoC project

These are few blog posts i wrote during the programe

• https://medium.com/@shubhamvashisht/gsoc18-with-jboss-and-community-bonding-period-afda735fc262

• https://medium.com/@shubhamvashisht/gsoc-2018-wrapping-up-the-project-for-final-submission-3884b494a47b