Quarkus Kafka Streams requirements

requirements.md

Requirements

Summary

There are three Kafka topics t1, t2 and t3 where, t1 has raw data and t2 has look-up values. Message(s) in t1 are used to look-up t2. If a match is found, then the message t1 is enriched with values in t2 and written out to t3. Messages in t1 are expected at a rate of around 100 messages per second. The expected deliverable is a Quarkus project such that, the mvn command results in a working JAR file.

Details

Messages in t1 will be JSON as shown below. Each message will have a UUID as the key. This UUID is same as meta.id. The data array may have hundreds of entries.

{
  "meta" : {
  "id" : "UUID",
  "ts" : "YYYYMMDDTHH:mm:ss.SSSS"
},
  "data" : [
    {"k": "k1", "v": "v1"},
    {"k": "k2", "v": "v2"},
    {"k": "k3", "v": "v3"},
    {"k": "k4", "v": "v4"}
  ]
}

Messages in t2 will be as shown below.

{
  "meta" : {
  "id" : "UUID",
  "ts" : "YYYYMMDDTHH:mm:ss.SSSS"
},
  "data" : [
    {"k": "k1", "l": "l1"},
    {"k": "k2", "l": "l1"}
  ]
}

The Java project must join these two topics - t1 as stream and t2 as table - in the following manner.

1. Start with an empty JSON array object, say, output.
2. For every data[i].k in t1, it should look for a match in data[i].k in t2.
3. If there is a match, then add an entry into output as {"k": "k1", "l": "l1"}. Else, add entry as {"k":"k(unmatched key number)"}
4. Write meta, data from t1 and output (as set in 2 or 3 above) to t3.

For example, with the sample messages for t1 and t2 as shown above, the following will be output message in t3.

{
  "meta" : {
  "id" : "UUID",
  "ts" : "YYYYMMDDTHH:mm:ss.SSSS"
},
  "data" : [
    {"k": "k1", "v": "v1"},
    {"k": "k2", "v": "v2"},
    {"k": "k3", "v": "v3"},
    {"k": "k4", "v": "v4"}
  ],
  "output" : [
    {"k": "k1", "l": "l1"},
    {"k": "k2", "l": "l2"},
    {"k": "k3"},
    {"k": "k4"}
  ]
}