-
Support polling on multiple streams in one command to avoid having a TCP connection per stream (similar to BLPOP) Potentially, this list could be very long, e.g. a distributed chat server where each stream represents a chat log.
-
Support compaction similar to what's done in Kafka, with the current design there's currently no way to do that AFAIK.
-
TREAD's GROUP feature is a very simple compared to Kafka consumer groups and will suit many cases. What it doesn't guarantee that messages with the same key will be sent to the same consumer in the group. This prevents consumers from being able to run fast distributed stateful aggregations like count / groupby.
-
Allow modules to poll streams
One of the strongest features Kafka offers is Kafka Streams which is a stream processing API on top of Kafka (JVM only).
I'd love to see Redis support some of that functionality so everyone could enjoy the benefits of materialized views, not only JVM lovers.
For reference please see these links:
- Intro to Kafka Streams
- Turning the database inside out (Outdated but important)
- Interactive Queries in Kafka Streams
I had an idea where we could provide a lot the functionality Kafka Streams has directly into redis which might eliminate the need for (3) above. We could add stateful aggregations directly into Redis similar to Kafka Streams or Couchbase's views.
- Latest state
MATERIALIZE <key> USING HSET <view>
- Counters
MATERIALIZE <key> KEYS USING HINCRBY <view> $ 1
- Distinct keys
MATERIALIZE <key> KEYS USING SADD <view>
- Distinct values
MATERIALIZE <key> VALUES USING SADD <view>
- Groupby
MATERIALIZE <key> USING RPUSH
- Groupby with prefix
MATERIALIZE <key> USING RPUSH group::$k $v
- Groupby using the JSON module
MATERIALIZE <key> USING JSON.ARRAPPEND
More complex filters / maps could be implemented outside of redis. I have some thoughts on how implement time windows and joins too but it's not baked enough.