RESP3 specification

Versions history:

  • 1.0, 2 May 2018, Initial draft to get community feedbacks.


The Redis protocol has served us well in the past years, showing that, if carefully designed, a simple human readable protocol is not the bottleneck in the client server communication, and that the simplicity of the design is a major advantage in creating a healthy client libraries ecosystem.

Yet the Redis experience has shown that after about six years from its introduction (when it replaced the initial Redis protocol), the current RESP protocol could be improved, especially in order to make client implementations simpler and to support new features.


Generate a payload.txt file using this Ruby script:

(1..1000000).each{|i| print "#{i} "}

The file will trivially contain "1 2 3 4 5 ..." so that out of order violations are trivial to spot.

  1. In one terminal open netcat in listen mode: nc -l 6379
  2. In another terminal execute netcat like this: cat payload.txt | nc 6379
  3. In the first terminal you should see like 20796 20797 20798 20799 20800... instead of 1 2 3 ...
  4. Does not happen always, so kill the netcat instances, and GOTO 10 to retry until it happens.


That's the C function (you can find it inside the Redis source code, inside rax.c).

raxNode *raxRemoveChild(raxNode *parent, raxNode *child) {
    debugnode("raxRemoveChild before", parent);
    /* If parent is a compressed node (having a single child, as for definition
     * of the data structure), the removal of the child consists into turning
     * it into a normal node without children. */

Why stream items are small hashes instead of single strings like many other Redis types elements is a good question indeed. At the end it's just a design decision, so I don't have the definitive answer. However I can try to explain the design process leading to this design.

What I wanted "Streams" to be, was actually just an Abstract Log. I was not able to call the data structure "log" because it's confusing in many contexts, but that was the idea, and a log better represents what Redis Streams are. Perhaps it's the consumer groups part of the Redis Streams that better characterize the streaming part, but the data structure itself is a log.

Now, what constitutes a log? In the original form, is just lines of text ending with "\n", one after the other, added in an append only fashion. But in general is some data in append only mode.

XADD captures this append only mode of operation. While we have more powerful deletion mechanisms, and will add more, but that is the general idea.

  • XGROUP CREATE <key> <groupname> <id or $>
  • XGROUP SETID <key> <id or $>
  • XGROUP DELGROUP <key> <groupname>
  • XGROUP DELCONSUMER <key> <consumername>
  • XPENDING <key> [<start> <stop>]
  • XCLAIM <key> <group-name> <consumer-name> <min-idle-time> <ID-1> <ID-2> ...
  • XACK <key> <ID-1> <ID-2> ...
  • XREAD-GROUP (wrapper for XREAD that accepts GROUP and CONSUMER options)
  • XINFO <key> [CONSUMERS <groupname>|GROUPS|STREAM|...]. STREAM is the default

Meltdown fix impact on Redis performances in virtualized environments

UPDATE: apparently kernel difference may have a serious impact, so I'll redo the test from scratch.

Test performed with AOF enabled, fsync policy 1 second, allowing the rewrites to be triggered.

Command lines used:

View twitter.c
#define F float
F f[]={-.1,-.1,1.7,.8,-.11,-6,.4,-.6,.15,-.3,.22,-3.1,-.1,-.06,3.2,.57,-1.1,19,-176,-4.8,-7.2,43,-86,40,48,65,34,803,39,9.65,-17,-33,-33,16,1167,63,-23,41,-22,9,.56,-.08,4.76,-8,-.28,.63,-1182,-7.27,26.6,-1213,198,5.3,-8.25,38.4,.22,-.78,-903,-7.4,-.26,.42,47.5,33.7,36.5,-184,14.5,-90,3};F o[15];F R(F A,F U){o[0]=A;o[1]=U;F a=0;int k=2,m=3,n=0,z=0,l=1;for(int j=0;j<67;j++){if(j==18)n=2,z=3,m=7,l=-17;if(j==60)n=8,l=-59;if((j+l)%m){a+=f[j]*o[n+(j+z)%m];}else{o[k++]=(F)1/(1+exp(-(a+f[j])));a=0;}}return o[14];}main(){for(int i=0;i<2201;i++){putchar(R(i%55,i/55-5)>.5?58:32);if(!(i%55))putchar(10);}}
View streams.txt
# Let's add a few entries in our stream "mystream". Every entry in a stream is like
# an hash, composed of fields and values, but every entry can have different fields.
# XADD appends a new entry in a stream, and returns an unique ID which is guaranteed
# be incrementing.
# The first part of the ID is the millisecond unixtime during the
# addition (or the time of the last entry in the stream, if greater than the current time).
# The second part of an entry is a 64 bit counter for entries added in the same ms.> XADD mystream name pamela nicknake kill-9
View fizzlefade.html
<!DOCTYPE html>
<canvas id="framebuffer" width="320" height="200"></canvas>
<script type="text/javascript">
/* Fizzlefade using a Feistel network.
View rdbcheck.txt
36143:M 27 Jun 11:18:39.096 # Server initialized
36143:M 27 Jun 11:18:39.096 # Internal error in RDB reading function at rdb.c:1378 -> Unknown RDB encoding type 7
[offset 0] Checking RDB file dump.rdb
[offset 32] AUX FIELD redis-ver = '999.999.999'
[offset 46] AUX FIELD redis-bits = '64'
[offset 58] AUX FIELD ctime = '1498554991'
[offset 73] AUX FIELD used-mem = '1014768'
[offset 89] AUX FIELD aof-preamble = '0'
[offset 139] AUX FIELD repl-id = 'f7a20a009964381cdcc61f401f91f9b228801647'
[offset 154] AUX FIELD repl-offset = '0'