Simple RPC design

rpc.md

I've designed a lot of RPC protocols in my career. One pattern that's worked well basically goes as follows:

// Client calls: print('Hello World\n')
-> [1, "print", "Hello World!\n"]
// Server sends return value (or lack of return vvalue)
<- [-1]

// Client calls: add(1, 2)
-> [2, "add", 1, 2]
// Server responds with result
<- [-2, 3]

// Client calls a function that throws an error
-> [3, "crasher"]
// Server sends error using lua/go style multiple return values.
<- [-3, null, "There was a problem"]

// Client passes in a callback to be called later
// The function is serialized into something special that both sides agree is a function.
-> [4, "setInterval", {$:1}, 1000]
// server returns with function to cancel the interval
<- [-4, {$:2}]
// Then later the callback fires every second from Server
<- [0, 1] // zero request id mean we don't need a response
// Server...
<- [0, 1] // server is calling anonymous onInterval callback again
// Eventually client wants to cancel interval
-> [0, 2] // client calls anonymous clearInterval function

Or defined generally:

// Request
[request_id, target, ...arguments]
// Response
[-request_id, ...return_values]
// Event
[0, target, ...arguments]

Where request_id is a positive integer, target is a string or positive integer, and arguments and return_values are any serializable values.

So a few things to note:

• Messages are always arrays. These serialize well and are compact in most formats. I tend to use either JSON or CBOR.
• First value is a request id (positive integer), response id (negative integer), or 0 for message that doesn't need response.
• second value if request or message is the function to call. It can be a named endpoint or the integer value of an anonymous function.
• The rest of the values are arguments for request/message and return values for responses
• the convention for errors is two return values null, message. This matches lua style error handling, but can be mapped to many other languages.

I tend to use cbor (used to use msgpack a lot) because it's compact and allows binary values to be passed through. Also cbor allows registering custom types, so serializing functions doesn't need the {$:id} format used in JSON.

This isn't perfect, but it works well for lots of use cases, is very fast and effecient, and is very easy to implement.

The namespace for request/response IDs is defined by the caller. Request 1 by one peer is not the same as request 1 by the other peer. This is why negative is used for responses.

In the same manner, anonymous callbacks are serialized using integers defined by the person who owns the function and sends the function. One interesting side effect is it's not possible to send a function back to it's owner because it would be interpreted as a new function owned by the sender.