msgpack codec study

mpakcodec_study.md

import "github.com/vmihailenco/msgpack" 有很多限制

In Nodejs, quite a lot more msgpack implementations debut after we chose smith/msgpackjs 3 years ago.

• msgpack2 and msgpack3 improve encode/decode performance via native C/C++ implementation.
• msgpack-rpcjs is a pure JS implementation of msgpack rpc.
• framed-msgpack-rpc uses the same technique as smith, which prepends message length to each msgpack packets. It claims that in this way
  receivers can efficiently buffer data until a full packet is available to decode. In an event-based context like node.js, framing simplifies implementation, and yields a faster decoder, especially for very large messages.
• msgpack-stream combines
  • improve performance by utilizing ES6 es.parse
  • smith alike length prepending
  • Its API design suitable for infinite stream process, which is GO idiomatic.

standard msgpack RPC binary

rpc call

  cl.Call("Arith.Multiply", Args{A: 2, B: 99}, result)

generates

 00000000  94 00 00 ae 41 72 69 74  68 2e 4d 75 6c 74 69 70  |....Arith.Multip|
 00000010  6c 79 91 82 a1 41 02 a1  42 63                    |ly...A..Bc|

which is encoding of the following

[0 0 "Arith.Multiply" [[map[A:2 B:99]]]

rpc call

 cl.Call("Arith.Add", []int{55, 33, 77}, result)

generates

 00000000  94 00 01 a9 41 72 69 74  68 2e 41 64 64 91 93 37  |....Arith.Add..7|
 00000010  21 4d 

which is encoding of

[0 1 "Arith.Add" [[55 33 77]]]

call ID is at the second integer. the first integer is always 0 in request and 1 in response.

func (c *msgpackSpecRpcCodec) WriteRequest(r *rpc.Request, body interface{}) error {
    ...
    r2 := []interface{}{0, uint32(r.Seq), r.ServiceMethod, bodyArr}
    ...
}                                                                                                   

smith's binary

4-byte length + msgpack encoded binary
[ map[$ Seq] ... ]

smith's binary

4-byte length + msgpack encoded binary
rpc call

ret = apicall(trconn, "pow", &VarList{3, 4})

encoded as

[pow 3 4 map[$:4]]

returns

[4 81]

rpc call

ret = apicall(trconn, "threesum", &VarList{4, 5, 7})

encoded as

 [threesum 4 5 7 map[$:5]]

returns

[5 16]

net/rpc call flow backtrace

• client make request

*[github.com/changtimwu/ugorji-go/codec.(*rpcCodec).write]
 [github.com/changtimwu/ugorji-go/codec.(*msgpackSpecRpcCodec).WriteRequest]
 [net/rpc.(*Client).send]
 [net/rpc.(*Client).Go]
 [net/rpc.(*Client).Call]

• server reads request reader * 3

*[github.com/changtimwu/ugorji-go/codec.(*rpcCodec).read]
 [github.com/changtimwu/ugorji-go/codec.(*msgpackSpecRpcCodec).parseCustomHeader]
 [github.com/changtimwu/ugorji-go/codec.(*msgpackSpecRpcCodec).ReadRequestHeader]
 [net/rpc.(*Server).readRequestHeader]
 [net/rpc.(*Server).readRequest]
 [net/rpc.(*Server).ServeCodec]

• serve reads request body

*[github.com/changtimwu/ugorji-go/codec.(*rpcCodec).read]
 [github.com/changtimwu/ugorji-go/codec.(*msgpackSpecRpcCodec).ReadRequestBody]
 [net/rpc.(*Server).readRequest]
 [net/rpc.(*Server).ServeCodec]

• server write response

*[github.com/changtimwu/ugorji-go/codec.(*rpcCodec).write]
 [github.com/changtimwu/ugorji-go/codec.(*msgpackSpecRpcCodec).WriteResponse]
 [net/rpc.(*Server).sendResponse]

• client reads response header * 3

*[github.com/changtimwu/ugorji-go/codec.(*rpcCodec).read]
 [github.com/changtimwu/ugorji-go/codec.(*msgpackSpecRpcCodec).parseCustomHeader]
 [github.com/changtimwu/ugorji-go/codec.(*msgpackSpecRpcCodec).ReadResponseHeader]

• client read response body

*[github.com/changtimwu/ugorji-go/codec.(*rpcCodec).read]
 [github.com/changtimwu/ugorji-go/codec.(*rpcCodec).ReadResponseBody]

• net/rpc provides
  • rpc.ClientCodec
  • rpc.ServerCodec
• ClientCodec asks

type ClientCodec interface {
        // WriteRequest must be safe for concurrent use by multiple goroutines.
        WriteRequest(*Request, interface{}) error
        ReadResponseHeader(*Response) error
        ReadResponseBody(interface{}) error
        Close() error
}

• ServerCodec asks

type ServerCodec interface {
        ReadRequestHeader(*Request) error
        ReadRequestBody(interface{}) error
        // WriteResponse must be safe for concurrent use by multiple goroutines.
        WriteResponse(*Response, interface{}) error
        Close() error
}

• rpcCodec is fundamental struct. It implements
  • low-level read/write functions which might be used while implementing ServerCodec and ClientCodec interfaces.

func (c *rpcCodec) BufferedReader() *bufio.Reader 
func (c *rpcCodec) BufferedWriter() *bufio.Writer
func (c *rpcCodec) write(obj1, obj2 interface{}, writeObj2, doFlush bool) (err error)
func (c *rpcCodec) read(obj interface{}) (err error)

• Simple parts of ServerCodec and ClientCodec interface like

func (c *rpcCodec) Close() error 
func (c *rpcCodec) ReadResponseBody(body interface{}) error

• goRpcCodec inherits rpcCodec(has-a relation) and implements BINC specific methods

type goRpcCodec struct {
  rpcCodec
}
func (c *goRpcCodec) WriteRequest(r *rpc.Request, body interface{}) error
func (c *goRpcCodec) WriteResponse(r *rpc.Response, body interface{}) error 
func (c *goRpcCodec) ReadResponseHeader(r *rpc.Response) error
func (c *goRpcCodec) ReadRequestHeader(r *rpc.Request) error

• msgpackSpecRpcCodec inherits rpcCodec(via has-a) and implements msgpack specific methods

type msgpackSpecRpcCodec struct {
  rpcCodec
}
func (c *msgpackSpecRpcCodec) WriteRequest(r *rpc.Request, body interface{}) error
func (c *msgpackSpecRpcCodec) WriteResponse(r *rpc.Response, body interface{}) error
func (c *msgpackSpecRpcCodec) ReadResponseHeader(r *rpc.Response) error
func (c *msgpackSpecRpcCodec) ReadRequestHeader(r *rpc.Request) error
func (c *msgpackSpecRpcCodec) ReadRequestBody(body interface{}) error

goRpcCodec implements both rpc.ServerCodec and rpc.ClientCodec
so the same instance can be passed to both rpc.NewClientWithCodec(cc) and rpc.NewServer().ServerCodec
msgpackSpecRpcCodec is the same story.