msgpack codec study

mpakcodec_study.md

import "github.com/vmihailenco/msgpack" 有很多限制

test if a type is msgpack "codecable"

func test() {
    msg:=  /* place your typed literals here */
    fmt.Println("msg=", msg)
    msgbytes, _ := msgpack.Marshal(msg)
    fmt.Println("msgbytes=", msgbytes)

    var outvl  /* place your typed here */
    msgpack.Unmarshal(msgbytes, &outvl)
    fmt.Println("outval=", outval)

}

struct OK!

type MSG struct {
    Idter string
    Seq   int
}
msg:=  MSG{Idter: "ready", Seq: 3}

map ok

msg := map[string]int{"af": 4}

array ok

msg := []string{"one", "two", "three"}

hybrid array -- not ok

func test5() {
    msg := []interface{}{"one", 3}
    fmt.Println("msg=", msg)
    msgbytes, _ := msgpack.Marshal(msg)
    fmt.Println("msgbytes=", msgbytes)
    outvl := []interface{}{}
    msgpack.Unmarshal( msgbytes, &outvl)
    fmt.Println("outvl=", outvl)
}

http://mattyjwilliams.blogspot.tw/2013/01/using-go-to-unmarshal-json-lists-with.html

GO doesn't have matched type to represent a heterogeneous array in JS
ex

ary = [ 1, "2nd", { "key":"val"}]

the only way to simultate most alike

ary:=[]interface{}{1, "2nd", map[string]string{key:"val"}}

Accessing a collective variable composed of interface{} types needs quite a lot type casting, which is tedious. With the above example, to properly access all members in that array, your code may look like the following. Please note that map variable requires twice casting.

  fmt.Println( "0:", v[0].(string) + " -- tail" )
  fmt.Println("1:", v[1].(int64)+3)
  v2map := v[2].(map[string]interface{})
  v2mapint := v2map["$"].(int64)
  fmt.Println("2:", v2mapint)

https://github.com/ugorji/go/tree/master/codec is better
You can decode a heterogeneous array into a struct. ex.

type SeqT struct {
  Idstr string
  Seq   int
  Dlmap map[string]int
}

/* []interface{} can be encode/decode to struct */
func test4() {
  membuf := make([]byte, 50)
  enc := codec.NewEncoderBytes(&membuf, h)
  //mm := []string{"aa","bb","cc"}
  mm := []interface{}{"qwer", 2, map[string]int{"$": 4}}
  enc.Encode(mm)
  fmt.Println("encoded to  :", membuf)
  dec := codec.NewDecoderBytes(membuf, h)

  var qq SeqT
  dec.Decode(&qq)
  fmt.Println("decoded back:", qq)
  fmt.Println("Idstr=", qq.Idstr)
  fmt.Println("Seq=", qq.Seq)
  fmt.Println("Dlmap=", qq.Dlmap["$"])
}

test interoperability between ugorji msgpack and smith.
It has been known that ugorji msgpack RPC as client doesn't work with a smith server.

TODO:

• wrap ugorji to work with smith (using the way to we wrap vmihailenco) -- done
• or make smith becomes standard msgpack rpc compliant?
  • compare their wired protocol -- done
• test pure data exchange
  • test a deep & fat object -- done
  • test a deep & fat object can be unmarshal to struct -- done
  • test nodejs Buffer (can it be deserialized to []byte?) -- failed
  • test js Undefined -- failed
• test the case of deep nest collective variables. How to decode/encode between typed data and schemaless binary. -- done
• test ugorji rpc
  • write example -- done
  • study its API design -- done
  • study its wired protocol with wireshark -- done by connection debugger
  • write example to test asynchronize RPC call -- done
• make this implementation asynchronize -- done by net/rpc

find a way to make following conversion

struct -> []interface{} -> msgpack binary

http://golang.org/pkg/reflect/

Solved! simple and neat!

  s := reflect.ValueOf(stvar)
  hary:=make( []interface{}, s.NumField())
  for i:=0; i<s.NumField(); i++ {
    hary[i]= s.Field(i).Interface()
  }

the call api design

  var result int
  cl.Call("mul", []int{3, 4}, &result)

• Call is from net/rpc. net/rpc has some excellent designs.
  • registering APIs in server is very easy. It's just registering an instance and then all methods of this instance will become callable apis.
  • It's encoding and transportation independent. You can adapt it on any encoding (json or msgpack) and any transportation(TCP/UNIX socket)
  • supports both synchronize/asynchronies call. Invoking asynchronies call (Go) is more into promise instead of callback.
  • arguments have to be collected into a single value, but they can be any type.
  • return variable's type can be pre-declared, not restricted to interface{}.

For more examples, please see Richard's talk https://speakerdeck.com/dlackty/ruby-and-friends-taking-go-as-an-example

In Nodejs, quite a lot more msgpack implementations debut after we chose smith/msgpackjs 3 years ago.

• msgpack2 and msgpack3 improve encode/decode performance via native C/C++ implementation.
• msgpack-rpcjs is a pure JS implementation of msgpack rpc.
• framed-msgpack-rpc uses the same technique as smith, which prepends message length to each msgpack packets. It claims that in this way
  receivers can efficiently buffer data until a full packet is available to decode. In an event-based context like node.js, framing simplifies implementation, and yields a faster decoder, especially for very large messages.
• msgpack-stream combines
  • improve performance by utilizing ES6 es.parse
  • smith alike length prepending
  • Its API design suitable for infinite stream process, which is GO idiomatic.

standard msgpack RPC binary

rpc call

  cl.Call("Arith.Multiply", Args{A: 2, B: 99}, result)

generates

 00000000  94 00 00 ae 41 72 69 74  68 2e 4d 75 6c 74 69 70  |....Arith.Multip|
 00000010  6c 79 91 82 a1 41 02 a1  42 63                    |ly...A..Bc|

which is encoding of the following

[0 0 "Arith.Multiply" [[map[A:2 B:99]]]

rpc call

 cl.Call("Arith.Add", []int{55, 33, 77}, result)

generates

 00000000  94 00 01 a9 41 72 69 74  68 2e 41 64 64 91 93 37  |....Arith.Add..7|
 00000010  21 4d 

which is encoding of

[0 1 "Arith.Add" [[55 33 77]]]

call ID is at the second integer. the first integer is always 0 in request and 1 in response.

func (c *msgpackSpecRpcCodec) WriteRequest(r *rpc.Request, body interface{}) error {
    ...
    r2 := []interface{}{0, uint32(r.Seq), r.ServiceMethod, bodyArr}
    ...
}                                                                                                   

smith's binary

4-byte length + msgpack encoded binary
[ map[$ Seq] ... ]

smith's binary

4-byte length + msgpack encoded binary
rpc call

ret = apicall(trconn, "pow", &VarList{3, 4})

encoded as

[pow 3 4 map[$:4]]

returns

[4 81]

rpc call

ret = apicall(trconn, "threesum", &VarList{4, 5, 7})

encoded as

 [threesum 4 5 7 map[$:5]]

returns

[5 16]

net/rpc call flow backtrace

• client make request

*[github.com/changtimwu/ugorji-go/codec.(*rpcCodec).write]
 [github.com/changtimwu/ugorji-go/codec.(*msgpackSpecRpcCodec).WriteRequest]
 [net/rpc.(*Client).send]
 [net/rpc.(*Client).Go]
 [net/rpc.(*Client).Call]

• server reads request reader * 3

*[github.com/changtimwu/ugorji-go/codec.(*rpcCodec).read]
 [github.com/changtimwu/ugorji-go/codec.(*msgpackSpecRpcCodec).parseCustomHeader]
 [github.com/changtimwu/ugorji-go/codec.(*msgpackSpecRpcCodec).ReadRequestHeader]
 [net/rpc.(*Server).readRequestHeader]
 [net/rpc.(*Server).readRequest]
 [net/rpc.(*Server).ServeCodec]

• serve reads request body

*[github.com/changtimwu/ugorji-go/codec.(*rpcCodec).read]
 [github.com/changtimwu/ugorji-go/codec.(*msgpackSpecRpcCodec).ReadRequestBody]
 [net/rpc.(*Server).readRequest]
 [net/rpc.(*Server).ServeCodec]

• server write response

*[github.com/changtimwu/ugorji-go/codec.(*rpcCodec).write]
 [github.com/changtimwu/ugorji-go/codec.(*msgpackSpecRpcCodec).WriteResponse]
 [net/rpc.(*Server).sendResponse]

• client reads response header * 3

*[github.com/changtimwu/ugorji-go/codec.(*rpcCodec).read]
 [github.com/changtimwu/ugorji-go/codec.(*msgpackSpecRpcCodec).parseCustomHeader]
 [github.com/changtimwu/ugorji-go/codec.(*msgpackSpecRpcCodec).ReadResponseHeader]

• client read response body

*[github.com/changtimwu/ugorji-go/codec.(*rpcCodec).read]
 [github.com/changtimwu/ugorji-go/codec.(*rpcCodec).ReadResponseBody]

• net/rpc provides
  • rpc.ClientCodec
  • rpc.ServerCodec
• ClientCodec asks

type ClientCodec interface {
        // WriteRequest must be safe for concurrent use by multiple goroutines.
        WriteRequest(*Request, interface{}) error
        ReadResponseHeader(*Response) error
        ReadResponseBody(interface{}) error
        Close() error
}

• ServerCodec asks

type ServerCodec interface {
        ReadRequestHeader(*Request) error
        ReadRequestBody(interface{}) error
        // WriteResponse must be safe for concurrent use by multiple goroutines.
        WriteResponse(*Response, interface{}) error
        Close() error
}

• rpcCodec is fundamental struct. It implements
  • low-level read/write functions which might be used while implementing ServerCodec and ClientCodec interfaces.

func (c *rpcCodec) BufferedReader() *bufio.Reader 
func (c *rpcCodec) BufferedWriter() *bufio.Writer
func (c *rpcCodec) write(obj1, obj2 interface{}, writeObj2, doFlush bool) (err error)
func (c *rpcCodec) read(obj interface{}) (err error)

• Simple parts of ServerCodec and ClientCodec interface like

func (c *rpcCodec) Close() error 
func (c *rpcCodec) ReadResponseBody(body interface{}) error

• goRpcCodec inherits rpcCodec(has-a relation) and implements BINC specific methods

type goRpcCodec struct {
  rpcCodec
}
func (c *goRpcCodec) WriteRequest(r *rpc.Request, body interface{}) error
func (c *goRpcCodec) WriteResponse(r *rpc.Response, body interface{}) error 
func (c *goRpcCodec) ReadResponseHeader(r *rpc.Response) error
func (c *goRpcCodec) ReadRequestHeader(r *rpc.Request) error

• msgpackSpecRpcCodec inherits rpcCodec(via has-a) and implements msgpack specific methods

type msgpackSpecRpcCodec struct {
  rpcCodec
}
func (c *msgpackSpecRpcCodec) WriteRequest(r *rpc.Request, body interface{}) error
func (c *msgpackSpecRpcCodec) WriteResponse(r *rpc.Response, body interface{}) error
func (c *msgpackSpecRpcCodec) ReadResponseHeader(r *rpc.Response) error
func (c *msgpackSpecRpcCodec) ReadRequestHeader(r *rpc.Request) error
func (c *msgpackSpecRpcCodec) ReadRequestBody(body interface{}) error

goRpcCodec implements both rpc.ServerCodec and rpc.ClientCodec
so the same instance can be passed to both rpc.NewClientWithCodec(cc) and rpc.NewServer().ServerCodec
msgpackSpecRpcCodec is the same story.