The state of MessagePack in Lua

messagepack-in-lua.md

The state of MessagePack in Lua

The new MessagePack website says that MessagePack for Lua is "coming soon". I thought it would be a good idea to make a quick recap on the state of MessagePack in Lua.

The historical implementation of MessagePack is mplua by Nobuyuki Kubota. It is the implementation linked from the MessagePack wiki. However, it is rather old (no updates for two years) and not very fast.

A bit less than a year ago I had to use MessagePack in LuaJIT and I wrote a FFI-based binding on the official C interface of MessagePack. I had to include a small C library that had to be compiled to wrap inline functions in the interface.

I did not like that design and it had performance issues in real use cases, mostly because I had to use calls that break the JIT mode in LuaJIT. I decided to re-implement MessagePack directly in LuaJIT with no calls to any external library but still using FFI ctypes. I called it luajit-msgpack-pure. This is the implementation I have been using to this day, it is a good solution for LuaJIT.

Starting from my code base, the awesome Kengo Nakajima ported it to pure Lua (5.1) -> lua-msgpack. His implementation is slower than luajit-msgpack-pure, mostly because there is no way to make a buffer as efficient as with FFI ctypes in pure Lua (EDIT: François Perrad proved this wrong).

Dissatisfied with the speed, he wrote lua-msgpack-native which uses a C library. I think this is probably the fastest implementation of MessagePack for any Lua platform as of now. It targets Luvit and can apparently also work with LuaJIT. I am not sure about PUC Lua (5.1, 5.2?).

A similar implementation is Salvatore Sanfilippo's lua-cmsgpack. It was originally written for the Redis project. It is a single self contained C file with no dependencies and targets Lua 5.1.

So, to sum things up:

• If you use LuaJIT and are OK with C modules -> lua-msgpack-native
• If you use PUC Lua 5.1 and are OK with C modules -> lua-cmsgpack
• If you want pure LuaJIT -> luajit-msgpack-pure
• If you want pure Lua 5.1 -> lua-msgpack

Regarding Lua 5.2: I don't now. I plan to try to make luajit-msgpack-pure compatible with Lua 5.2 + luaffi someday, it should not be too hard. Ping me if you need this! It is probably not too hard to port lua-cmsgpack to Lua 5.2 either.

lua-cmsgpack should work for LuaJIT as well, no?

Probably, I haven't tried it (with anything actually). Antirez says he hasn't compared it to lua-msgpack-native.

Cool!
There is a similar situation with MessagePack for Go (historical implementation which is not actively maintained and another implementation).

Here let me explain why I wrote "coming soon.":

In my opinion, as an open source project, single project with many developpers is better than many projects with few developpers. In addition, from the users' point of view, "MessagePack" should be an unified project which provides a reliable implementation for each language to establish a good reputation and get widely adopted (Adoption is essentially important because msgpack is a data exchange format which should be universal).

But it's true that sometimes we can't avoid having multiple implementations. For example, the author (or major committers) may quit maintenance. It likely occurs in MessagePack projects because historically, most of them started as a weekend hack. In fact, I once suffered this issue with PHP and C# implementations, and they were replaced by other implementations.

Or, some languages need multiple implementations for a technical reason such as: LuaJIT C module/PCU Lua C module/Pure Lua, Cython/PyPy/Pure Python, CRuby/JRuby/mruby/Rubinius/Pure Ruby, etc.

For now, I don't have a perfect idea how to write that section. If you have an idea, I'd like you to let me know.

OK, I will try to learn more about lua-cmsgpack first (it looks like it could be a good default choice for most use cases). I also intend to write tests validating some implementations against others in the next few days / weeks. Then I'll try to sum it up and send you the result.

I don't see why lua-msgpack-native wouldn't work for PUC lua too?
Otherwise if you really want speed; consider using d'arc: http://hdiedrich.github.com/darc/

Updated to mention lua-MessagePack. My current opinion is: lua-cmsgpack if you tolerate C extensions, lua-MessagePack otherwise. luajit-msgpack-pure is still useful for some edge cases. More here.

@daurminator Sorry I didn't answer you. For lua-msgpack-native as I wrote in the gist I don't know. Regarding d'Arc it is definitely not something I would use in a library.

The module names has ran out.