The rest of this guide will be based off the following string:
DSC8i2aZu6y0tMLNB0PBhLM2MZw2U2EWgeL4CeghO42QyxI30UveL3DsdAx2U
QyIb3vboyg2yF0dP4gajEf17c76B2cMtNZ1G57xk0FLz5s4BRVnw0VVs3G12n
xvw1eQUv60Vg9an3KWUKb3JJHBQ3iyF290xR1mT1RrEJp2pekYH30fFut1ToS
vC1ek5Gl0bA9OP2UiVaA4UzsKP0yT4oR31Glje0wYKFh2Cajqe0qc3ER0PIc5
Y0bWajY0K6MzE3UOlFU0wYPRf0lG4Sa29vMUB0tHOcY08dZj43ZQUlU49Zv
note: line-breaks added for legibility
First, the header:
- The first two characters
DSidentify this as Desynced data, and can be otherwise ignored - The next character,
C, identifies this as a behaviour
The remaining characters 8i2aZ... must then be converted to 6-bit numbers:
- digits
0-9become 0-9 (0x00 - 0x09 in hex) A-Zbecomes 10-35 (0x0a - 0x23)a-zbecomes 36-61 (0x24 - 0x3d)
Note that there is no encoding for 0x3d and 0x32 (111110 and 111111 in binary), so we can't simply concatenate this sequence of 6-bit numbers to make a sequence of 8-bit bytes. Base 64 fills in the blanks here with + and /, but we're not using Base64...
Next, we read a single integer from stream of base-62 encoded digits, to do this we apply this formula to each successive 6-bit number n until n ≥ 31 - and then return u
u = (u * 31) + (n % 31)
This specifies the size of the subsequent compressed data, or is 0 if it's uncompressed.
Before further processing, the data must be converted to true 8-bit bytes. The algorithm to do this in Javascript can be found here
todo: link to the scala version as well once published, and give an actual explanation once I fully understand it
This byte steam is then passed through "Deflate" if a compressed length was specified, otherwise it's passed straight to the next stage of processing. Deflate is readily available on almost every language/platform in existence.
With Base62 decoding and optional decompression complete, the data is ready to be parsed into a more useable form.
For this, Stage have used a custom derivative of the msgpack format, with some notable exceptions and special handling due to Lua oddities.
Fixed positive and negative numbers, nils, floats, integers, and unsigned integers all follow the msgpack format. Msgpack binary and extension data is unsupported, and arrays and maps use a different representation.
The convention below is to denote hex sequences with the prefix 0x and binary sequences with no prefix
Listed here for convenience
- null:
0xc0 - false:
0xc2 - true:
0xc3 - invalid:
0xc4 - user data:
0xc1(considered invalid in clipboard strings) - float32:
0xcafollowed by 4 bytes - float64:
0xcbfolowed by 8 bytes - uint8:
0xccfollowed by 1 bytes - uint16:
0xcdfollowed by 2 bytes - uint32:
0xcefollowed by 4 bytes - uint64:
0xcffollowed by 8 bytes - int8:
0xd0followed by 1 bytes - int16:
0xd1followed by 2 bytes - int32:
0xd2followed by 4 bytes - int64:
0xd3followed by 8 bytes - fixstring:
101xxxxxfollowed by a String ofxxxxxbytes - string8:
0xd9followed by 1 byte of length then a string of that many bytes - string16:
0xdafollowed by 2 bytes of length then a string of that many bytes - string32:
0xdbfollowed by 4 bytes of length then a string of that many bytes
The u prefix on uint denotes unsigned numbers
These are used in table representations, and start with the sequence aaaaaaab
If b=0 then the 7-bit number aaaaaaa is returned.
Otherwise, a second byte cccccccd is read and the low bits are concatenated to give the 14-bit number aaaaaaaccccccc
This is repeated until a number is read without the high bit being set.
As an untyped language, Lua makes no distinction between Maps and Arrays (which are just seen as Maps with integer keys). The encoded data format reflects this, with both data types being read in much the same way.
There are three forms of map. A fixed map, a map16, and a map32.
- fixmap:
1000xxxy - map16:
0xdefollowed by the two-byte sequencexxxxxxxx xxxxxxxy - map32:
0xdfthen a 4-byte sequence ending with a singleybit
These all denote a map containing 2 ^ xxx explicitly keyed entries, where y is a flag indicating if it also contains array-type entries (with the key being derived from the position). This is followed by:
- If
y = 1then a packed-int denoting the length of the array portion - a packed-int that can be ignored but always seems to be
0x01[todo: this is due to Lua, need to figure out what it actually is] - the array-type entries
- the map-type entries
This means a fixmap can specify up to (2^7=) 128 entries, a map16 up to 2^65536 entries (which has almost 20 digits), and a map32 serves no rational purpose.
Arrays follow a similar scheme:
- fixarray:
1001xxxxwhere thexxxxdenotes the true size (not2^xxxas with maps - array16:
0xdcfollowed by a size byte - array32:
0xddfollowed by two size bytes
Then followed by the array entries.
Tables entries can be of two types. A map-type entry consists of two pieces of data: first the value, then its key. Either of these can be any of the formats described in this document (with the exception that tables can't be keys). In practice, you'll likely only see strings and unsigned integers for keys. An array-type entry is a single value keyed by position.
For Lua compatibility, you should treat first index of an array as 1, not 0.
A Map-type pair will always be followed by a packed-integer, this can be safely ignored when reading the format [note: yes, more Lua weirdness, need to find and document how this is calculated]
Entries are read (and written, obviously) in blocks of 8, with each such block being preceded by a byte of "vacancy bits". If this was e.g. 00000010 (bit 2 is set) then you skip attempting to read the 2nd entry from the block and proceed straight to the third. You still, however, increment the index for the purpose of keying array-type entries. This mechanism allows you to have a number of map-type entries that isn't an exact power of two.
This is from the string at the start of this page, after stripping the prefix, converting from base62, and inflating. Each byte is shown in decimal, hex, and binary, with square brackets used to denote relevant sub-sequences of bits with a given binary sequence. For brevity, all the bytes within strings are not shown - just the leading one.
Indentation denotes that we're processing entries in a map
133 0x85 [1000][010][1] -> fixed map, size=2^2=4 +array
016 0x10 [0001000][0] -> array size 8
002 0x02 [0000001][0] -> padding packed-int = 1
000 0x00 [00000000] -> vacancy bits
--- numbered entry 1 / 8
128 0x80 [1000][000][0] -> fixed map, size=2^0=1
--- numbered entry 1 / 1
002 0x02 [0000001][0] -> padding packed-int = 1
000 0x00 [00000000] -> vacancy bits
170 0xaa [101][01010] -> fixed string "disconnect"
162 0xa2 [101][00010] -> fixed string "op"
000 0x00 [0000000][0] -> packed int / lua hack = 0
--- numbered entry 2 / 8
129 0x81 [1000][000][1] -> fixed map, size=2^0=1 +array
002 0x02 [0000001][0] -> array size = 1
002 0x02 [0000001][0] -> padding packed-int = 1
000 0x00 [00000000] -> vacancy bits
--- numbered entry 1 / 1
001 0x01 [0][0000001] -> +ve int = 1
--- named entry 1 / 1
166 0xa6 [101][00110] -> fixed string "domove"
162 0xa2 [101][00010] -> fixed string "op"
000 0x00 [0000000][0] -> packed int / lua hack = 0
--- numbered entry 3 / 8
129 0x81 [1000][000][1] --> fixed map, size=2^0=1 +array
004 0x04 [0000010][0] -> array size = 2
002 0x02 [0000001][0] -> padding packed-int = 1
000 0x00 [00000000] -> vacancy bits
--- numbered 1 / 2
001 0x01 [0][0000001] -> +ve int = 1
--- numbered 2 / 2
194 0xc2 [11000010] -> false
--- named entry 1 / 1
168 0xa8 [101][01000] -> fixed string "dopickup"
162 0xa2 [101][00010] -> fixed string "op"
000 0x00 [0000000][0] -> packed int / lua hack = 0
--- numbered entry 4 / 8
129 0x81 [1000][000][1] -> fixed map, size=2^0=1 +array
004 0x04 [0000010][0] -> array size = 2
002 0x02 [0000001][0] -> padding packed-int = 1
000 0x00 [00000000] -> vacancy bits
--- numbered entry 1 / 2
161 0xa1 [101][00001] -> fixed string "A"
--- numbered entry 2 / 2
001 0x01 [00000001] -> +ve int = 1
--- named entry 1 / 1
178 0xb2 [101][10010] -> fixed string "get_inventory_item"
162 0xa2 [101][00010] -> fixed string "op"
000 0x00 [0000000][0] -> packed int / lua hack = 0
--- numbered entry 5 / 8
132 0x84 [1000][010][0] -> fixed map, size=2^2=4
002 0x02 [0000001][0] -> padding packed-int = 1
001 0x01 [00000001] -> vacancy bits
--- named entry 1 / 4
VACANT
--- named entry 2 / 4
174 0xae [101][01110] -> fixed string "checkfreespace"
162 0xa2 [101][00010] -> fixed string "op"
008 0x08 [0000100][0] -> packed int / lua hack = 4
--- named entry 3 / 4
194 0xc2 [11000010] -> false
164 0xa4 [101][00100] -> fixed string "next"
006 0x06 [0000011][0] -> packed int / lua hack = 3
--- named entry 4 / 4
161 0xa1 [101][00001] -> fixed string "A"
002 0x02 [0][0000010] -> +ve int = 2
000 0x00 [0000000][0] -> packed int / lua hack = 0
--- numbered entry 6 / 8
129 0x81 [1000][000][1] -> fixed map, size=2^0=1 +array
002 0x02 [0000001][0] -> array size = 1
002 0x02 [0000001][0] -> padding packed-int = 1
000 0x00 [00000000] -> vacancy bits
--- numbered entry 1 / 1
002 0x02 [0][0000010] -> +ve int = 2
--- named entry 1 / 1
166 0xa6 [101][00110] -> fixed string "domove"
162 0xa2 [101][00010] -> fixed string "op"
000 0x00 [0000000][0] -> packed int / lua hack = 0
--- numbered entry 7 / 8
129 0x81 [1000][000][1] -> fixed map, size=2^0=1 +array
004 0x04 [0000010][0] -> array size = 2
002 0x02 [0000001][0] -> padding packed-int = 1
000 0x00 [00000000] -> vacancy bits
--- numbered entry 1 / 2
002 0x02 [0][0000010] -> +ve int = 2
--- numbered entry 2 / 2
194 0xc2 [11000010] -> false
--- named entry 1 / 2
166 0xa6 [101][00110] -> fixed string "dodrop"
162 0xa2 [101][00010] -> fixed string "op"
000 0x00 [0000000][0] -> packed int / lua hack = 0
--- numbered entry 8 / 8
131 0x83 [1000][001][1] -> fixed map, size=2^1=2 +array
002 0x02 [0000001][0] -> array size = 1
002 0x02 [0000001][0] -> padding packed-int = 1
000 0x00 [00000000] -> vacancy bits
--- numbered entry 1 / 1
161 0xa1 [101][00001] -> fixed string "B"
--- named entry 1 / 2
178 0xb2 [101][10010] -> fixed string "get_inventory_item"
162 0xa2 [101][00010] -> fixed string "op"
004 0x04 [0000010][0] -> packed int / lua hack = 2
--- named entry 2 / 2
006 0x06 [0][0000110] -> +ve int = 6
164 0xa4 [101][00100] -> fixed string "next"
000 0x00 [0000000][0] -> packed int / lua hack = 0
001 0x01 [00000001] -> vacancy bits
--- named entry 1 / 4
VACANT
--- named entry 2 / 4
146 0x92 [1001][0010] -> fixed array size=2
000 0x00 [00000000] -> vacancy bits
--- numbered entry 1 / 2
164 0xa4 [101][00100] -> fixed string "Mine"
--- numbered entry 2 / 2
167 0xa7 [101][00111] -> fixed string "Storage"
166 0xa6 [101][00110] -> fixed string "pnames"
000 0x00 [0000000][0] -> packed int / lua hack = 0
--- named entry 3 / 4
146 0x92 [1001][0010] -> fixed array size=2
000 0x00 [00000000] -> vacancy bits
--- numbered entry 1 / 2
194 0xc2 [11000010] -> false
--- numbered entry 2 / 2
194 0xc2 [11000010] -> false
170 0xaa [101][01010] -> fixed string "parameters"
006 0x06 [0000011][0] -> packed int / lua hack = 3
--- named entry 4 / 4
85 0xb9 [101][11001] -> fixed string "Transport only full/empty"
164 0xa4 [101][00100] -> fixed string "name"
000 0x00 [0000000][0] -> packed int / lua hack = 0
In json, this can be represented as:
{
"1": { "op": "disconnect" },
"2": { "1": 1, "op": "domove" },
"3": { "1": 1, "2": false, "op": "dopickup" },
"4": { "1": "A", "2": 1, "op": "get_inventory_item" },
"5": { "2": "A", "op": "checkfreespace", "next": false },
"6": { "1": 2, "op": "domove" },
"7": { "1": 2, "2": false, "op": "dodrop" },
"8": { "1": "B", "op": "get_inventory_item", "next": 6 },
"pnames": ["Mine", "Storage"],
"parameters": [false, false],
"name": "Transport only full/empty"
}Note that the array indexing is 1-based, reflecting how this is represented in Lua