UPDATE: The new home of this is https://github.com/20kdc/byond-data-docs/
This is posted now so that should I fail to complete the rest, this research will still be available. THIS IS A DRAFT. IT IS INCOMPLETE. But it is more than others have posted.
Sorry, I don't actually have any documentation on this yet.
However, MCHSL & SpaceManiac's project, extools, is known to contain a disassembler for this.
Instance initializer procs are theoretically not special.
However, they have relatively special syntax, being part of DMM files.
As such, they seem to only contain the following opcodes (as named by extools dmdism):
PUSHI (80, (unsigned int))
PUSHVAL (96, (type), (val), (only present if type = 42: val ex.))
SETVAR src.? (52, 65500, 65486, (StringID))
END (0)
These are always in the form PUSHVAL or PUSHI followed by SETVAR, with the list terminated with END.
(I haven't yet seen any others, but that doesn't necessarily mean they don't exist.)
An "ObjectID" is a generic object ID which there isn't details for yet.
These all use a format dependent on a flag in the file header.
A "ClassID" is a reference into the Class Table.
A "MobTypeID" is a reference into the Mob Types Table.
A "StringID" is a reference into the String Table.
A "ListID" is a reference into the List Table.
A "ProcID" is a reference into the Proc Table.
A "InstanceID" is a reference into the Instance Table.
A "CacheFileID" is a reference into the Cache Files Table.
A lot of these IDs allow 0xFFFF to mean "none". (Even if LARGE OBJECT IDs is set, when it would logically be 0xFFFFFFFF - if making a DMB, have fun with that)
These will be referred to as "nullable" IDs.
There are sometimes references to IDs which aren't actually in ObjectID form.
These are in the form "ClassID-as-Uint32" (i.e. a ClassID in the form of a Uint32)
There are 3 version numbers for a DMB file.
There is the GEN Version, the LHS Version, and the RHS Version.
All of these are important to loading a DMB file for reasons I do not understand.
Why it is not a single version number I do not understand.
The DMB file format, for the grand total of 3 plain ASCII lines it has, uses Unix newlines.
The DMB file format starts with an optional "shebang line" ("#!") for Unix executable interpreter business.
A value which we'll call the "base pointer/key" (which is presumably used so that file-position-based encryption doesn't royally fail for shebang interpreter changes) is set to the file position at this point.
This is a 32-bit value that gets downcast as necessary.
It follows (or starts) with a line of the form world bin v512\n (at time of writing).
This version is the GEN Version.
If it's below 222, information is not available on these versions at this time.
Known versions always follow with one of two line types:
A compatibility line of the form min compatibility v512 468\n (at time of writing).
A compatibility line of the form min compatibility v222\n (not checked).
In the first case, the LHS version and RHS version are available.
In the second case, the number is both the LHS version and the RHS version.
Integer types in BYOND data are little-endian, to align with the x86 processor.
A UInt32 follows. This is a set of game flags.
Bit 31 (highest): Another UInt32 follows. (Unsure on meaning.) Bit 30: LARGE OBJECT IDs
It's important to note that ObjectID is Uint16 normally and Uint32 if LARGE OBJECT IDs is set.
For analysis reasons, it's important to note that LARGE OBJECT IDs did not exist until at least after v368.
Next, three UInt16s follow: Width, Height, Z-Levels.
The arrangement of the grid is as you would expect:
However, unlike what you would expect, Y coordinates in BYOND go upwards and coordinates start at 1.
The part where coordinates start at 1 is not preserved in the file, but Y coordinates going upwards is.
This data is run-length-compressed, and made up of the following 'groups':
As far as I can tell, the DMM order is: additionalIDs, turfID, areaID.
If you get weird "protomem"-related errors, it's this field.
This is the set of classes.
This starts with an ObjectID, which is the amount of entries.
Each class has:
For GEN Versions >= 307, additionally, another Uint8. If it's 0x0F (not 0xFF!), then a Uint32 follows to replace it. Unknown. If the format version is insufficient, this defaults to 1.
For RHS Versions >= 494, additionally {
}
For RHS Versions >= 508, additionally {
}
For GEN Versions >= 267, additionally {
}
For RHS Versions >= 500, additionally {
}
For RHS Versions >= 509, additionally {
}
For GEN Versions >= 306, additionally {
}
The Defining Var List is a set of key/value pairs implicitly terminated.
The Defining Var List uses a VarID key (the actual var).
The values are flags.
0x0001 means "global".
0x0002 means "const". This implies "global" and is always seen with "global".
0x0004 means "tmp".
The Overriding Var List is a set of key/value pairs implicitly terminated.
The Overriding Var List uses a StringID key (the name of the var to be overridden).
This is followed with a type value (see The Var Table for further information on the general concepts).
Details on what happens for large-object-IDs ARE NOT CERTAIN.
But specifically for 16-bit object IDs and float values, the value is written in two pieces, 'semi-big-endian'.
So 225, 42, 16896, 0 means [string 225] = float 32.0.
It's important to note that some classes are actually BYOND built-in types.
Additionally, some class code is part of BYOND and is always generated for every file.
Like Sub-Block 1, this starts with an ObjectID entry count.
Each entry attaches mob metadata to a /mob-derived class.
For each entry:
key = "PlayerName")If hasFancyStuff is >= 0x80:
The string table contains all DMStrings.
As is the pattern so far, there is some amount of entries represented as an ObjectID.
An ongoing 32-bit hash (NQCRC)[../algorithms/NQCRC.md] is maintained throughout the string table, initialized to -1.
For each entry {
The entries use encrypted lengths, and encrypted data - this is where the "base pointer/key" is involved.
Lengths are Uint16 XOR'd with the file position of the value minus the "base pointer/key".
The entry length is prefixed over and over with (encrypted) 0xFFFF to reach the required total.
The 0xFFFF prefix adds 0xFFFF to the length total, and then reads another length (which can itself be 0xFFFF).
Otherwise the final length value is added to the total and length reading stops.
With official software, if the string length happens to be exactly equal to 0xFFFF, it doesn't write 0xFFFF 0x0000, it just writes 0xFFFF.
#define F "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
#define G F+F
#define H G+G
#define I H+H
#define J I+I
#define K J+J
#define L K+K
#define M L+L
#define N M+M
#define O N+N
#define P O+O
/mob
text = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"+F+G+H+I+J+K+L+M+N+O+P
Finally, the string is written, encrypted with (XORJUMP9)[../algorithms/XORJUMP9.md].
The key is again the file position (at string start) minus the "base pointer/key".
Strings, including empty ones, are hashed - including the terminating null byte, which is not actually present.
}
Importantly, after the entries, there is a footer.
For GEN Versions >= 468 {
}
DMB has a number of very complex string escaping rules.
Like Sub-Block 1, this starts with an ObjectID entry count.
This doesn't really have a set content format; it's a generic list container for other bits to use for dynamically sized lists.
It's guaranteed to contain only lists of ObjectIDs, but that's as far as guarantees go.
For each entry:
Like Sub-Block 1, this starts with an ObjectID entry count.
For each entry:
For GEN Versions >= 224 OR if large object IDs are on {
}
Like Sub-Block 1, this starts with an ObjectID entry count.
For each entry {
}
Type is a BYOND type number (this type numbering is shared with the protocol). Of these, at least the following are valid:
If the GEN Version and the LHS Version are both >= 512, this has a footer {
}
Like Sub-Block 1, this starts with an ObjectID entry count.
Each entry is a ProcID.
Like Sub-Block 1, this starts with an ObjectID entry count.
For each entry:
A turf has the values (10, (some class ID), 0xFFFF).
It is important to note that as far as I am aware, most properties of an initializer proc don't matter.
Check the DMB.Bytecode.md file for more information.
This attaches instances to the map.
Uint32, total objects.
For each object:
That is the offset in tiles (based on the same universal map index logic as for the map itself) from the last object. If there is no last object, the start of the map.
It is reasonable to assume that null instances are used to pad out long empty regions of space. (i.e. they don't mean anything)
If GEN Version < 368 (YES, UNDER) {
}
If GEN Version >= 308 {
}
If GEN Version >= 415 {
}
If GEN Version >= 230 {
}
If GEN Version >= 507 {
}
If GEN Version < 507 (yes, under) {
}
If GEN Version >= 232 {
}
If GEN Version >= 235 && GEN Version < 368 {
}
If GEN Version >= 236 && GEN Version < 368 {
}
If GEN Version >= 341 {
}
If GEN Version >= 266 {
}
If GEN Version >= 272 {
}
If GEN Version >= 276 {
user.game hub name)}
If GEN Version >= 305 {
}
If GEN Version >= 360 {
}
If LHS Version >= 455 {
}
Like Sub-Block 1, this starts with an ObjectID entry count.
For each entry:
These correspond to entries in the relevant RSC files.
(The order is swapped. This is known and correct.)
NQCRC is an MSB-first CRC-32 with 0xAF as the polynomial.
The way NQCRC is written implies that the initial value of the hash and what happens to "finalize it" is situation-dependent.
The following documentation exists because just saying all this is not how you write proper format documentation.
NQCRC, as with most CRC-32 implementations, uses a table of 256 32-bit values, one for each possible byte.
The per-byte NQCRC algorithm is uint32_t hash = (hash << 8) ^ nqcrcTable[(hash >> 24) ^ byt];.
This exactly matches the description of an msbit-first CRC-32 in: https://en.wikipedia.org/wiki/Computation_of_cyclic_redundancy_checks#Multi-bit_computation
Following with the theme, the table computation is the same.
private static final int[] nqcrcTable = new int[256];
static {
for (int i = 0; i < nqcrcTable.length; i++) {
int value = i << 0x18;
for (int j = 0; j < 8; j++)
value = (value << 1) ^ (value < 0 ? 0xAF : 0);
nqcrcTable[i] = value;
}
}The result is the NQCRC value for that index.
An RSC file is actually a concatenated list of RSC entries.
It has no encryption whatsoever, just verification.
Values are little-endian as with DMB.
The outer entry structure is:
The outer structure allows skipping through entries easily.
The inner structure identifies entry components and is the actual data.
Regarding the mapping to the DMB file: The order is swapped. This is known and correct.
XOR Jump 9 is an obfuscation algorithm.
It isn't really worth calling it encryption.
BYOND is not the first or the last piece of software to feature this sort of thing. I had a rant here about this pattern, but removed it before publishing. Just know that it's stupid and it annoys me.
XOR Jump 9 is a symmetric algorithm.
It takes some input plaintext or ciphertext and an input key byte.
It is only different a constant XOR because it adds 9 to the key byte after each byte has been processed.
You might notice that this doesn't sound much better than a constant XOR.
local function xorjump9(data, key)
local result = ""
for i = 1, #data do
result = result .. string.char(data:byte(i) ~ key)
key = (key + 9) % 256
end
return result
end