Blotter File Format.md

Blotter File Format - v7

The Blotter File Format, successor to the format known as ".tung files", is a system for storing the components and wires of a Logic World world. It is used for both full world saves as well as subassemblies, but there are slight differences between the two.

File types

• World files:
  • Represent a full, playable world space
  • May have zero or more root components at different positions/rotations in the world
  • Use .logicworld file extension
  • All circuit states are stored contiguously, from circuit state 0 through the highest state.
• Subassembly files:
  • Represent a substructure within a playable world space; can be loaded and added to a world at arbitrary position/rotation
  • Must have at least one root component
  • Use .lwsubassembly file extension
  • Has a list of circuit state indexes that are On. All indexes not listed here are inferred as being Off.

File structure

• Header (16 bytes)
• Save info
  • Save Format Version (1 byte)
  • Game version (16 bytes)
  • Save type (1 byte)
  • Number of components and wires (8 bytes)
  • Mod versions (4+ bytes, dependent on the number of mods in the save)
  • Component IDs Map (4+ bytes, dependent on the number of loaded component IDs)
• Object data
  • Components data (0+ bytes, dependent on the number of components)
  • Wires data (0+ bytes, dependent on the number of wires)
  • Circuit states (4+ bytes, variable, has a different format for worlds and subassemblies)
• Footer (16 bytes)

The "Save Info" data is at the beginning of the file so that it is easy to extract as metadata. This is done by the game itself in a few places, as well as logicworld.net when uploading a save there.

Header and footer

Header

The first 16 bytes of every Blotter file are 4C-6F-67-69-63-20-57-6F-72-6C-64-20-73-61-76-65, which is UTF-8 for “Logic World save”.

Footer

The final 16 bytes of every Blotter file are 72-65-64-73-74-6F-6E-65-20-73-75-78-20-6C-6F-6C , which is UTF-8 for “redstone sux lol”.

The header and footer exist to protect against save corruption. If the file does not have them, we know that something is wrong with the file.

Save Format Version

One byte specifies the save format version. Currently this is 07.

Game version

16 bytes store the game version that this save was created in.

Save type

There are two types of file that use the Blotter format, worlds and subassemblies. One byte specifies which type of file this is.

Byte	Meaning
00	Unknown/error
01	World
02	Subassembly
03 - FF	Reserved for future use

Component and wire counts

Here there are two ints. The first specifies the number of components in the save, the second specifies the number of wires.

Mod versions

Mods can opt-in to storing their versions in save files created while they are installed. This is so that if a later mod version has changes, older save files can be converted.

• An int for the number of mod versions stored in this file
• For each mod version:
  • A string for the text ID of the mod
  • A version for the version of the mod that this save was created with

Component IDs Map

When humans add component types to Logic World, they use text IDs like MHG.CircuitBoard. To make save files and network packets small in size, the game internally uses numeric IDs to reference component types. The relationship between text IDs and numeric IDs is not deterministic or consistent between different save files, so the map from text IDs to numeric IDs must be stored with the save.

First we write an int for the number of component IDs in this file.

One by one the mappings are written to the file:

• A 2-byte unsigned integer for the numeric ID of the component
• A string for the text ID of the component

Components data

One by one, every component is listed in the file. The first component must be a root component -- i.e. have an address of C-0 for its parent -- and every subsequent component must have a parent that was listed before it.

It is explicitly okay to have a world file with no components in it (i.e. an empty world). However, a subassembly file must have at least one component.

Each component is written like so:

• The component address of this component

• The component address of the component's parent

• 2 bytes for the numeric ID of the component's type (see Component IDs Map)

• Three ints for the local fixed position of the component; x y z

  • One unit in the fixed position corresponds to a distance 0.001 game units, or 1mm

• Four floats for the local rotation of the component as a quaternion; x y z w

• Information about the component's inputs:

  • An int for the number of inputs the component has
  • For each input:
    • An int for the input's circuit state ID

• Information about the component's outputs:

  • An int for the number of outputs the component has
  • For each output:
    • An int for the output's circuit state ID

• The component's custom data:

  • An int for the number of bytes in the custom data
    • If the component has no custom data, -1 may be written instead of 0
  • All the bytes of the custom data, in order

Wires data

One by one, every wire is listed in the file.

• The peg address of the wire's first point
• The peg address of the wire's second point
• An int for the wire's circuit state ID
• A float for the wire's rotation, relative to the default rotation it would have

We do not need to save wire addresses; nothing in the save file needs to reference them, and there is no need for them to be consistent between save loads. Instead, wire addresses are dynamically assigned when the save is loaded.

Circuit states

We've saved the circuit state IDs of all the pegs and wires in the save. Now we need to save the values of those state IDs, so states can be looked up.

A state is either on or off. At runtime, the game stores all circuit states from zero through the highest state in a contiguous block of memory.

World files

Worlds are large, and generally use most circuit states, starting from state 0. All circuit states are listed.

• First an int for the number of bytes in this sequence
• Then, that number of bytes. Each bit represents the value of the next circuit state; each byte contains 8 states.
  • If, when saving the game, the number of circuit states is not divisible by 8, the last byte will be padded.

Subassembly files

Subassemblies represent a subset of a world. Most of the circuit states in a world will be irrelevant to a particular subassembly, so it would be silly and wasteful to save all of them with every subassembly.

Instead, we save a list of circuit state IDs which are both used in this subassembly and have a state of on. All unlisted state IDs are assumed to be in the off state.

• First an int for the number of ints in this sequence
• Then, that number of ints. Each int signifies a state ID with a value of on.

Sub-standards

Int (4 bytes)
32 bit signed integers, little endian.

Float (4 bytes)
32 bit floating point values, little endian.

Boolean (1 byte)
An entire byte that is either 00 for false or 01 for true.

String (4+ bytes)
An int for the number of bytes in the string, followed by that number of bytes, representing that string in UTF-8.

Version (16 bytes)
Versions contain four numbers (i.e. 1.0.3.1069). These four numbers are written to the file in order, each as a 4-byte int. Thus, 4x4 bytes = 16 bytes for this.

Component Address (4 bytes)
Component Addresses are written as 32 bit unsigned integers, little endian.

Peg Address (9 bytes)

• One byte to indicate the peg type:

  Byte	Meaning
  00	Undefined
  01	Input peg
  02	Output peg
  03 - FF	Reserved for future use

  Any value other than 01 or 02 is invalid.

• A component address for the component of the peg

• An int for the zero-based index of the peg (i.e. is it the 0th input, the 1st input, ect )

A note on seemingly-bizarre design choices

This file structure is not the most efficient, most beautiful, or most logical way to lay out the data. The primary purpose of the design of the Blotter format is that it is easy to integrate them with the game code. Therefore, the format uses the exact same data structures used by the game. This makes it fast and simple to convert the game state to a save file and vice versa, but since these data structures are optimized for runtime, the files do look a little weird.

• "Why are circuit states stored so weirdly? Why not directly save the state of an input/output, instead of a reference to a state index?"
  • This is how the game stores circuit states at runtime, for performance and for ease of synchronization between the client and server. Storing it this way in the file makes for easy interoperability with the runtime game code, and allows us to use the same component/wire data structures in the saves and in the game.
• "Must all the circuit states be stored? Surely you only need the states of the inputs or outputs, and everything else (including the wires) can be inferred from those."
  • This is true for the server, but the client has no ability to run a simulation and infer other states. However, the client needs to know all the states so that it can render a subassembly correctly before it is placed and added to the server's world/simulation. Storing all the states is also just simpler, and requires fewer steps when loading a save.
• "Why in god's name are all the integers signed when come on they really should be unsigned?"
  • Logic World is written in C#. The .NET runtime has a few weird quirks, and one of those quirks is that most integers are signed, even things that can never be below zero like array lengths and indexes. Since this file format is designed for maximum interoperability with the code, this quirk is copied, and we use signed ints a lot.

Please keep in mind that this is still subject to change before release. It might not, but also it might.

Changes since Draft 0:

• A list of mods, with versions, which affect the save is now present
• Peg indexes now use ints instead of bytes, as the game now supports >255 inputs/outputs per component
• Input exclusivity is no longer stored, as exclusive inputs have been removed from the game
• If a component does not have custom data, -1 can be written for the byte count instead of 0. This indicates null rather than an empty array.
• Improved some formatting

Quick question: would you mind if projects implementing BlotterV5 include a copy of this readme and is there a license or copyright notice that should also be included?

I hereby release this document & the file format itself into the public domain.

            DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE 
   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 

  0. You just DO WHAT THE FUCK YOU WANT TO.

Knock yourself out :)

quality licence.

I am very fond of WTFPL, it perfectly captures my attitudes towards open source and towards software in general.

Joho, I just wrote my own parser for this file format for the first time.

And I stumbled across some issue, which makes parsing be much more bloated for me.

We have the Circuit-State section in this file, which can live by its own without issues. It does not depend on anything.
But wires and components do depend on it. Sadly they appear before Circuit-States in this format.

In my project (OpenTUNG), circuit states have a different form. I choose an Object Oriented solution here.
That means, that there are no CircuitStateIDs at all. I only care for the Output-Pegs states (on/off) and generate the rest of the network with only these. That would normally be a quick lookup into the circuit-states + the simulation post processing/validation.
But these have not been loaded yet. The file also does not provide me any means to skip to there quickly.
So I have to either skip all the component once, by reading them. And then processing/loading them again later.
Or create a temporary data structure for each output-peg to update its powered state later on.

Either way, this is overhead, which I would not have, if Circuit-States are listed before Components.

So maybe consider this when BlotterV6 gets released, it would make the parsing for some third-party parsers much more easy.

---

Another question:
Do components need more pegs than 65K? I think even for modding, that might be a bit over the top.
So if there are no plans to compress the file by default, one could shrink the peg count down to 2 bytes. That is 4 less bytes per wire and component. But for small worlds we are only talking about KB optimizations. So its just an idea.

Changes in Blotter v6 (used since Logic World v0.91):

• Instead of saving component local positions as three floats, we now save component local fixed positions as three ints
• Subassembly files may now have multiple root components

Changes in Blotter v7 (used since Logic World v0.91.2):

• The first byte of a serialized Peg Address is no longer a boolean with true indicating an input peg and false indicating an output peg. Instead, a value of 01 indicates an input peg, 02 indicates an output peg, and the other values for that byte are undefined/reserved.