Example external representations

README.md

possible ideas for serialisation formats

This gist has some possible formats:

jsonl with inline items

This is the format from the first spike.

jsonl with items encoded as strings

This is the same as the other jsonl format, but with items as escaped strings instead of inline json objects.

tsv with multiple columns

This is a bit of a bodge. Normally, a tsv file has a bunch of rows, each of which represents the same kind of data. For example, you might have a school column, a diocese column, and a head-teacher column, and each row represents a school and has values for each of these fields.

In our use case, however, we have different types of rows which provide different types of data:

• add-item which adds an item
• append-entry which appends an entry with a given timestamp and item hash
• assert-root-hash which asserts the current value of the merkle tree root

We worked around this by using the generic column names command, val1 and val2.

tsv with only 2 columns

We wondered if it makes sense to record the entry as the canonical JSON for the entry which is used in the Merkle tree. This involves including the entry number in the format (which the other options don't do).

There are some interesting questions this raises:

• should we include the entry number in the serialisation format? (we had previously said "no" to this)
• should we revisit the canonical JSON for the entry?

The canonical JSON for the entry is used in the verifiable log for computing the hash of an entry. However, the entry-number is strictly redundant here; it corresponds precisely with the position within the Merkle tree.

jsonl-blob-item.jsonl

{ "assert-root-hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" }
{ "add-item": "{\"end-date\":\"1991-12-25\",\"country\":\"SU\",\"official-name\":\"Union of Soviet Socialist Republics\",\"name\":\"USSR\",\"citizen-names\":\"Soviet citizen\"}" }
{ "append-entry": {"entry-timestamp":"2016-04-05T13:23:05Z","item-hash":"sha-256:e94c4a9ab00d951dadde848ee2c9fe51628b22ff2e0a88bff4cca6e4e6086d7a"} }
{ "add-item": "{\"end-date\":\"1990-10-02\",\"country\":\"DE\",\"official-name\":\"Federal Republic of Germany\",\"name\":\"West Germany\",\"citizen-names\":\"West German\"}" }
{ "append-entry": {"entry-timestamp":"2016-04-05T13:23:05Z","item-hash":"sha-256:e03f97c2806206cdc2cc0f393d09b18a28c6f3e6218fc8c6f3aa2fdd7ef9d625"} }

jsonl-inline-item.jsonl

{ "assert-root-hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" }
{ "add-item": {"end-date":"1991-12-25","country":"SU","official-name":"Union of Soviet Socialist Republics","name":"USSR","citizen-names":"Soviet citizen"} }
{ "append-entry": {"entry-timestamp":"2016-04-05T13:23:05Z","item-hash":"sha-256:e94c4a9ab00d951dadde848ee2c9fe51628b22ff2e0a88bff4cca6e4e6086d7a"} }
{ "add-item": {"end-date":"1990-10-02","country":"DE","official-name":"Federal Republic of Germany","name":"West Germany","citizen-names":"West German"} }
{ "append-entry": {"entry-timestamp":"2016-04-05T13:23:05Z","item-hash":"sha-256:e03f97c2806206cdc2cc0f393d09b18a28c6f3e6218fc8c6f3aa2fdd7ef9d625"} }

tsv-column-bodge.tsv

command	val1	val2
assert-root-hash	e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855	
add-item	{"end-date":"1991-12-25","country":"SU","official-name":"Union of Soviet Socialist Republics","name":"USSR","citizen-names":"Soviet citizen"}	
append-entry	2016-04-05T13:23:05Z	sha-256:e94c4a9ab00d951dadde848ee2c9fe51628b22ff2e0a88bff4cca6e4e6086d7a
add-item	{"end-date":"1990-10-02","country":"DE","official-name":"Federal Republic of Germany","name":"West Germany","citizen-names":"West German"}	
append-entry	2016-04-05T13:23:05Z	sha-256:e03f97c2806206cdc2cc0f393d09b18a28c6f3e6218fc8c6f3aa2fdd7ef9d625

tsv-entries-as-json.tsv

command	val
assert-root-hash	e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
add-item	{"end-date":"1991-12-25","country":"SU","official-name":"Union of Soviet Socialist Republics","name":"USSR","citizen-names":"Soviet citizen"}
append-entry	{"entry-number":"1","entry-timestamp":"2016-04-05T13:23:05Z","item-hash":"sha-256:e94c4a9ab00d951dadde848ee2c9fe51628b22ff2e0a88bff4cca6e4e6086d7a"}
add-item	{"end-date":"1990-10-02","country":"DE","official-name":"Federal Republic of Germany","name":"West Germany","citizen-names":"West German"}
append-entry	{"entry-number":"2","entry-timestamp":"2016-04-05T13:23:05Z","item-hash":"sha-256:e03f97c2806206cdc2cc0f393d09b18a28c6f3e6218fc8c6f3aa2fdd7ef9d625"}