raw ipfs + blake2b vs. sodium-universal

gistfile1.txt

$ echo 'record a' > record-a.bin

$ hexdump -C record-a.bin
00000000  72 65 63 6f 72 64 20 61                           |record a|
00000008

$ ipfs dag put -f raw --input-enc raw --hash blake2b-256 record-a.binzCT5htkeC8GjDQn53ve8HDuhFQS8q4UVh7NX9KhNHSjnAczWEUbS

$ cid format zCT5htkeC8GjDQn53ve8HDuhFQS8q4UVh7NX9KhNHSjnAczWEUbS -b base16 -f '%c%L %h %D'
raw 32 blake2b-256 d4309131176a941e14881dc27f85a10b17a1f35970a87a2e28c85179f2558851

$ node
> var buf = Buffer.from('record a')
undefined
> buf
<Buffer 72 65 63 6f 72 64 20 61>
> var sodium = require('sodium-universal')
undefined
> var digest = Buffer.alloc(32)
undefined
> sodium.crypto_generichash_batch(digest, [buf])
undefined
> digest
<Buffer d4 30 91 31 17 6a 94 1e 14 88 1d c2 7f 85 a1 0b 17 a1 f3 59 70 a8 7a 2e 28 c8 51 79 f2 55 88 51>
> var multihashing = require('multihashing')
undefined
> multihashing.digest(buf, 'blake2b-256')
<Buffer d4 30 91 31 17 6a 94 1e 14 88 1d c2 7f 85 a1 0b 17 a1 f3 59 70 a8 7a 2e 28 c8 51 79 f2 55 88 51>
> var CID = require('CID')
Error: Cannot find module 'CID'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)
    at Function.Module._load (internal/modules/cjs/loader.js:507:25)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:20:18)
> var CID = require('cids')
undefined
> new CID('zCT5htkeC8GjDQn53ve8HDuhFQS8q4UVh7NX9KhNHSjnAczWEUbS')
CID {
  codec: 'raw',
  version: 1,
  multihash:
   <Buffer a0 e4 02 20 d4 30 91 31 17 6a 94 1e 14 88 1d c2 7f 85 a1 0b 17 a1 f3 59 70 a8 7a 2e 28 c8 51 79 f2 55 88 51> }
   

Note: js-multihash doesn't work...

> var multihash = require('multihashes')
undefined
> multihash.encode(buf, 'blake2b-256')
<Buffer a0 e4 02 08 72 65 63 6f 72 64 20 61>

For Dat/hypercore, the hashes stored in the tree entries for the leaf note cover more than just the raw data... from the spec:

To protect against a "second preimage attack", hash functions have constants prepended
to their inputs based on the type of data being hashed. These constants are:

0x00 - Leaf
0x01 - Parent
0x02 - Root

Hashes will frequently include the sizes and indexes of their content in order
to describe the structure of the tree, and not simply the data within the tree.

https://github.com/datprotocol/DEPs/blob/master/proposals/0002-hypercore.md

You can see the code that generates the hash here:

https://github.com/mafintosh/hypercore-crypto/blob/ffc8f339bac0060e1eadd273ba8bcbfab33d27c8/index.js#L36

So it is hashed on "position code" + "length" + "data", which is a bit unfortunate from our perspective, as the blake2b hashes in the hypercore tree structure can't be used for content addressing.

There are some proposals floating around to add the ability to add content hashes, eg.:

• holepunchto/hyperdrive#203
• dat-ecosystem-archive/datproject-discussions#77 (comment)

I think I had a short chat with Brad about this in SF, now that I recall, and I'm actually going to meet Martin in Japan next month!