First draft about IPLD Layers

ipld-layers.md

IPLD Layers

Different applications built on top of IPLD need different abstractions. IPLD originates as tightly integrated piece of IPFS. When it was decoupled, two layers of abstractions were built. On the lower level there's IPLD Formats and on the higher level there's IPLD. The idea is that you can implement your own format easily and then get the traversal across formats for free.

Those two layers are not enough anymore. There are more concerns that should be separated. This document is describing those layers abstractly without going too much into implementation details. Some layers might be merged when implemented.

Glossary

• Binary data: Data encoded in raw bytes

General overview

                  ┌───────────────┐
                  │   IPLD Main   │ e.g. `ipld.resolve()`
 Multiple Blocks  ├───────────────┤
                  │ IPLD Exchange │ e.g. `ipld.put()`
───────────────── ├───────────────┤
                  │  IPLD Blocks  │ e.g. `ipld.toCID()`
  Single Block    ├───────────────┤
                  │  IPLD Formats │ e.g. `ipld.encode()`
                  └───────────────┘

Detailed explanations

IPLD Formats

Creating new encoders/decoders for IPLD should be as easy as possible. Functionality that would be duplicated across formats, should be done more generically on a higher level.

The major primitives of this layer are the IPLD Data Model, IPLD Path and binary data.

The binary data can be accessed through IPLD Paths and return IPLD Data Model primitives.

Encoding data conforming the IPLD Data Model into binary data may also be supported.

The codec that the IPLD Format is specified, which can then be used by other layers to generate CIDs.

IPLD Blocks

This layer adds the concept of binary data together with CIDs. That's useful if you want exchange data, as the binary data is meaningless and can't be interpreted without knowing its codec.

You can access the data within the IPLD Blocks independent of the IPLD Format it is stored in.

All operations are still within a single IPLD Block boundary, you can't access anything outside the Block.

IPLD Exchange

This layer is about exchanging data. This may be about storing/loading data locally from disk, or transferring it over the network.

IPLD Main

This layer allows operations across several blocks. You can e.g. request a value with a path that spans several blocks. The traversal is IPLD Format agnostic, so you can freely move between different Blocks.

Another implementation on this layer could be IPLD Selectors.

Is there a reason we have to have the CID creation in the format interface?

The only reason I can think that it has to be here is because some format’s require specific hashing algorithms if we want to remain compatible, but could we find a way to surface that requirement along with the codec identifier so that a higher level library can do that universally across all the codec implementations?

Also, don’t we need decode?

Is there a reason we have to have the CID creation in the format interface?

It's not in the IPLD Format layer. The IPLD Format layer provides the coded so other layers can use this information to create a CID.

Also, don’t we need decode?

Sure. It is implicitly in "The binary data can be accessed through IPLD Paths and return IPLD Data Model primitives.". If you pass in the root path, it is a decode. But perhaps I should make this more explicit.

Other than those things, does it sound reasonable?

Yup, looks good