dat-core-tech-roadmap.md

Dat Core Technical Roadmap

A small roadmap for the core tech in dat. this mostly relates to the core dependencies such as hypercore/hyperdrive and not the cli or dat.land

Implementation

1. Finish random access APIs

Adding var cursor = drive.createByteCursor(file). A cursor api that quickly allows you to seek anywhere in the file content.

Make drive.createFileReadStream(file) support start/end byte offsets (should use a cursor internally)

@andrewosh has been very helpful with getting this finished.

2. Improve sparse replication

Adding a replication prioritization api to hypercore

The API is projected to look something like this

feed.prioritize({start: 0, end: 1000}, function (err) {
  console.log('called when range is downloaded')
})

feed.unprioritize({start: 0, end: 1000}) // cancels the above prio

Options will include

{
  start: startBlock,
  end: endBlock,
  priority: number // at which prio should this be fetched, higher number -> higher prio
  linear: bool // prio linear downloading of the range
  bisect: bool // prio bisecting download of the range
  hotswap: bool // allow faster peers to override ongoing requests to slower peers (will reduce latency but increase used bandwith)
}

Per default the entire feed will be synced with prio 0. This can be disabled in the feed constructor

Implicitly expose the above api to hyperdrive

hyperdrive will replicate the metadata (i'm gonna start calling this the changes feed) with higher priority than the content.

File read and write streams will have a start, end option that prioritizes that range.

var stream = drive.createFileReadStream({ // prioritizes range
  start: ...,
  end: ...
})

drive.download(file, function (err) { // prioritizes that file
 
})

3. Hyperdrive filename indexing

We'll start maintaining a leveldb index for filename -> change number to support quick lookups

4. Expose history apis

drive.createChangesStream() will create a stream of changes to the drive.

otherDrive = drive.checkout(change) will allow you checkout an older version of it.

5. Diffing / deduplication

The theory is all done for how to diff multiple drives with each other. This needs to get implemented in hypercore.

Other

All of the theory behind hyperdrive / hypercore is done but we need to update the specification with the remaining missing parts.

https://github.com/datproject/docs/blob/master/hyperdrive.md

Demos

• Wikipedia in hyperdrive (most of the hard work was already done in peerwiki, just need to migrate it)
• Dat-chat, a p2p multi writer chat system on top of hypercore. This will lay the foundation for dat-dropbox
• Dat-dropbox, a multi writer dat.