A small roadmap for the core tech in dat. this mostly relates to the core dependencies such as hypercore/hyperdrive and not the cli or dat.land
Adding var cursor = drive.createByteCursor(file)
. A cursor api that quickly allows you to seek anywhere in the file content.
Make drive.createFileReadStream(file)
support start/end byte offsets (should use a cursor internally)
@andrewosh has been very helpful with getting this finished.
The API is projected to look something like this
feed.prioritize({start: 0, end: 1000}, function (err) {
console.log('called when range is downloaded')
})
feed.unprioritize({start: 0, end: 1000}) // cancels the above prio
Options will include
{
start: startBlock,
end: endBlock,
priority: number // at which prio should this be fetched, higher number -> higher prio
linear: bool // prio linear downloading of the range
bisect: bool // prio bisecting download of the range
hotswap: bool // allow faster peers to override ongoing requests to slower peers (will reduce latency but increase used bandwith)
}
Per default the entire feed will be synced with prio 0. This can be disabled in the feed constructor
hyperdrive will replicate the metadata (i'm gonna start calling this the changes feed) with higher priority than the content.
File read and write streams will have a start, end option that prioritizes that range.
var stream = drive.createFileReadStream({ // prioritizes range
start: ...,
end: ...
})
drive.download(file, function (err) { // prioritizes that file
})
We'll start maintaining a leveldb index for filename -> change number to support quick lookups
drive.createChangesStream()
will create a stream of changes to the drive.
otherDrive = drive.checkout(change)
will allow you checkout an older version of it.
The theory is all done for how to diff multiple drives with each other. This needs to get implemented in hypercore.
All of the theory behind hyperdrive / hypercore is done but we need to update the specification with the remaining missing parts.
https://github.com/datproject/docs/blob/master/hyperdrive.md
- Wikipedia in hyperdrive (most of the hard work was already done in peerwiki, just need to migrate it)
- Dat-chat, a p2p multi writer chat system on top of hypercore. This will lay the foundation for dat-dropbox
- Dat-dropbox, a multi writer dat.