Corestore API proposal

corestore.md

Corestore API Proposal

Requirements

• Storing multiple hypercores in a single "group"
• Purging storage for a given hypercore key
• Able to replicate whole group
• Specifying which key should be used for replication handshake

API (from the SDK)

const someKey = `cabal://kjlkajsdlaskjdljsadk`.slice('cabal://'.length)

// Optionally pass in a default key, if you don't specify one, it'll be generated
const store = new Corestore(somekey, {
  // Support extension messages
  extensions: ['foobar', 'baz']
})

// Get the default hypercore from the store
const defaultCore = store.default()

// Key gets generated
const writableCore = store.get()

// Use an existing key to initialize Hypercore
const readableCore = store.get('some key here')

store.remove('some key' || someCore)

store.delete('some key' || someCore)

store.on('extension', (type, message, peer) => {
  if(type === 'foo' || type === 'bar') {
    // Load a core that we leared about from the peer
    const otherCore = store.get(message)
  }
  
  if(type === 'foo') {  
    // Tell the peer about our core
    peer.extension('bar', writableCore.key)
  }
})

// Once we get a new peer, tell them about our core so they can load it
store.on('peer-add', (peer) => {
  peer.extension('foo', writableCore.key)
})

---

Replication strategy

Managing hypercore replication in a single swarm instance is kinda hard. Here's some of the things a replication manager should be able to handle:

• Start replication feeds for new connections
• Support extensions for connections (set of all possible extension types)
• When the peer sends a feed event, it should replicate with a hypercore (corestore?)
• When a peer is discovered for a given key, and there's already a connection to that IP+Port combo, it should be possible to prevent additional connections and replicate the feed over that connection directly.

API

const replicator = new Replicator({
  extensions: []
})

// These will likely be invoked from hypercore or whatnot
replicator.handleConnection(connection, peer, discoveryKey) // A new incoming connection, or outgoing connection. Peer should have IP and port
replicator.handlePeer(peer, discoveryKey, connect) // Invoke when a new peer is found for a given discoveryKey, add option to connect to them. Peer should have an IP and port. This will be used to deduplicate connections

// Events to handle so you can pass them on to discovery layer
replicator.on('listen', (discoveryKey) => etc)
replicator.on('unlisten', (discoveryKey) => etc)

// Add more extension types for the replicator to handle
replicator.addExtensions(['foo', 'bar'])
const extensions = replicator.getExtensions()

// Extensions should be listened on in your hypercores rather than on the replicator since extension messages are emitted for specific feeds

// Add a core to replicate when there's a `feed` event for it's discovery key
replicator.addCore(core)

// Do something fancy when a perticular discovery key pops up
replicator.on(discoveryKey, (stream) => etc)