ipfs-pubsub-object-blocking.md

Proposed low-perf IPFS/IPNS pubsub

A rough proposal to implement the widely desired pub/sub feature on ipfs without requiring additional capabilities or message-passing behavior from the network or the clients. It could be implemented as an IPFS application right now.

I've just started diving into IPFS so this could be a super naïve approach.

The proposal's core concepts:

• subscription is modeled as querying a non-existent block whose future multihash is known, but has not been added to the DAG. Once this block exists, its payload will use IPNS to provide a forward linked list to one or more future blocks in the publication stream.
• publication is modeled as publishing the block with that desired multihash, and providing a payload that links forward to one or more future blocks in the publication stream.

The proposal's key properties:

• It requires the IPNS router so it's not the fastest ride in town.
• It requires that clients patiently wait for wantlist blocks that can't currently found on the network
  • I believe this is the current behavior of go-ipfs
• It is loosely inspired by mutable value chains
• It requires no additional capabilities or message-passing behavior from any actor in the network.

The proposal's minimum necessary flow (or just jump down to the example):

• PUBLISHER creates a PUBLICATION EVENT json object
  • The link contains a reference to an /ipns/ address under PUBLISHER's control.
  • The reference will contain the future payload for the PUBLICATION EVENT.
    • henceforth <PublicationEvent.PayloadLocation>
  • The link contains a sequence ID unique in this stream (e.g. "1")
    • In reality we will want to hash this with the private key to avoid spoofing
• PUBLISHER calculates the PUBLICATION EVENT multihash, but does not add it to the DAG
  • Uses ipfs add --only-hash
  • Henceforth <PublicationEvent.MultiHash>
• PUBLISHER distributes <PublicationEvent.MultiHash> widely
• SUBSCRIBER adds <PublicationEvent.MultiHash> to his wantlist.
• SUBSCRIBER waits for the object to exist in the DAG.
• PUBLISHER writes content and adds it to the DAG.
  • It has ID <Content.MultiHash>
  • Nobody knows about it yet but PUBLISHER.
• PUBLISHER creates a tree/folder at <PublicationEvent.PayloadLocation> under his IPNS root.
  • Henceforth PayloadTree
• PUBLISHER writes <Content.Multihash> into PayloadTree/data
• PUBLISHER calculates the hash for a NEW PUBLICATION EVENT using the same process as before
  • This link now has sequence ID 2
  • It should probably also contain an ipfs (not ipns) backlink to PayloadTree
  • Henceforth <NewPublicationEvent.MultiHash>
• PUBLISHER writes <NewPublicationEvent.MultiHash into PayloadTree/next
• PUBLISHER publishes the new version of his IPNS root.
  • This new version contains the payload we just prepared
  • The first publication event's payload is now ready to consume, but nobody knows how to find it.
• PUBLISHER adds and pins the first PUBLICATION EVENT json object to the DAG.
• The publication event (which is really more of a notification) starts percolating through the network as it satisfies subscribers wantlists.
• SUBSCRIBER, who has patiently waited <PublicationEvent.MultiHash> in his wantlist for ages, receives and re-pins the publication event
  • Repinning helps other subscribers.
• SUBSCRIBER dereferences <PublicationEvent.PayloadLocation>, finding both the content and the next publication event within PayloadTree/data and PayloadTree/next respectively.
• SUBSCRIBER acquires the content using ipfs get
• SUBSCRIBER adds <NewPublicationEvent.Multihash> to his wantlist, and waits.

Example of basic necessary flow:

Step 1: Alice distributes a publication event for her blog. Bob subscribes.

Publisher (Alice)
"Think I'll start a blog. Let me give the world a publication event for when the first post is done."

// Alice creates a publication event: this JSON content.
// She saves it to ~/first-publication.json
{
  "payloadLocation":"/ipns/<Alice.PeerId>/ipfs-pubsub/blog/1",
  "publishedBy":["<Alice.PeerId>"],
  "sequenceId":1
}
 // sequenceId numeric for simplicity. Should probably be Sign(1, PrivateKey) or some other scheme

# Alice discovers the hash of this publication event.
# She does not add it to the DAG.
alice~> ipfs add --only-hash ~/first-publication.json
added <FirstPublication.MultiHash> ~/first-publication.json # Not true! We didn't add it, we just discovered the hash.

Publisher (Alice)	Subscriber (Bob)
"Hey subscribe to my blog at <FirstPublication.MultiHash>!"
	"Sure, Alice!"
	bob~> ipfs object get <FirstPublication.MultiHash>
	Prepares to wait the long haul. Distributes this object on his wantlist, as does every other future subscriber to Alice.

Step 2: Alice writes her first blog content, publishes it.

Publisher (Alice)
"Time for the first post!"

# Alice writes and adds her first post to the DAG.
# Nobody but her knows about the published content.
alice~> echo "I ate shit for breakfast" > post1.txt && ipfs add post1.txt
added <Post1.MultiHash> post1.txt

# Alice prepares the first publication event's payload.
# She chose her blog's multihash as the payload's data.
alice~> PAYLOAD_DIR=$IPNS_ROOT/<Alice.PeerId>/ipfs-pubsub/blog/1
alice~> mkdir -p $PAYLOAD_DIR
alice~> echo "<Post1.MultiHash>" > $PAYLOAD_DIR/data

// Before publishing the first publication, she needs to prepare the second publication event
// so that her followers will know when she posts again!
// She creates the following json and saves it to ~/second-publication.json
{
  "payloadLocation":"/ipns/<Alice.PeerId>/ipfs-pubsub/blog/2",
  "publishedBy":["<Alice.PeerId>"],
  "sequenceId":2
}

# Alice calculates the new publication event's hash, and posts that hash as the "next" reference
# in the first event's payload. Now we have a forward linked list!
alice~> ipfs add --only-hash ~/second-publication.json
added <SecondPublication.MultiHash> ~/second-publication
alice~> echo "<SecondPublication.MultiHash>" > $PAYLOAD_DIR/next

# The message payload for the first publication is ready to go.
# Alice updates her IPNS in preparation of broadcasting her blog post
alice~> ipfs add --recursive $IPNS_ROOT
added <NewIpnsRoot.MultiHash>
alice~> ipfs name publish <NewIpnsRoot.MultiHash>

# Now she finally publishes to her subscribers, satisfying the wantlist of her many blog followers and notifying
# them that content has updated
alice~> ipfs add --pin ~/first-publication.json

Step 3: Bob grabs the new content, waits for more

Subscriber (Bob)
"Oh look, we finally received FirstPublication.MultiHash. I guess Alice finally posted!"
"Well let's look at the content."

# Now that Bob has the contents of first-publication.json, he can go get the payload. He does this by dereferencing
# the payloadLocation of publication event he just received, and grabbing the "data" link from that tree.
# You'll recall that the "data" link contained the ID to the actual blog content, and the "next" link
# contained the future ID for the next publication event.

bob~> ipfs cat <FirstPublication.payloadLocation>/data | ipfs cat
I ate shit for breakfast

# "Gross. I wonder what she'll say next. Let's subscribe to the next one."
bob~> ipfs get <FirstPublication.payloadLocation>/next | ipfs get
# This will block until Alice writes her next blog post