A rough proposal to implement the widely desired pub/sub feature on ipfs without requiring additional capabilities or message-passing behavior from the network or the clients. It could be implemented as an IPFS application right now and in fact I've got sample JS that uses the js-http-api to do so on the publish side. Hopefully building out the subscribe side this week.
I've just started diving into IPFS so this could be a super naïve approach.
The proposal's core concepts:
- subscription is modeled as querying a non-existent block whose future multihash is known, but has not been added to the DAG. Once this block exists, its payload will use IPNS to provide a forward linked list to one or more future blocks in the publication stream.
- publication is modeled as publishing the block with that desired multihash, and providing a payload that links forward to one or more future blocks in the publication stream.
The proposal's key properties:
- It requires the IPNS router so it's not the fastest ride in town.
- It requires that clients patiently wait for
wantlistblocks that can't currently found on the network- I believe this is the current behavior of
go-ipfs
- I believe this is the current behavior of
- It is loosely inspired by mutable value chains
- It requires no additional capabilities or message-passing behavior from any actor in the network.
The proposal's minimum necessary flow (or just jump down to the example):
- PUBLISHER creates a SUBSCRIPTION LINK json object
- The link contains a reference to an
/ipns/address under PUBLISHER's control. - The reference will contain the future payload for the SUBSCRIPTION LINK.
- henceforth
<SubscriptionLink.PayloadLocation>
- henceforth
- The link contains a sequence ID unique in this stream (e.g. "1")
- In reality we will want to hash this with the private key to avoid spoofing
- The link contains a reference to an
- PUBLISHER calculates the SUBSCRIPTION LINK multihash, but does not add it to the DAG
- Uses
ipfs add --only-hash - Henceforth
<SubscriptionLink.MultiHash>
- Uses
- PUBLISHER distributes
<SubscriptionLink.MultiHash>widely - SUBSCRIBER adds
<SubscriptionLink.MultiHash>to hiswantlist. - SUBSCRIBER waits for the object to exist in the DAG.
- PUBLISHER writes content and adds it to the DAG.
- It has ID
<Content.MultiHash>
- It has ID
- PUBLISHER creates a tree/folder at
<SubscriptionLink.PayloadLocation>under his IPNS root.- Henceforth
PayloadTree
- Henceforth
- PUBLISHER writes
<Content.Multihash>intoPayloadTree/data - PUBLISHER calculates the hash for a NEW SUBSCRIPTION LINK using the same process as before
- This link now has sequence ID 2
- It should probably also contain an ipfs (not ipns) backlink to
PayloadTree - Henceforth
<NewSubscriptionLink.MultiHash>
- PUBLISHER writes
<NewSubscriptionLink.MultiHashintoPayloadTree/next - PUBLISHER publishes the new version of his IPNS root.
- The first subscription link's payload is now ready to consume, but nobody knows how to find it.
- PUBLISHER adds and pins the first SUBSCRIPTION LINK json object to the DAG.
- The subscription link (which is really more of a notification) starts percolating through the network as it satisfies subscribers wantlists.
- SUBSCRIBER, who has patiently waited
<SubscriptionLink.MultiHash>in hiswantlistfor ages, receives and re-pins the subscription link- Repinning helps other subscribers.
- SUBSCRIBER dereferences
<SubscriptionLink.PayloadLocation>, finding both the content and the next subscription link withinPayloadTree/dataandPayloadTree/nextrespectively. - SUBSCRIBER acquires the content using
ipfs get - SUBSCRIBER adds
<NewSubscriptionLink.Multihash>to hiswantlist, and waits.
Example of basic necessary flow:
| Publisher (Alice) |
|---|
| "Think I'll start a blog. Don't know what I'll write yet, but let me give the world a subscription link." |
// Alice creates a subscription link: this JSON content.
// She saves it to ~/first-publication.json
{
"payloadLocation":"/ipns/<Alice.PeerId>/ipfs-pubsub/blog/1",
"publishedBy":["<Alice.PeerId>"],
"sequenceId":1 // or Sign(1, PrivateKey)
}# Alice discovers the hash of this subscription link.
# She does not add it to the DAG.
alice~> ipfs add --only-hash ~/first-publication.json
added <FirstPublication.MultiHash> ~/first-publication.json # Not true! We didn't add it, we just discovered the hash.| Publisher (Alice) | Subscriber (Bob) |
|---|---|
"Hey subscribe to my blog at <FirstPublication.MultiHash>!" |
|
| "Sure, Alice!" | |
bob~> ipfs object get <FirstPublication.MultiHash> |
|
| Prepares to wait the long haul. Distributes this object on his wantlist, as does every other future subscriber to Alice. |
| Publisher (Alice) |
|---|
| "Time for the first post!" |
# Alice writes and adds her first post to the DAG.
# Nobody but her knows about the published content.
alice~> echo "I ate shit for breakfast" > post1.txt && ipfs add post1.txt
added <Post1.MultiHash> post1.txt
# Alice prepares the payload for the first publication
# She chose her blog's multihash as the payload.
alice~> PAYLOAD_DIR=$IPNS_ROOT/<Alice.PeerId>/ipfs-pubsub/blog/1
alice~> mkdir -p $PAYLOAD_DIR
alice~> echo "<Post1.MultiHash>" > $PAYLOAD_DIR/data// Before publishing the first publication, she needs to prepare the second subscription link
// so that her followers will know when she posts again!
// She creates the following json and saves it to ~/second-publication.json
{
"payloadLocation":"/ipns/<Alice.PeerId>/ipfs-pubsub/blog/2",
"publishedBy":["<Alice.PeerId>"],
"sequenceId":2 // or Sign(2, PrivateKey)
}# Alice calculates the new subscription link's hash, and posts that hash as the "next" reference
# in the first subscription's payload
alice~> ipfs add --only-hash ~/second-publication.json
added <SecondPublication.MultiHash> ~/second-publication
alice~> echo "<SecondPublication.MultiHash>" > $PAYLOAD_DIR/next
# The message payload for the first publication is ready to go.
# Alice updates her IPNS in preparation of broadcasting her blog post
alice~> ipfs add --recursive $IPNS_ROOT
added <NewIpnsRoot.MultiHash>
alice~> ipfs name publish <NewIpnsRoot.MultiHash>
# Now she finally publishes to her subscribers, satisfying the wantlist of her many blog followers and notifying
# them that content has updated
alice~> ipfs add --pin ~/first-publication.json| Subscriber (Bob) |
|---|
| "Oh look, we finally received <FirstPublication.MultiHash>. I guess Alice finally posted!" |
| "Well let's look at the content." |
# Now that Bob has FirstPublication, he can go get the payload. He does this by dereferencing
# the payloadLocation of publication event he just received, and grabbing the "data" link from that tree.
# You'll recall that the "data" link contained the ID to the actual blog content, and the "next" link
# contained the future ID for the next publication event.
bob~> ipfs cat <FirstPublication.payloadLocation>/data | ipfs cat
I ate shit for breakfast
# "Gross. I wonder what she'll say next. Let's subscribe to the next one."
bob~> ipfs get <FirstPublication.payloadLication>/next | ipfs get
# This will block until Alice writes her next blog post