Skip to content

Instantly share code, notes, and snippets.

@whyrusleeping
Created September 28, 2018 23:42
Show Gist options
  • Save whyrusleeping/b84aeb7c32a9d535ac51874dc7e14a42 to your computer and use it in GitHub Desktop.
Save whyrusleeping/b84aeb7c32a9d535ac51874dc7e14a42 to your computer and use it in GitHub Desktop.
Some notes on how we can reduce the bid-deal overhead

Deal Aggregation

Calculations are still showing things not scaling as well as we would like them to.

The current (as of writing) bottleneck is that we are assuming the 'average file size' to be 100MB, and that clients will want (on average) a 5x encoding blowup (effectively, 5 replicas), this leads to requiring hundreds of bytes on-chain per gigabyte. This, in turn, constrains the maximum storage we can provide. To get past this bottleneck, we need to reduce that ratio. This means that we need to increase the effective 'average file size' per bid/deal pair.

Payment Channel Bidding

One 'simple' thing we can try is to have clients lock up funds to be stored in payment channels to their chosen repair miner. Then, to make a deal, the client sends (off-chain) a bid/deal proposal, along with a set of contingent payments, to the miner. The miner (assuming they accept) would post a deal containing all the information that was previously in the bid, and tell the client this. Then later, they can claim their funds by redeeming the payment channel updates.

Now, for a single deal between the client and the miner, this has a significant overhead as compared to the non-payment channel approach. The 'deal' part is slightly smaller than the combined size of the previous bid and deal, but the payment channel setup, and the payment channel redemption definitely push this over the threshold (especially because the information needed to validate the payment channel redemption is potentially large). The wins here come when a single client wants to make many deals with a single repair miner, and is willing to lock up money for all of that storage up front.

Deal Dataroot Aggregation

A repair miner accepting many deals could take multiple deals that they are making concurrently, and combine the dataroots of each of the independent deals into a single hash, and post that on chain. This allows the 'deal' to really just be a single hash commitment from the miner that updates the set of data they are responsible for replicating (need to think about deal expiry, how do miners ever remove these?) The complicated bit here is payment. When making the deal, the client should submit their proposal to the miner, along with a set of payment channel updates contingent on the existence of a deal on-chain whose merkletree is, or contains, the data the client wishes to store. The repair miner can then make several deals like that, and then aggregate all those deal data roots into a single deal, then go back to each client and ask them to switch their payment channel conditions to being contingent on that specific deal being valid and on-chain still (for a discount).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment