This document intends to be kind of an outline as to what we'd like to do for the Gopher Gala hackathon. The idea behind it is to gather and polish our ideas a bit so that we don't run around like some headless chickens durng the weekend, but rather have a solid approach as to what and how we'd like to accomplish.
I chose a GitHub gist as the medium for this document mainly because it's simple to write using all the features of markdown and because we can move it into the Gopher Gala repo when the hackathon starts. Feel free to fork it and modify any parts and I'll merge them in straight away, this extra step is really only needed as long as we need to keep this document private.
First thing's first, the timeline of the hackathon is 48 hours, starting at Janaury 22 at 5pm PST (that's 01:00 UTC on January 23). Or to put it another way:
| Start | End | |
|---|---|---|
| Alex | 23:00 Firday, January 22 | 23:00 Sunday, January 24 |
| Jeff | 02:00 Saturday, January 23 | 02:00 Monday, January 25 |
| Peter | 03:00 Saturday, January 23 | 03:00 Monday, January 25 |
Please register to the hackathon before the deadline!!
With the advent of cloud computing, everyone is trying to create some sort of online service that you can access and interact with via an online API. As the popular iOS slogan goes "there's an app for that", in the API world it's a similar mantra: "there's an API for that", irrelevant what you're trying to build: machine translation, mapping service, email sending, video rendering and streaming, etc., you'll find a large or small company that does exactly that, and the only thing you need to do is to hook into their system.
However, the online service and API ecosystem has a less attractive side, a problem yet unsolved, which leads to a high barrier of entry for small projects: trust. New and geographically remote projects are inherently untrusted, which makes payment for said services very problematic.
There are two main types of API providers: large corporations and small to medium scale projects, the two of which have very different offers when it comes to paid APIs.
- Large corporations (e.g Google, Amazon) require a consumer to only have a credit card on file, and whenever the user consumes some API, it is measured and counted against the user's quota. At the end of the payment cycle, the client is charged with the outstanding amount. This however has the drawbacks that I need to provide all my personal and payment information to the API provider, as well as there's a risk that I consume more than I would like (e.g. forgot to turn off the test VMs), causing a huge charge that I have to pay nonetheless.
- Small and medium companies on the other hand do not have the necessary resources to "lend" API capacity for an entire month and only afterwards get their own costs covered. Additionally, they do not have the means to force clients with bad payment information to actually come through. These two monetary shortcomings force the smaller API providers into a pre-paid subscription model, where an API consumer needs to pay a potentially hefty sum up front, which also expires at the end of the payment cycle. Opposed to large corporations (which can be somewhat trusted), a user is not comfortable with giving away personal infos to unknown API providers from far away countries, neither is he comfortable with paying a lot of money up front, just to realize the API is unreliable, or worst the provider a scammer.
A prominent solution currently available is offered by mashape: an API marketplace, broker and payment gateway acting as an escrow service between providers and consumers. Providers register their API endpoints into mashape's centralized brokerage service (configuring an associated pricing model), which on the other hand exposes those hidden APIs to the general public via their own servers. As all API invocations pass through them, those can be individually authorized, accounted for and charged at the end of the payment cycle. The value mashape brings to the API ecosystem is a payment escrow service, where API providers and consumers don't have to trust each other, but rather they all trust mashape itself to do the right thing. In exchange, mashape charges a 20% flat fee on all transactions that go through its brokarage service.
The problem with the current API ecosystem is that it requires blind trust in one or both of the API participants and true pay-per-use can only be achieved by the big players in the industry. All the existing solutions to these problems take the shape of centralization, which again requires a mutually trusted party, but also introduce a potential point of failure, a potential bottleneck and last but not least, significant privacy concerns.
The proposed solution that we're pushing for is the re-decentralization of online services and APIs (like they originally were meant to be), but in a fully trustless way, where neither of the API participants has to know the other party or even trust it with funds (pre-lent or pre-paid).
By binding the financial contracts to the Ethereum blockchain, the API consumer and provider can remain in charge or their communication and the payment execution is enforced and secured by the censorship resistant Ethereum blockchain. Payment is made on a per-call basis, resurrecting the dream of a "pay for what you use": an API consumer only pays for what he actually uses, at the exact moment when he uses it; whereas an API provider can at any point redeem the payments made to his service, without the consumer being able to refuse the already authorized payments.
So, what could this decentralized and trustless API ecosystem support:
- Stateless APIs: each call authorizes payment for its own data processing requirement
- E.g. machine translation, geolocation, image rendering
- Streaming APIs: payment is authorized for some amount of future processing, renewed until both parties desire
- E.g. music streaming services, video rendering services
- Private APIs: the system can handle the payment for privately negotaited and executed calls
- E.g. business to business private API execution
So why is this project better that anything out there?
- Decentralized: there isn't a single point of failure, single point of bottleneck, or single point of trust
- Trustless: nethier the API provider or consumer needs trust, everything is enforced by the blockchain
- Private: interacting peers are anonymous, communication is direct between them, no middleman
- Secure: Ethereum ensures all transactions are final, inalterable and non censurable
How much does such a setup cost?
- Registering a new API, subscribing to an API, charging all accumulated payments is ~$0.0015 (blockchain fees)
- Searching for APIs and making payments for API calls is completely free (done off chain)
- Payment contract charges a service fee of 1% (opposed to the 20% of existing competitors)
I think the important part here is that we have a clear idea on who will do approximately what and an estimate roadmap with some milestones so that we can heave a barebone system up and running fast, that we can polish till the end of the hackathon. Everything below here is only a suggestion that I came up with to have a base for discussion and further clarification. As this project has three mayor components (website, contract, client), within each milestone I split up the discussion into these parts.
For the first milestone we should aim for a very basic but functioning version of the system. This would entail having a simple website that just states what the project is, what can be done with it and how. Contract wise it should just be the simplest possible thing that can register APIs, search and subscribe to them and make payments. Client wise it should be a CLI wrapped up Ethereum client interfacing the IPC/RPC, making it possible to interact with the contract manually.
My suggestion is that in this initial milestone Alex should focus on the website, as he only has experience with these kind of stuff, Jeff shoud focus on the contract as it's something he knows best how to do fast/well, and I'll (Peter) focus on getting the CLI stuff set up and interface it with Ethereum, as it's the parts I've been working with during the node separation PRs.
IMHO, the website should be purely static, without requiring any backend. This way we can push it up to Google Cloud Storage and host it straight from there (I've done this a few times so I can set this up fairly quickly) and we can point Cloudflare in front of it to keep things fast, snappy and cool. Cloudflare would also provide us with an SSL certificate so it would look very professional :).
Content wise, I would aim for a similar design that we have with Ethereum, where the landing screen would contain a graphic summary of the problem we want to solve and our value proposition (i.e. somehow we need to condense those sections on a one scree teaser), after which I would have recipes similar to Ethereum, but each of them addressing one of the use cases (i.e. stateless api, streaming api, private api).
Contract wise the first iteration should contain just enough functionality to allow integrating it with the rest of the client and allow the client so evolve on its own writing the API request proxying, payment verifications etc. In the beginning we can probably test this via JavaScript as we have the automatic ABI generation, but we should definitely not spend time writing a .js API.
The minimum essentials to me would be:
- API endpoint registration, specifying:
providerthat can charge the payments (not necessarillyself, but onlyselfcan unregister)pricethat can be charged for each call or some resource unit (cpu time, bandwidth, etc)name,description,urland other meta fields, all of them optional (private APIs)- Would return some API ID (e.g.
SHA3(auto id | provider | price)) - Would be nice to maintain a map APIs/provider, to allow listing them
- API subscription
api ididentifier from the above registrationconsumerthat can authorize payments on this subscription (not necessarillyself, but onltselfcan cancel)fundinginitial funds allocated to the subscription- Would return some subscription ID (e.g.
SHA3(api id | consumer)) - Would be nice to maintain a map of subscriptions/consumer, to allow listing them
- Allocate further funds into subscription
sub idto add further funds tofundingamount to add to the balance (i.e. Ethereumvalue)
- Authorizing the payment of N calls (always +1 from previous authorization)
- This is done off chain to be instant and not incur costs
sub idis the subscription to authorizecountis the number of units to authorize (calls, bytes, cycles, whatev.)
- Charge subcription
sub idto chargeauthorizationsignature + charge allowance- This must maintain the previous charge and only charge from there onwards (sliding window)
- Cancel subscription
sub idthe subscription to cancel- Obviously subscirption must be owned by
self - Returns all remaining balance to owner (the withdraw protection will be in a later milestone)
- Unregister API
api idthe API endpoint to remove- Obviously the endpoint must be owned by
self - Subscription associated would remain live (allows supporiting old APIs/pricings for existing customers)
- Listing and searching methods (read only, free)
- List all available APIs, or filter those provided by a specific account
- List all the subscriptions of a particular user (mostly interesting to list mine)
This milestone would be where we actually provide the users with some kind of demo capability. This would entail creating and distributing the pre-loaded giveaway accounts; assembling the demo codes and scripts to allow anyone to try the thing.
rating, reliability, usage statistics
The hackathon requires regular pushes to the project repository (I'm guessing to ensure that nobody's cheating by checking in huge chunks of pre-written code). To cater for this, my suggestion is that we commit and push often; there's not much point in doing pull requests or reviews at this stage. To prevent screwing someone else's work, I think the best approach would be to use feature branches that can kind of simulate a PR, with maybe giving the team the heads up that there's a change incoming so rebase fast; though those should also be keps small to avoid ugly rebase conflicts. Squashing is probably out of the question to maintain the integrity of the progress made.
A potential structure of the repository so that we don't walk all over outselves could be:
/client
/contracts
/website
Although we could launch and maintain a private network solely for this project's demo purposes, it would kind of beat the point of using Ethereum as the driver behind the scenes, so I'd opt for using the live test network for contract deployment and general API registry/lookup, subscriptions etc.
The added benefit of the test network is that we have the live bootstrap servers, live miners, preconfigured Ethereum client and last but not least, etherscan.io has a live testnet explorer that we can direct users to to investigate their accounts, our contract etc; and ethernodes.org also have a live testnet topology viewer which would make the whole thing a lot more interactive and visual.
I'll create, fund and post here the testnet account that we can use for the development when the Gala starts (don't want to cross the rules by pre-creating it before hand). We can assume a total balance of about 20K ethers at our disposal to cover all out needs.
Account:
To be created...
Password:
gophergala
As the test network's a bit unstable when it comes to block creation (mainly because people turn miners on and off constantly, whacking the difficulty all over the place), I propose that we creat a batch (~1K) of demo accounts and pre-load them with some (~10) ether.
It would probably also make sense to host a few demo APIs ourselves, some of which the demo accounts would be already subscribed to and some of which not. The idea would be that if someone wants to play with our API code, we should provide enough functionality on our website/playground to let them do so from within the browser, instead of having to run any custom code on their machine. It would probably be a very nice teaser into the idea.