Create a gist now

Instantly share code, notes, and snippets.

@karalabe /hackathon.md Secret
Last active Nov 11, 2016

What would you like to do?
Gopher Gala - Ether APIs

Gopher Gala - Ether APIs

This document intends to be kind of an outline as to what we'd like to do for the Gopher Gala hackathon. The idea behind it is to gather and polish our ideas a bit so that we don't run around like some headless chickens durng the weekend, but rather have a solid approach as to what and how we'd like to accomplish.

I chose a GitHub gist as the medium for this document mainly because it's simple to write using all the features of markdown and because we can move it into the Gopher Gala repo when the hackathon starts. Feel free to fork it and modify any parts and I'll merge them in straight away, this extra step is really only needed as long as we need to keep this document private.

Timeline

First thing's first, the timeline of the hackathon is 48 hours, starting at Janaury 22 at 5pm PST (that's 01:00 UTC on January 23). Or to put it another way:

Start End
Alex 23:00 Firday, January 22 23:00 Sunday, January 24
Jeff 02:00 Saturday, January 23 02:00 Monday, January 25
Peter 03:00 Saturday, January 23 03:00 Monday, January 25

Please register to the hackathon before the deadline!!

Project summary (story we need to sell)

With the advent of cloud computing, everyone is trying to create some sort of online service that you can access and interact with via an online API. As the popular iOS slogan goes "there's an app for that", in the API world it's a similar mantra: "there's an API for that", irrelevant what you're trying to build: machine translation, mapping service, email sending, video rendering and streaming, etc., you'll find a large or small company that does exactly that, and the only thing you need to do is to hook into their system.

However, the online service and API ecosystem has a less attractive side, a problem yet unsolved, which leads to a high barrier of entry for small projects: trust. New and geographically remote projects are inherently untrusted, which makes payment for said services very problematic.

Pain point

There are two main types of API providers: large corporations and small to medium scale projects, the two of which have very different offers when it comes to paid APIs.

  • Large corporations (e.g Google, Amazon) require a consumer to only have a credit card on file, and whenever the user consumes some API, it is measured and counted against the user's quota. At the end of the payment cycle, the client is charged with the outstanding amount. This however has the drawbacks that I need to provide all my personal and payment information to the API provider, as well as there's a risk that I consume more than I would like (e.g. forgot to turn off the test VMs), causing a huge charge that I have to pay nonetheless.
  • Small and medium companies on the other hand do not have the necessary resources to "lend" API capacity for an entire month and only afterwards get their own costs covered. Additionally, they do not have the means to force clients with bad payment information to actually come through. These two monetary shortcomings force the smaller API providers into a pre-paid subscription model, where an API consumer needs to pay a potentially hefty sum up front, which also expires at the end of the payment cycle. Opposed to large corporations (which can be somewhat trusted), a user is not comfortable with giving away personal infos to unknown API providers from far away countries, neither is he comfortable with paying a lot of money up front, just to realize the API is unreliable, or worst the provider a scammer.

A prominent solution currently available is offered by mashape: an API marketplace, broker and payment gateway acting as an escrow service between providers and consumers. Providers register their API endpoints into mashape's centralized brokerage service (configuring an associated pricing model), which on the other hand exposes those hidden APIs to the general public via their own servers. As all API invocations pass through them, those can be individually authorized, accounted for and charged at the end of the payment cycle. The value mashape brings to the API ecosystem is a payment escrow service, where API providers and consumers don't have to trust each other, but rather they all trust mashape itself to do the right thing. In exchange, mashape charges a 20% flat fee on all transactions that go through its brokarage service.

The problem with the current API ecosystem is that it requires blind trust in one or both of the API participants and true pay-per-use can only be achieved by the big players in the industry. All the existing solutions to these problems take the shape of centralization, which again requires a mutually trusted party, but also introduce a potential point of failure, a potential bottleneck and last but not least, significant privacy concerns.

Solution

The proposed solution that we're pushing for is the re-decentralization of online services and APIs (like they originally were meant to be), but in a fully trustless way, where neither of the API participants has to know the other party or even trust it with funds (pre-lent or pre-paid).

By binding the financial contracts to the Ethereum blockchain, the API consumer and provider can remain in charge or their communication and the payment execution is enforced and secured by the censorship resistant Ethereum blockchain. Payment is made on a per-call basis, resurrecting the dream of a "pay for what you use": an API consumer only pays for what he actually uses, at the exact moment when he uses it; whereas an API provider can at any point redeem the payments made to his service, without the consumer being able to refuse the already authorized payments.

Use cases (recipies ala the current Ethereum website)

So, what could this decentralized and trustless API ecosystem support:

  • Stateless APIs: each call authorizes payment for its own data processing requirement
    • E.g. machine translation, geolocation, image rendering
  • Streaming APIs: payment is authorized for some amount of future processing, renewed until both parties desire
    • E.g. music streaming services, video rendering services
  • Private APIs: the system can handle the payment for privately negotaited and executed calls
    • E.g. business to business private API execution

Value proposition

So why is this project better that anything out there?

  • Decentralized: there isn't a single point of failure, single point of bottleneck, or single point of trust
  • Trustless: nethier the API provider or consumer needs trust, everything is enforced by the blockchain
  • Private: interacting peers are anonymous, communication is direct between them, no middleman
  • Secure: Ethereum ensures all transactions are final, inalterable and non censurable

How much does such a setup cost?

  • Registering a new API, subscribing to an API, charging all accumulated payments is ~$0.0015 (blockchain fees)
  • Searching for APIs and making payments for API calls is completely free (done off chain)
  • Payment contract charges a service fee of 1% (opposed to the 20% of existing competitors)

Hackathon attack plan :)

I think the important part here is that we have a clear idea on who will do approximately what and an estimate roadmap with some milestones so that we can heave a barebone system up and running fast, that we can polish till the end of the hackathon. Everything below here is only a suggestion that I came up with to have a base for discussion and further clarification. As this project has three mayor components (website, contract, client), within each milestone I split up the discussion into these parts.

Milestone 1: Frontier

For the first milestone we should aim for a very basic but functioning version of the system. This would entail having a simple website that just states what the project is, what can be done with it and how. Contract wise it should just be the simplest possible thing that can register APIs, search and subscribe to them and make payments. Client wise it should be a CLI wrapped up Ethereum client interfacing the IPC/RPC, making it possible to interact with the contract manually.

My suggestion is that in this initial milestone Alex should focus on the website, as he only has experience with these kind of stuff, Jeff shoud focus on the contract as it's something he knows best how to do fast/well, and I'll (Peter) focus on getting the CLI stuff set up and interface it with Ethereum, as it's the parts I've been working with during the node separation PRs.

Website

IMHO, the website should be purely static, without requiring any backend. This way we can push it up to Google Cloud Storage and host it straight from there (I've done this a few times so I can set this up fairly quickly) and we can point Cloudflare in front of it to keep things fast, snappy and cool. Cloudflare would also provide us with an SSL certificate so it would look very professional :).

Content wise, I would aim for a similar design that we have with Ethereum, where the landing screen would contain a graphic summary of the problem we want to solve and our value proposition (i.e. somehow we need to condense those sections on a one scree teaser), after which I would have recipes similar to Ethereum, but each of them addressing one of the use cases (i.e. stateless api, streaming api, private api).

Contract

Contract wise the first iteration should contain just enough functionality to allow integrating it with the rest of the client and allow the client so evolve on its own writing the API request proxying, payment verifications etc. In the beginning we can probably test this via JavaScript as we have the automatic ABI generation, but we should definitely not spend time writing a .js API.

The minimum essentials to me would be:

  • API endpoint registration, specifying:
    • provider that can charge the payments (not necessarilly self, but only self can unregister)
    • price that can be charged for each call or some resource unit (cpu time, bandwidth, etc)
    • name, description, url and other meta fields, all of them optional (private APIs)
    • Would return some API ID (e.g. SHA3(auto id | provider | price))
    • Would be nice to maintain a map APIs/provider, to allow listing them
  • API subscription
    • api id identifier from the above registration
    • consumer that can authorize payments on this subscription (not necessarilly self, but onlt self can cancel)
    • funding initial funds allocated to the subscription
    • Would return some subscription ID (e.g. SHA3(api id | consumer))
    • Would be nice to maintain a map of subscriptions/consumer, to allow listing them
  • Allocate further funds into subscription
    • sub id to add further funds to
    • funding amount to add to the balance (i.e. Ethereum value)
  • Authorizing the payment of N calls (always +1 from previous authorization)
    • This is done off chain to be instant and not incur costs
    • sub id is the subscription to authorize
    • count is the number of units to authorize (calls, bytes, cycles, whatev.)
  • Charge subcription
    • sub id to charge
    • authorization signature + charge allowance
    • This must maintain the previous charge and only charge from there onwards (sliding window)
  • Cancel subscription
    • sub id the subscription to cancel
    • Obviously subscirption must be owned by self
    • Returns all remaining balance to owner (the withdraw protection will be in a later milestone)
  • Unregister API
    • api id the API endpoint to remove
    • Obviously the endpoint must be owned by self
    • Subscription associated would remain live (allows supporiting old APIs/pricings for existing customers)
  • Listing and searching methods (read only, free)
    • List all available APIs, or filter those provided by a specific account
    • List all the subscriptions of a particular user (mostly interesting to list mine)
Client

Milestone 2: Homestead

This milestone would be where we actually provide the users with some kind of demo capability. This would entail creating and distributing the pre-loaded giveaway accounts; assembling the demo codes and scripts to allow anyone to try the thing.

Website
Contract
Client

Milestone 3: Metropolis

Website
Contract
Client

Milestone 4: Serenity

Website
Contract

rating, reliability, usage statistics

Client

Logistics

Code repository

The hackathon requires regular pushes to the project repository (I'm guessing to ensure that nobody's cheating by checking in huge chunks of pre-written code). To cater for this, my suggestion is that we commit and push often; there's not much point in doing pull requests or reviews at this stage. To prevent screwing someone else's work, I think the best approach would be to use feature branches that can kind of simulate a PR, with maybe giving the team the heads up that there's a change incoming so rebase fast; though those should also be keps small to avoid ugly rebase conflicts. Squashing is probably out of the question to maintain the integrity of the progress made.

A potential structure of the repository so that we don't walk all over outselves could be:

 /client
 /contracts
 /website

Ethereum test network

Although we could launch and maintain a private network solely for this project's demo purposes, it would kind of beat the point of using Ethereum as the driver behind the scenes, so I'd opt for using the live test network for contract deployment and general API registry/lookup, subscriptions etc.

The added benefit of the test network is that we have the live bootstrap servers, live miners, preconfigured Ethereum client and last but not least, etherscan.io has a live testnet explorer that we can direct users to to investigate their accounts, our contract etc; and ethernodes.org also have a live testnet topology viewer which would make the whole thing a lot more interactive and visual.

I'll create, fund and post here the testnet account that we can use for the development when the Gala starts (don't want to cross the rules by pre-creating it before hand). We can assume a total balance of about 20K ethers at our disposal to cover all out needs.

Account:

To be created...

Password:

gophergala

Demo accounts

As the test network's a bit unstable when it comes to block creation (mainly because people turn miners on and off constantly, whacking the difficulty all over the place), I propose that we creat a batch (~1K) of demo accounts and pre-load them with some (~10) ether.

It would probably also make sense to host a few demo APIs ourselves, some of which the demo accounts would be already subscribed to and some of which not. The idea would be that if someone wants to play with our API code, we should provide enough functionality on our website/playground to let them do so from within the browser, instead of having to run any custom code on their machine. It would probably be a very nice teaser into the idea.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment