Skip to content

Instantly share code, notes, and snippets.

@protolambda

protolambda/api_call_notes.md Secret

Created April 20, 2020 15:01
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save protolambda/51152013648367e4d9bd1874d0860bc9 to your computer and use it in GitHub Desktop.
Save protolambda/51152013648367e4d9bd1874d0860bc9 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
api_call_notes.md

Agenda

format eth2.0-apis/issues/24
    OpenAPI / JSON -- general agreement (?)
        int vs str-int
        bytes (0x vs base64)
    optional SSZ where easy/sensible
    optional protobuf refs (?)
    versioning
api conversions/CI eth2.0-apis/issues/26
    any updates
bn apis eth2.0-apis/issues/25
    Identifiers in the context of our types (vs. query params)
        /beacon/{id}/resource <- very nice construct, but would require some amount duck-typing

Format

  • JSON primary support
  • SSZ useful
  • proto (bufs) also, but not primary
  • Marin has some approach to convert to different formats (no link/ref?)

JSON issues:

  • integer size: thoughts on int vs string?
    • Jim: uint64 as string. ugly but necessary
    • needs consistency
    • felix: eth1 has 0x (0 = 0x0) and DATA vs QUANTITY types
    • felix: eth1 has 256 bit numbers too
    • felix: have been using json api for long. No complaints about strings
    • felix: api wrappers help a lot here
    • danny: need debugger readability
    • marin: issue on geth to use dec strings instead of hex
    • jim: 0x is regarded as bytes, decimal string makes sense for ints
    • jim: json is not performant anyway, should just look at readability, no hex
    • mamy: hex to native int is much faster. Since utf8 bytes can just be shifted by 4 bits
    • jacek: never seen 0x as byte array, only as big number
    • danny:
      • int is out
      • string dec/hex is in
      • compat eth1 something to consider
      • decide in next 48 hours
      • issue, hash it out

Danny: API conversions in CI:

  • interest in what is in repo and work with it
  • people want to experiment with conversions
  • don't shoot ourselves in the foot with CI

Danny:

  • thanks protolambda for API collection work and strawman proposal
  • thanks Mikhail

Danny:

  • lots of different ways to identify objects
  • challenge to get succinct APIs, find right balance
  • lots hinges on beacon state

Resource idea:

  • specify something in context of a beacon state resouce
    • solves validator index issue
    • ducktyping involved

Mikhail:

  • words on REST api, try to make it restful, another round of review
  • probable REST is not the best. There are others, like GraphQL.
  • Try to do restful first, continue from there

Danny: form working group to continue forward with iteration on format

Danny:

  • there are exceptions, e.g. with FFG endpoint
  • may not all fit current REST approach

Streams

  • Felix:
    • something we had for long and appreciate, but not made its way back
    • specification really bad
    • ability for realtime events
    • not really abvailable through rest api
  • Danny:
    • stream endpoints in proposal
    • prysm has stream with grpc
    • lighthouse has websockets
    • marin proposed server-side-events
  • Felix:
    • Geth has something (name?)
    • Subscriptions, as if the server was calling a callback method on the client
    • identifier + data
    • Support available
  • Raul: avoid polling
  • Terence:
    • two way streaming
    • beacon node can stream reassignment to validator
  • Felix
    • documentation about subscriptions approach is not good
    • quite a simple protocol, very happy, really easy to implement on top of json rpc
    • works on all transports, except http
  • Jim
    • Streaming preferable
    • But high workload
    • Idea: just a single stream, share notifications
    • E.g. chain splits, reorgs, etc.
    • Efficiency and api endpoint ease
    • Pull when interested as consumer
    • Simpler notification system, less work
  • Mamy
    • Push and pull api
    • GraphQL was designed to do that
  • Jacek
    • GrapQL is advanced feature, not good for low level adoption
  • Danny
    • keep it simple is good
    • when we do discuss these things, highlight as advanced features
  • Prysm
    • Care mostly about supporting streaming at all
    • Flexible in how it is carried out
  • Marin
    • Describes small somewhat stateful server stream approach
      • client asks for events since certain id
    • Jim/Felix: don't want stateful system
    • Felix: want to forget about past event
    • Felix: avoid replay later
    • Marin: clarifies, just as catch-up approach, don't want to miss events
    • Felix: interested now
    • Marin: maybe needs per api discussion
    • Proto: implemented/hacked the buffer approach already, with browser consuming lighthouse testnet data
    • Jacek: concerned about reorgs still
    • Felix: needs to get view of the chain right, display wallet example, quite a few states.
    • Felix: realtime events make it easier, but ensuring state is more difficult
    • Felix: single event stream, just the new blocks, call apis to update local state based on it. More RPC traffic, but easier to get consistent data.
    • Felix: who are we optimizing for? If browser, need to optimize back and forth information

Danny: TODOs

  • decimal/hex string
  • event streaming
  • formation of small working group to hammer out beacon api
  • not addressed validator apis, need to look at block explorers now
  • reopen validator api updates later

Proto: before close, post links of your API projects, interesting

Danny/jacek: advanced innovation

  • Rocketpool is supporting pooling software
  • validator api for advanced setups

Jacek:

  • Endpoint over libp2p to serve api RPC requests?
  • Run everything on single libp2p stack
  • Danny: who as consumer? Some endpoints not to be exposed to random people
  • Zahary: requests could be authenticated. REST or other new endpoints could pose as new exploit vectors
  • Danny: separation of API and p2p is also nice.
  • Felix: went wrong in eth1: only getting to it now with light client functionality
  • Felix: who will use APIs? What if web browsers need it? Same dilemma as eth1, use something as metamask that talks to centralized provider essentially.
  • Felix: how are browsers going to access? Can libp2p help?
  • Felix: suggestion of jacek makes sense
  • Zahary: something that acts as proxy between browser and network
  • Danny: Nimbus team can open issue for standard mapping between API and libp2p

Danny: working group for API endpoints:

  • Mikhail
  • Marin
  • Jim

Danny: working group telegram chat, present something in a week Danny: anybody else is welcome, but keeping small for fast/dynamic progress

Close

Chat:

16:02:05	 From danny : https://github.com/ethereum/eth2.0-pm/issues/143
16:11:31	 From Mikhail Kalinin : I vote for 0x instead of Base64, it sometimes handy for debugging to mentally convert values. Base64 would make it almost impossible.
16:12:56	 From Tomasz Stanczak : let us not use Base64 please
16:13:01	 From Tomasz Stanczak : losing readability a lot
16:13:52	 From Tomasz Stanczak : hex encoded numbers in Eth1 can be mentally converted to some extent - this helps a lot
16:14:24	 From Tomasz Stanczak : I stand by Felix and Mikhail here
16:14:59	 From Mamy : Base64 is for bytes though, we are talking about integer as stringified int vs hex
16:15:18	 From Mamy : (I agree that Base64 should be avoided)
16:15:27	 From Tomasz Stanczak : ok @Mamy, thanks
16:15:41	 From Mikhail Kalinin : May we end up with the default encoding of ints and keep the other option for those cases where it will be appropriate
16:15:42	 From Felix : V
16:15:43	 From Mamy : bytes serialisation is next i assume ;)
16:15:52	 From Tomasz Stanczak : as for numbers I am less in favour of 0x but it is acceptable
16:15:55	 From Felix : https://github.com/ethereum/wiki/wiki/JSON-RPC#hex-value-encoding
16:16:07	 From Tomasz Stanczak : but it caused us trouble in Nethermind - some libs are hard ot modify to handle these as numbers
16:16:15	 From Felix : for reference, the eth1 API encoding
16:17:20	 From Tomasz Stanczak : also we started allowing passing ints in decimal in our CLI and Geth is sometimes inconsistent - small numbers are sometimes sent as normal ints (but very rarely and I think it is just some minor technical debt from far in the past)
16:18:00	 From Mamy : Parsing a base 10: https://github.com/status-im/nim-stint/blob/master/stint/io.nim#L242-L261
16:18:12	 From Mamy : I didn’t put a shortcut for Hex but it’s significantly faster
16:28:01	 From Mikhail Kalinin : I will be capable of doing the work on API on Thursday or Friday this week.
16:28:32	 From Marin Petrunić : I can also help if needed :)
16:29:55	 From Mikhail Kalinin : That would be great! I am open for any help.
16:30:28	 From Felix : https://geth.ethereum.org/docs/rpc/pubsub
16:30:56	 From Marin Petrunić : I would just remind that this is pretty standard tehnology with support everywhere: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
16:31:13	 From Raul Jordan (Prysmatic) : +1
16:31:43	 From Chih-Cheng Liang : come across this quote, but not sure how to fit in to the current discussion https://twitter.com/realLedgerwatch/status/1210519703304708096
16:31:45	 From Chih-Cheng Liang : "What is JSON RPC for? Is it a Web API (needs to be 99% available, but not 100% accurate), or database access API (needs to be 100% accurate, but not necessarily 99% available)"
16:36:52	 From Raul Jordan (Prysmatic) : +1, graphql should be optional and not required for client implementers
16:36:54	 From Mike Goldin : Can GraphQL schemas be generated from OpenAPI?
16:37:06	 From Felix Lange : not really
16:37:18	 From Raul Jordan (Prysmatic) : https://github.com/IBM/openapi-to-graphql
16:37:26	 From Felix Lange : oh, OK
16:37:31	 From Felix Lange : didn't know that
16:37:33	 From Jim McDonald : Agree that simple here is good, JSON meets that goal and not sure that graphql does
16:50:36	 From Raul Jordan (Prysmatic) : https://github.com/gobitfly/eth2-beaconchain-explorer
16:51:33	 From Jim McDonald : https://github.com/wealdtech/ethdo talks to prysm nodes today to obtain bits and pieces of information, and has features such as sending voluntary exits.  I stopped work on the API pieces until it standardises, so lots still to do
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment