| eip | title | description | author | status | type | category | created |
|---|---|---|---|---|---|---|---|
<to be assigned> |
Beacon State API endpoint |
A proposal for a general purpose beacon state access endpoint in the `light_client` API |
Zsolt Felföldi (@zsfelfoldi) <zsfelfoldi@ethereum.org> |
Draft |
Standards Track |
Interface |
2022-11-04 |
This document proposes an addition to the light_client beacon node API that provides access to the beacon state with Merkle multiproofs. It is based on subscriptions, meaning that consumers of the API can specify in advance which parts of the state they are interested in, either for the current head, a single future slot, each future slot or every Nth slot periodically. This allows collecting beacon state data as the chain is being processed, without requiring the underlying beacon node to store or re-generate past beacon states.
A simple post-merge light client should be able to process sync committee updates and signed beacon headers in order to obtain the latest canonical beacon header, then get a Merkle proof for latest_execution_payload_header.root in order to prove the corresponding execution block header, from which it can prove actual user data. More advanced light client designs might also make use of other beacon state fields. For example, the post-merge Geth light client design uses the historical_roots and state_roots trees to prove any canonical block from the latest head. It stores a partial proof of the beacon state for each canonical block, including latest_execution_payload_header.root for each block, historical_roots and state_roots for the most recent blocks and also committee_root and next_committee_root for potential checkpoints. It also maintains entire state_roots subtrees of past periods as subtrees of the historical_roots tree, which is something that the consensus nodes might not even store in that form. This data structure allows access with proofs to past events, transactions and receipts. It also allows propagating this entire dataset (or parts of it) to other nodes with proofs, addressing the bandwidth scalability issues of the old PoW Geth light client design that relied on only full consensus processing nodes serving clients.
Though other light client designs may have different needs, it looks generally useful to think about light client serving nodes as a role different from consensus processing nodes. The former have lot smaller bandwidth/storage requirements in terms of beacon chain/state data but might also store data structures that the consensus nodes do not. Also, more of them might be needed in order to serve all client needs. Therefore, for a future proof general purpose interface, an API endpoint is proposed that allows collecting the required parts of the beacon state data for each block while the chain is being processed, without making any strong assumptions on the architecture and internal data structures of the underlying client.
The proposed endpoint is accessible at the /eth/v1/beacon/light_client/state URL. It always returns a binary application/octet-stream response unless the request is malformed. Both subscribe and fetch requests can return either a proof result, an error result, and empty result or no result at all (see below).
A valid state request is either a subscribe or a fetch request. A subscribe request has a mandatory format parameter which describes the shape of the requested beacon state tree subset in a hex encoded format (see below). It also has optional start and period integer parameters that define the slot or series of slots where the described state proof is requested. If start is not specified then currentHead.slot is used as the starting point. If period is specified and greater than 0 then slots are requested periodically, for each slot start + k*period where k >= 0. If a requested slot is empty then a proof should be generated for the next non-empty slot. Note that for periodic requests start can also be in the past (in practice it will probably often be 0) but the API is not expected to generate state proofs for previously processed blocks.
A subscribe request always returns an 8 byte subscription ID. It also returns a result immediately for the current head state if the head slot is requested (there is an integer k >= 0 for which start + k*period = currentHead.slot). A fetch request should always refer to a subscription by specifying its ID in the id parameter in hex encoded format. If an optional slot integer parameter is specified then the result for the given canonical slot (or the next non-empty slot if the given slot is empty) is returned if available. If no slot is specified then the result for the highest canonical slot number is returned if any results are available for that subscription.
If a subscription with the same format, start and period parameters already exists then the API should return the existing subscription ID. This allows sharing subscriptions between multiple instances of the same application. For the same reason applications are advised to use start = 0 for periodic subscriptions (or start < period if an offset is required).
A valid proof result according to the given format is a series of 32 byte tree nodes (see below); the number of nodes corresponds to the number of 1 bits in the format descriptor. A proof result is only ever generated in the requested format; if this proof cannot be generated because the requested parts of the tree are not available or do not exist (children of a leaf node are requested) then an error result is generated. This consists of the state root and the generalized tree index of one of the non-existent tree nodes in 8 byte little endian format. The error result is 40 bytes long and therefore always distinguishable from a proof result.
If an existing block is specified but no result is available then an empty result response is given which is 32 bytes long and contains the state root of the block. It means either that the block has been processed before the subscription was created or that the result has already been discarded. In either case the client should not expect a proof for that block in the future.
Note that the block that the result belongs to can be identified in all of the three cases above through its state root (in case of the proof result the state root is the result of the proof verification). It is assumed that the verifier knows the recent canonical headers that the state requests apply to because during proof verification the reconstructed roots should be compared against the state roots in the headers. Therefore it is also assumed that the client can identify canonical blocks by their state roots or notice if a result is not part of their view of the canonical chain and refresh their chain and/or their state proofs until they match each other. Also note that since the results always belong to the current canonical chain, in case of a reorg existing results may need to be re-generated.
If the block does not exist yet (a future slot is specified or no slot is specified and no results exist yet at all) then no result is returned.
A subscribe request returns the subscription ID in the first 8 bytes of the response (little endian encoded). An immediate result for the head state (if requested) is returned after the id. If a subscription is unsuccessful then an empty byte vector is returned. A fetch request receives the ID as a parameter so it only returns the result itself. The type of the result can always be determined from the size of the returned byte vector as shown below (N is the number of 1 bits in the proof format descriptor):
If subscribe returns
- 0 bytes: subscription failed
- 8 bytes: subscription active for future slot
- 48 bytes: subscription active, error result for current head state
- 8+N*32 bytes: subscription active, proof result for current head state
Note that an empty result is never returned for the head state as it is always available for processing so either a proof result or an error result is returned if the head is requested.
If fetch returns
- 0 bytes: no result (future slot requested)
- 32 bytes: empty result for requested slot
- 40 bytes: error result for requested slot
- N*32 bytes: proof result for requested slot
The beacon state tree subset specified in the request is encoded as a bit string and represented in the request parameters as a hex string with MSB first encoding order. The bit string follows the path of a depth-first, left-to-right traversal of the subtree, starting from the root; a 1 bit represents a tree node (either leaf or internal) included in the proof response while a 0 bit means that the two child subtrees of that node are encoded recursively.
For example, a format descriptor for a proof that proves the generalized tree index 42 (marked with *):
1* 1
\ /
1 0
\ /
0 1
\ /
1 0
\ /
0 1
\ /
0
Proof format: 00100101111 = 0x25e
The format descriptor of a proof consisting of N tree nodes contains N 1 bits and N-1 0 bits. The API should support proof sizes up until at least N = 1024 (in which case the format descriptor is 2047 bits or 512 hex digits long). The proof result consists of N 32 byte tree nodes, stored in the same depth-first, left-to-right traversal order as the format bit vector.
The following fields and structures of the beacon state might be relevant to a light client design and therefore are expected to be fully available on request:
genesis_time: uint64
genesis_validators_root: Root
slot: Slot
fork: Fork
block_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT]
state_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT]
historical_roots: List[Root, HISTORICAL_ROOTS_LIMIT]
finalized_checkpoint: Checkpoint
current_sync_committee: SyncCommittee
next_sync_committee: SyncCommittee
latest_execution_payload_header: ExecutionPayloadHeader
Note that the historical_roots list that contains the root hashes of HistoricalBatch containers is not expected to go deeper than these root hashes; any design that intends to use this structure to prove historical canonical blocks can collect the individual state roots and block roots as the chain is being processed and reconstruct the linked sub-structures internally.
Whenever a slot is processed that an active subscription is subscribed to, a (subID, slot) -> result association is stored in memory where result is either a proof result or an error result. Note that empty result is returned when no such association exists for the queried (subID, slot) while no result is returned when no such canonical slot exists, therefore these result formats do not require an explicit in-memory representation. Subscriptions do not belong to a single API consumer so that multiple consumers running the same client logic can share them. Therefore there is no way to explicitly delete stored data and some strategy is required to discard both individual stored results and entire subscriptions whenever they are not needed anymore. A simple approach is proposed here but this strategy might evolve later according to application needs.
The amount of stored results is limited both per subscription and globally. Each subscription has a FIFO queue of slot -> result associations. The length of the queue depends on the period parameter; if period = 0 then queueLength = 1, otherwise queueLength = (255+period) // period. This ensures that even if the beacon client is in a syncing phase, all results can be catched with fetch requests. It also ensures that a subscription will require storing at most queueLength * N tree nodes or roughly queueLength * N * 32 bytes where N is the number or tree nodes per proof and is defined by the proof format of the subscription.
Single-slot subscriptions are discarded 30 seconds after generating the result for the requested slot. Periodic subscriptions are discarded if the most recently generated result has not been fetched for 30 seconds. Global limit for all subscriptions should be at least SUM(sub.queueLength * sub.N) <= 65536, meaning that the memory required for storing proofs is limited at roughly 2Mb max plus some memory allocation overhead. If a new subscription would exceed this limit then it fails.
Note: though single-slot subscriptions to the current head return their results immediately, they should also be treated as subscriptions which in practice means that the results are still retained for 30 seconds and recalculated if a reorg happens. Also the global subscription limits apply to them, which prevents overloading the serving node with expensive calculations.
Note that this strategy is satisfactory when the API is not open to the public and there is no risk of others spamming or overusing the API and making it unusable for the intended use case. Applications like the Geth light client service that collects data locally and then serves remote clients can work fine with such an only locally enabled light_client API. For public endpoints a more sophisticated method (separate limits for individual consumers) might be necessary. The details of such a strategy are outside the scope of this document.
The proposed API specification was driven by the need to collect partial beacon state proofs from each canonical block. It is based on the assumption that past states may not be easily available for the underlying consensus client, therefore the required state subset should be specified in advance and the proofs should always be generated from the current (or currently processed) state. One possible way to do this would be to use SSE or WebSocket and once a subscription is created automatically transmit new proofs as states are processed. In theory these might seem to be a logical choice for a subscription API but since the rest of the light_client API uses simple HTTP request-response model, it is probably easier to implement and use this feature within the same framework. Using subscribe and fetch requests does not require unnecessary frequent polling because the API consumer is expected to know about new heads and possess the canonical headers for the requested slots anyway and therefore should know when to request the generated results.
Though the subscriptions are not completely stateless like the rest of the light_client API, once they are created and IDs are assigned their parameters are immutable. Multiple API consumers using the same subscription parameters can receive the same ID and use the same subscription without interfering with each other. A consumer does not need to care about whether a subscription already exists or care about its internal state after getting the ID at startup.
A testing tool will be implemented in the Geth codebase soon (currently WIP).
Does not exist yet.
The only potential security risk related to this API could be a DoS attack vector against the serving consensus client if the endpoint is exposed to the public. Limiting subscriptions as suggested is necessary as it avoids both CPU and memory overuse. Spamming with fetch requests is not a bigger issue than the possibility of spamming any other public API endpoint because serving fetch is cheap as it only returns pre-processed byte vectors.
Copyright and related rights waived via CC0.