Last updated: Wednesday, July 8th, 2020.
The API layer of a Harmony node is its main interface with the outside world. Therefore, it must be consistent, robust, and well-documented in order to provide easy integration with developers who wish to build on Harmony.
- Inconsistancies of response types (hex v.s. hex string v.s. number) and names (cammelCase v.s. kebab-case v.s. snake_case).
- Adding 1 RPC call consists of 5-7 files to edit and often contains duplicate code.
- Silent crashes and no logs of any error during a failed procedure calls.
- RPC's Http server occasionally fails to start on node boot.
- Documentation is not part of the dev cycle and must be manually created afterward.
- Testing is not part of the dev cycle.
For Harmony to grow and get external contributors, we must provide a professional interface for developers to hook into. Moreover, an easily consumable Node API can attract developers to take a deeper dive into Harmony's codebase and contribute to the core protocol.
Since Harmony has completed open staking, all core functionalities for an API layer have been defined. Therefore, now is an opportune time to correct the issues stated above as we are in the design phase of other major core features.
It is important to note that Harmony has numerous tools and partners that use the current JSON-RPC based API. Therefore, we must be careful to NOT break/change the existing RPC output format during the overhaul.
What this overhaul aims to do:
- Create a 'sanity-check' test suite that ensures the format & functionality of the current JSON-RPC.
- Clean-up the existing
./internal/hmyapipackages to reduce code duplication & dead code. - Create an additional API layer that provides consistency in all responses and retains all functionality from the v1 & v2 JSON-RPC services.
- Integrate documentation as part of the API dev cycle.
- Integrate testing as part of the API dev cycle.
- Provide error reports of call crashes in the existing & new API layer (low priority).
What this overhaul aims NOT to do:
- Fix race conditions of the underlying function calls.
- Break old v1 & v2 JSON-RPC functionalities.
- Wrap around the existing v1 & v2 JSON-RPC functions to create a new API.
Harmony uses JSON-RPC 2.0 for its node API (carried over from ethereum).
At a high level, the Harmony node starts an Http server (attached to some given port) for the RPCs and gives it a set of methods (referenced as API in the code) that are offered by the RPC interface.
Harmony uses the geth RPC package found here to handle the RPC's Http server. Said server gets started by the node on boot (here) by ultimately calling the geth function here. The APIs consumed by said function call is defined (here) specifically for the Harmony Node.
The 'common' backend interface, which connects the RPC layer with the node, can be found in ./internal/hmyapi/backend.go. The implementation of the interface that is used by all Harmony nodes can be found at ./hmy/api_backend.go as the APIBackend object, which just wraps around the Harmony object with a cache. Note that the Harmony object is not used in a meaningful way other than just a container of major data structures.
Harmony currently supports 2 RPC method namespaces, hmy & hmyv2. The supposed 'only' difference between the two is that hmyv2 returns values in decimal, whereas hmy returns values in hex. However, there are numerous occasions where hmyv2 is missing a method that hmy has and visa-vera. Moreover, there are various methods that have slightly different signatures, causing further confusion. The primary cause of these issues is that there are 2 distinct copies of each namespace's service and respective methods, which can be found here and here. Because of this code duplication, one has to define a new backend interface in 3 different places and implement it in 2 different places to properly add a new RPC. Couple this with the fact that we don't have any check for consistency between the namespaces, and it becomes easy to see why we have the issues mentioned above.
Lastly, Harmony's RPCs often returns the JSON text marshalled version of various data structures. However, we have inconsistent key naming conventions for said structures.
At a high level, we will first fix the existing JSON-RPC methods (focusing on ./internal/hmyapi), then we will implement the Rosetta node API.
The overhaul will have 7 stages:
- Lock the current RPC behavior with tests.
- Refactor JSON-RPC helper functions & method definitions.
- Implement a new node API (Rosetta).
- Docker node deployment.
- Integrate Rosetta's testing tool into our CI.
- Extend our implementation of Rosetta's node API for staking functions.
- Setup API documentation generation.
Similar to our existing PR testing script, we will orchestrate a series of transactions on a localnet and do RPC calls to assert the format and values of the output. Quite some work has been done here using pytest, and we shall follow/extend this format for the tests. We can also add more go-tests as we see fit.
Moreover, we will script the deployment of a localnet and execution of the test script within a (public) dockerfile & image.
A benefit of doing this first is that we are able to ensure that our existing tools won't break with any subsequent changes. Moreover, we should be able to replace the existing PR testing Jenkins job for a simple docker run command that we can integrate with our TravisCI.
We shall create a repo for all PR testing scripts & docker files for clarity.
The goal of the refactor is to remove duplicate code, dead code, and needless abstraction.
2.1 Remove the Backend interface
Since all RPC methods use a Backend, we can start there to reduce the needless abstraction. Note that we only have 1 implementation of this Backend interface with no plans (or need) to create more implementations. Therefore, we should remove the interface and use the Harmony object as the interface between the Node and API/RPC layer (for reasons described in 2.2).
2.2 Fuse APIBackend and Harmony together
As stated in the section above, APIBackend just wraps around the Harmony object. Therefore, we can just define all interfaces between the Node and API/RPC layer within the Harmony object.
We have heavy code duplication at the method definition for each RPC namespace here (apiv1 & apiv2). We should undo the code duplication introduced earlier this year and just carry a flag within each API struct (for example PublicBlockChainAPI) to indicate the RPC namespace. Then, the methods can adjust the output format (hex or decimal) according to this flag.
Since there are some inconsistencies between the signatures of each namespace, we should change the v2 signature to be the same as the v1 signature if we cannot find a suitable fix. Justification for this is that
apiv1has been out for longer and more tools (from Harmony & partners) are built on top ofapiv1.
There are numerous things leftover from ethereum that Harmony has not implemented (such as addrlock) that can be removed. We should take this time to check & remove this code.
As mentioned in the issues section, we have inconsistent key naming conventions for the responses of our JSON-RPCs. This is partially due to our inconsistent naming conventions for the JSON text marshall of our data structures. As these text marshalls will be used in the implementation of the additional node API, we should make the naming convention consistent with 'snake_case'. However, we CANNOT break the output format of our current RPCs. Therefore, we will add a 'reformatter' of sorts (when necessary) to keep the current output format.
The goal of this overhaul is to provide a consistent, robust, and well-documented API. Rosetta's node API provides a low-enough level API specified using OpenAPI 3.0 and works well for Harmony's needs. Moreover, Rosetta provides a SDK along with tooling and tests in their rosetta-cli that will speed up implementation and ensure correctness. We see Rosetta as a standard that other chains will follow, therefore it would be advantageous for us to use this API specification as it will be easier to integrate with future partners and developers.
Rosetta does not provide a spec for staking related functionalities. However, we can extend our implementation of Rosetta's node API for staking commands (done at a later stage).
Note that the implementation of this node API can hook into the Node's 'backend' interface cleanly with the refactor from stage 2. This will greatly speed up the implementation time of this stage.
To satisfy the Rosetta node API expectation, we must provide a dockerfile for running a Harmony node. Expectation is outlined here. Note that the dockerfile must build the node and run it (with certain storage restrictions). Fortunately, we have some work done in our dockerfile here that we leverage along with the dockerfile created in stage 1.
Once this stage is done, we would have a complete implementation of Rosetta's node API.
Rosetta provides tests for the node API via the Rosetta CLI (as seen here). We will integrate said tests into our CI test, either as a stage in a Jenkins job, or as an additional step in the new test suite (stage 1).
As mentioned in stage 3, Rosetta's node API does not provide a spec for staking related functions. We can integrate staking transactions by defining these transaction types as a supported operation. However, fetching staking related network information will be done by extending our implementation of the spec via an additional /staking endpoint. This way, we don't add fields or overload existing fields in the Rosetta spec, thus making our implementation more upgrade friendly for future versions of Rosetta.
Note that the Rosetta spec was generated using Open API 3.0, therefore we can extend this spec by defining the staking endpoint APIs (example here) and generating the necessary code to implement it (example here). Our intention is to stay as close as we can to the design direction of Rosetta yet remain independent enough for future upgrades of Rosetta's node API.
A separate, design doc will be created when we get to this stage and will be linked here (not specified yet).
With a new set of staking endpoints in this API extension, we must create our own set of tests to integrate into our CI. We shall add a test suite to the pytest from stage 1 since it will more 'future proof' than duplicating & extending the checks done with the Rosetta CLI.
Since the Rosetta node API is specified using Open API 3.0, we can use existing tools like Swagger UI and Swagger Inspector to generate documentation (with sample requests & responses) from the API specification. Example here. The sample calls can be done on a 'documentation' node to provide consistent responses.
Dates are TENTATIVE.
Each milestone will have a pull request associated with it.
Start Date: July 9th, 2020
End Date: August 31st, 2020
| PR | Discription | Target Date |
|---|---|---|
| Stage 1 | Main repo PR RPC test suite + CI setup | July 16th, 2020 |
| Stage 2.1 | 'Backend' RPC/API structure refactor (2.1, 2.2 & 2.3) | July 21st, 2020 |
| Stage 2.2 | JSON key name fix + 'reformatter' & remove dead code (2.4 & 2.5) | July 24th, 2020 |
| Stage 3.1 | Rosetta server on node boot + rosetta service router | July 25th, 2020 |
| Stage 3.2 | Rosetta's /network & /account node endpoint services |
July 31st, 2020 |
| Stage 3.3 | Rosetta's /block, /construction, and /mempool node endpoint services |
Auguest 7th, 2020 |
| Stage 4 | Dockerfile for node build & deployment (Rosetta's node API fully implemented) | Auguest 11th, 2020 |
| Stage 5 | Integrate Rosetta CLI check tool into our CI | Auguest 12th, 2020 |
| Stage 6.1 | Formal design doc with staking API spec | Auguest 14-15th, 2020 |
| Stage 6.2 | Types & function code gen + integration with rosetta-sdk-go | Auguest 19th, 2020 |
| Stage 6.3 | Add /staking node endpoint services to new node API |
Auguest 26th, 2020 |
| Stage 7 | Setup documentation code generation & webpage | Auguest 31st, 2020 |
This solution would be to just create another JSON-RPC version that fixes all of the aforementioned issues (after the refactor in stage 2). This would be a simpler approach and would be faster to implement. However, we would like to support a standard that would help partners & developers onboard faster and we see Rosetta's node API spec as the best solution for Harmony to achieve this.
That said, the design of this overhaul should allow for easy implementation of a v3 JSON-RPC in the future if we feel like we need it.
The solution here would be to drop the v2 RPC to simplify stage 2 of the implementation. This would speed up the implementation time, however, we have to consider key partners and developers that currently use v2.
As we have had the v2 RPC for some time now, we should continue to support it. Furthermore, it is likely that the time saved in the implementation now would be repaid later in extra time spent on coordination, support, and patching our existing tools.
While this is possible to do, it will likely introduce further code duplication, which may lead to inconsistencies between the JSON-RPCs and the node API. Also, we need to make our codebase more consumable for external developers, and it is likely that the RPC/API layer will be their introduction to it.
Furthermore, we still need to support the JSON-RPCs for the foreseeable future and will likely add RPCs or functionality to the new node API in the future. However, it is currently very cumbersome to add such things and plenty of dev time is wasted in the process. Therefore, it is worth it to take the time now to refactor the backend.
We want the extensions that we make to Rosetta's node API to be as independent as possible so that it would be easy to upgrade our implementation of the API in the event that the node API spec receives an update. Moreover, this simplifies the implementation of staking functionalities as we do not have to worry about violating the expected returns of the node API.
There is a possibility that partners may suffer some downtime when refactoring the RPC's 'backend'. However, extra time has been allocated to define thorough tests that would minimize this possibility. Moreover, we shall do a testnet deployment first and encourage partners (with the help of the bis team) to test their functionality on our testnet. We will also ask P-OPS to test.
This is where updates will be posted regarding current progress & hurdles.
Current Stage: 1
7/9: TBD
Love the idea to integrate testing from the very beginning.