Skip to content

Instantly share code, notes, and snippets.

@idoqo

idoqo/gsoc-final-submission.md

Last active October 19, 2022 20:56
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save idoqo/ee3da5de75cae761b72e940f63f336db to your computer and use it in GitHub Desktop.
Save idoqo/ee3da5de75cae761b72e940f63f336db to your computer and use it in GitHub Desktop.
Download ZIP
Final Submission for GSoC '20
Raw
gsoc-final-submission.md

Final GSOC Submission for the Moira Project: OpenAPI Description and Client Libraries of the Moira API

Name: Michael Okoko

Organization: Moira Alert

Project: https://summerofcode.withgoogle.com/projects/#5733016854331392/

Proposal: https://drive.google.com/file/d/14HpvkkBGcfaP676FWr-5qECOI7Gebkbp/view?usp=sharing/

Overview

The goal of the project was to:

  • Develop an up-to-date OpenAPI/Swagger description of the endpoints exposed by the API.
  • Generate language-specific client libraries from the API description file.
  • Setup automated API tests to test the API specification (contract) against the actual implementation.

OpenAPI Description

In my initial proposal, I had planned to document the API as code comments and transform them to an OpenAPI spec file with go-swagger. However, after discussions with my mentors, I realized writing the yml files by hand was better because it does not clutter the code base, and it provides more flexibility in the long run.

To document the APIs, I generated all the method/path pairs available in the Moira API using chi-walker, and created a folder per route group. For instance, methods, requests and responses relating to the /contact route group are kept in the contact folder. This will hopefully make maintaining/updating the documentation easier.

Results

Publishing OpenAPI spec to SwaggerHub

To make the API documentation accessible to Moira users, it is published to a public SwaggerHub repository. The multiple yml files are bundled together using the swagger-cli tool, and validated to ensure that the schema is correctly written. Schema validation is handled by the amazing openapi-generator-cli tool.

The build-validate-publish process is automatically executed as part of the CI flow. In my proposal, I planned on using TravisCI but later realized Github Actions can do the job just fine so I settled for that instead. The CI pipelines are executed as follows:

  • On PR to master branch: Merge all the yml files into a single openapi.yml file and validate the schema.
  • On Merge to master branch: Merge the yml files, validate the schema and publish the schema to SwaggerHub as "version master".
  • On Push to feature branches (feature/* e.g., feature/add-teams) and version branches (v[0-9]+.[0-9]+.[0-9]+ e.g., v2.5.13), merge the yml files, validate the schema, and publish to SwaggerHub - using the branch name as the version name.

Results

Auto-Generated Client Libraries

With a valid OpenAPI spec file, it is possible to generate client libraries for different languages using openapi-generator-cli. The languages it supports are listed here and luckily, it supports the languages most important for the Moira team (i.e Python, C#, TypeScript, and Golang).

I wrote Makefile rules to generate the clients and work is ongoing to check the code quality of the auto-generated code.

Ideally, the generated clients will be published to their respective package repositories (i.e Python => Pypi, TypeScript => npm, C# => Nuget, Go => pkg.go.dev).

Results

Contract Testing of the Moira API

It is one thing to have an API documentation, ensuring it is correct is another thing, especially as the documentation will be updated when the core Moira APIs changes too. To help this, I set up a process to test the OpenAPI spec against the real Moira API and add it to the CI pipelines.

I looked at Dredd and Schemathesis and in the end, settled on using Schemathesis because it had better support for OpenAPI 3, and it can use OpenAPI links to improve how tests are run.

Schemathesis works by generating request payload (i.e request bodies, query parameters, and path parameters) from the examples in an OpenAPI document, making the request to the paths and comparing the response it gets with what is defined in the document. For example, given this OpenAPI schema,

ContactRequest:
  type: object
  description: "Format of the request body for POST/PUT requests to `/contacts` and related sub-paths."
  properties:
    type:
      type: string
      example: "mail"
    value:
      type: string
      example: "devops@example.com"

The request payload below will be generated.

{
  "type": "mail", 
  "value": "devops@example.com"
}

It also makes it possible to specify "fixed" request data using hooks (such as when data in the database needs to be re-used).

My mentor suggested an alternative that involved setting up our own custom script with dummy data, that way, we can easily reproduce the tests at any time. It is currently a work in progress.

Result

PR #68 - add test rule to Makefile and integrate with CI

Future Work

I hope to see the OpenAPI documentation fully integrated into Moira and see it updated as changes are made to the core Moira API.

We could also explore using a stricter validator (such as IBM/openapi-validator or Redocly/openapi-cli) as they supports a lot more rules for validation e.g missing descriptions and missing examples).

Learnings/Challenges

Working on this project taught me a lot about how important API documentation is. I also learnt a lot about how OpenAPI works and the ecosystem around it (mock servers, library generation, converters, etc).

My challenges mostly revolved around testing API contracts and working with Github Actions (and CI pipelines), especially as Actions is a relatively new tool and there are not much resources online.

In the end though, I'm thankful to Emil, Alexey, and Nixolay - my mentors - for always being there to help and making it easier than it would have been.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment