This gist summarizes the work done by me under the guidance of my mentors Alexander Kanitz and Álvaro González during the Google Summer of Code 2021 for the Global Alliance for Genomics and Health organization on the project Brokering Continuous Delivery through the ELIXIR Cloud service registry.
The Global Alliance for Genomics and Health is a nonprofit alliance that works towards creating frameworks and standards for secured sharing of genomic and health-related data under human rights framework.
ELIXIR Cloud & AAI is a subgroup of ELIXIR and the driver project of GA4GH that works on the implementation and development of policies and standards for GA4GH.
ELIXIR is a multinational Europe-based initiative that organizes and coordinates life science resources to enable easy access to data and services to researchers and industries across Europe.
There are various API standards implemented by ELIXIR Cloud & AAI in the form of container images which are used to share, run, execute and access data easily at GA4GH. A major problem in rolling out service updates is the need for close communication between the developers and the deployment team, which needs to coordinate if, when, and how deployments should be updated on the production server.
This whole process is repeated for every new update and system administrators need to update their deployments manually for every new release or change.
The main objective of this project is to implement a microservice to bridge the gap between developer and system administrators (deployment side), and also maintain Service registry. It is a centralized registry set up at ELIXIR Cloud & AAI which gives track of available deployments and provides lists of specific services.
The project is an open-source decoupled, publish-subscribe-based continuous integration (CI) and continuous delivery (CD) solution that sits between CI and CD to allow developers to notify deployments of available updates. System Administrators can decide updates accordingly using notification on their callback from project.
The project uses the following tools/technologies/language for implementation:
- Foca
- Python
- Kubernetes
- OpenAPI
- Docker
- PyTest
The implementation is a Kubernetes deployment consisting of 3 components:
- Broker (Main-Service)
- Build-Complete-Updater
- MongoDB (Database)
This service is a foca application responsible for the following:
- Fetching committed changes from git repositories.
- Building container images.
- Uploading images to container registry.
- Managing subscriptions.
- Calling callbacks of all subscriptions in order to notify them of updates.
This service uses
gitpythonto clone repositories and checkout to specified branch, commit or tag and useskubernetesCoreV1Apito create kaniko deployments in k8s-cluster. Container images are build using kaniko. It is a tool to build container images from Dockerfile inside a container or Kubernetes cluster.
Kaniko deployments change to completed state when their task (to create and upload image) is completed successfully. So, this service continuously looks for kaniko pods in completed state and notifies the main-service by using endpoint PUT /repositories/{repo_id}/builds/{build_id}. Main-service then eliminates kaniko pod, updates build information, and notifies subscriptions.
A simple MongoDB service used to store application data.
During the contribution period in the last 2 months, I was able to achieve the following milestones:
- Created OpenAPI specifications: Designed application routes and payloads for the endpoints using OpenAPI 3.0. Created swagger UI.
- Implemented controllers for endpoints: Controllers to manage repository/builds/subscriptions, checkout repository, update build completion, notifying subscriptions.
- Docker-in-Docker image build using kaniko: Functions to create/delete kaniko pods and deployment files.
- Notify subscriptions: Send requests on callback URLs of all subscriptions that are listening to the specific repository on build creation.
- Unit testing: Write unit tests and mock functions for the project.
Milestones that still need some work:
- Connection to service registry.
- Side-Car service: It is a microservice to be co-deployed with project at deployment end to listen for update notifications from the project and update the existing service accordingly.
- Integration testing.
Future milestones could be:
- Handling build failures using kaniko-pod states.
- User Registration for
/subscriptions(System Administrators).
I am extremely grateful to my mentors Alexander Kanitz, Álvaro González, Kushagra Nagori, Sarthak Gupta for their earnest advice and constant guidance to me for the project "Brokering Continuous Delivery through the ELIXIR Cloud service registry"
During the course of this project, I did not only learn to write quality code but also have been able to research on some best development practices (especially for Docker-in-Docker solution) and successfully implemented a quality project.
I would like to thank the GA4GH organization and my mentors for believing in my capabilities and providing me with a wonderful experience. My journey with this organization is not over yet and we still have a long way to go...
