- Name: Ayush Kumar
- Project: Federating workflow execution using GA4GH Cloud APIs
- Organization: Global Alliance for Genomics and Health
- Mentor: Alexander Kanitz, Alvaro Gonzalez, Anurag Gupta, Thanasis Vergoulis
This report summarizes the work done by me in the Google Summer of Code 2022 program as a contributor for the Global Alliance for Genomics and Health organization, under the guidance of the mentors Alexander Kanitz, Alvaro Gonzalez, Anurag Gupta, Thanasis Vergoulis.
The Global Alliance for Genomics and Health is a nonprofit alliance that works towards creating frameworks and standards for secured sharing of genomic and health-related data under human rights framework.
ELIXIR Cloud & AAI is a subgroup of ELIXIR and the driver project of GA4GH that works on the implementation and development of policies and standards for GA4GH.
ELIXIR is a multinational Europe-based initiative that organizes and coordinates life science resources to enable easy access to data and services to researchers and industries across Europe.
The Task Execution Service (TES) API and the Workflow Execution Service (WES) are two of the several standards defined by the Global Alliance for Genomics and Health (GA4GH). The global picture is that TES and WES are designed to complement each other in running workflows by first identifying and scheduling the different individual tasks in the right order (WES) and then executing them in a containerized manner (TES).
- Current state of workflow execution is
TES-client -> TES instance. - In the above-mentioned setup, the WES or TES specs does not include any provision for task distribution, but in federatic analytic setup task distribution is important for use cases like load balancing, sending compute to data etc.
- Instead of adding such functionality into every client, this approach uses a 'gateway TES' that from outside looks like a TES, but that really is a client that can be used to receive and relay incoming TES requests and inject task distribution logic and any other middleware so that it can be used between any TES client and service.
- Then integrating 'gateway TES' in the stack,(
TES-client -> gateway TES-> TES Instance) (with task distribution logic).
Implemented a federated mechanism of task execution, i.e., different tasks of the same workflow will be executed in different TES instances.These TES instances can be in different geographical locations and be managed by different institutions.
- This can be implemented by introducing 'gateway TES' named as proTES, where proTES is a proxy-like implementation of the GA4GH TES OpenAPI specification based on Flask and Connexion built for distributing TES tasks over different TES service instances and injecting other middleware into TES requests.
- Here, the TES client i.e. cwl-TES breaks the workflow into different tasks request and sends them in orderly manner to gateway TES i.e. proTES.
- Then a task distribution middleware is called to identify, out of several options, the most suitable TES instance, then relays the incoming task to that instance.
- TESK (An implementation of a task execution engine based on the TES standard running on Kubernetes.) is used as TES instance for federated analytics setup.
Distribution of task is done by task-distribution-middleware as follows:
Command used :
cwl-tes --tes http://localhost:8080/ga4gh/tes --remote-storage-url ftp://vm4466.kaj.pouta.csc.fi/upload/foivos hashsplitter-workflow.cwl --input test.txt
Execution of hashsplitter-workflow (cwl-TES -> proTES -> TESK):
- Hashsplitter workflow is passed through the TES client (here cwl-TES) which breaks it into different tasks that are ( md5, sha, whirlpool and unify).
- md5, sha and whirlpool are passed to 'gateway TES' while unify waits until they are executed successfully.
- Here, 'gateway TES' (proTES) calls a task distribution middleware where it injects the best optimal TES instance into the TES request.
- Finally, the tasks forwarded to the instance and the output are stored at the mentioned remote storage (here, ftp).
- During the process of task execution at the instance the proTES asynchronously fetches the task status from TES instance, which is ultimately fetched by cwl-TES to know the state of the task.
- If all the tasks i.e. md5,sha and whirlpool are executed successfully ,unify is passed for further execution where it checks all the output are at right location or not if found, the overall workflow ends successfully.
During the contribution period in the last 2-3 months, I was able to achieve the following milestones:
- Refactored proTES to make use of FOCA
v0.9.0and removed boilerplate code. (PR #71). - Fixed the
POST /tasksendpoint and made controllers fully functional in proTES. (PR #71). - Added tracking of TES task progress asynchronously using celery workers where database updates are made from the workers directly. (PR #74).
- Added task distribution middleware and basic task distribution logic that randomly sends tasks to TES instances on which they can be executed. (PR #75).
- Added Integration test in CI and added unit test. (PR #76).
- Performed end-to-end test i.e. cwl-TES -> proTES -> TESK.
- Updated documentation of proTES. (in progress).
Though most of the milestones are achieved some milestones that still needs some work are:
- To Design a better abstraction/design of middleware.
- To write a better task distribution logic, like TEStribute.( Initial goal was to integrate TEStribute to proTES, but the TEStribute has little outdated task distribution logic and also not very well suited for Kubernetes that’s why we left it and written a dummy logic for now.)
- To do end-to-end testing with cwl-wes.
- To do end-to-end testing with Funnel.
I am extremely grateful to all the mentors Alexander Kanitz, Alvaro Gonzalez, Anurag Gupta, Thanasis Vergoulis and the whole ELIXIR Cloud & AAI community without their guidance the project would not have been possible. Also like to thank the organisation Global Alliance for Genomics and Health (GA4GH) and ELIXIR Cloud for giving me the opportunity for the project Federating workflow execution using GA4GH Cloud APIs. Special thanks to Alexander Kanitz,for making the coding period smooth.
During this GsoC period I learned to write great quality code and best practices that should be kept in mind by any developer and learned so many things in this short period of time. This journey is very special to me and is not over yet looking forward to learn and contribute more....
