GSOC_2022_Work_Submission_Final_Report.md

Final Work Submission Report of my GSOC 2022 work

1.Project Overview:

• Name: Sunny Tarawade
• Organisation: EMBL-EBI (European Bioinformatics Institute)
• Organization Project: FAANG (Functional Annotation of ANimal Genomics)
• GSOC Project Title: New FAANG backend with Elasticsearch and GraphQL
• Proposal: https://drive.google.com/file/d/1H3KCIB2zHQxwiiG56IGx56J2PhTSbYUY/view?usp=sharing
• Code:
  • Frontend : https://github.com/FAANG/dcc-portal-frontend/tree/st/data-join-ui
  • Backend : https://github.com/FAANG/GSOC-dcc-validate-metadata/tree/develop

2. Mentors:

I want to thank my mentors who gave me the guidance because of which I have been able to accomplish the work which I have done so far. Thank you so much Alexey Sokolov, Akshatha Nayak, Raheela Aslam and Yogmatee Roochun, for this fantastic experience!

3. Deployed Work URL:

Please take a look at the deployed work for a hands on experience here:

• Frontend (FAANG data portal ‘/data-join’ route)
  • Deployed Work : http://45.88.80.127/data-join
  • Docker Image : https://hub.docker.com/repository/docker/sunnytarawade/angular-app
• Backend (GraphQL server with Graphiql UI)
  • Deployed Work : http://45.88.81.159:8000/subscriptions/
  • Docker Image : https://hub.docker.com/repository/docker/sunnytarawade/django-app

4. Weekly Work Logs:

Please find the relevant commits here:

• Backend Commit history
• Frontend Commit history

Week 1: Completed Local Setup for Django Project with GraphQL. Added Schemas for Analysis, Experiment and Organism indices.

Week 2: Added logic for performing join between analysis and experiment indices. Also added a function to check the depth of join nested query (how many indices are we joining). Added logic for sending join query in the filter argument of GraphQL Schemas.

Week 3: Handled the case where the foreign field of the right index of joining is of list type. Also handled the Elasticsearch term querying for “keyword” type fields. Added Schemas and resolvers for all remaining indices.

Week 4: Added all join relations for all schemas. Handled one to one join relations. Fixed bugs and errors. Handled fetching of records by their primary keys. Handled parallel join queries.

Week 5: Wrote tests and documentation for the code written do far. Also kept fixing bugs.

Week 6: Handled the test-case for join relations of specimen index. Enhanced the documentation.

Week 7: Added a new route for the UI for fetching the joined data in the FAANG data portal. Added Index Selection Drop Down, form for selecting fields for one index.

Week 8:

• Frontend: Modified the UI as per discussion with the mentors. Added tabs for selected indices which contain the fields to be selected as well as an input which takes filter text input. Wrote the logic to convert the angular form data to an intermediate JS object which can be converted to graphQL query string using the npm package json-to-graphql-query. Added logic to pretty print the dynamic graphql query string. Completed the Apollo GraphQL Setup in Angular app.
• Backend: Handled CORS error. Completed Celery Setup.

Week 9:

• Frontend: Added sockets for fetching data. Added all the indices and their respective fields in the UI.
• Backend: Added new socket route for the server. Completed Django Channels Setup.

Week 10: Finished frontend integration with backend. Setup the flow for making sockets, setting up the tasks in celery and fetching the task results. Also deployed all the microservices (except frontend angular app) in the GSOC Kubernetes cluster provided by the mentors.

Week 11: Worked on displaying the fetched data in a table format in the UI, and fixing some bugs. Deployed frontend angular app.

Final Week : Fixing Bugs, writing documentation, completing final evaluation.

5. Project Status:

• Completed: Core functionality to fetch data, implement join between two or more Elasticsearch indices, filter joined data. UI to select the required fields and apply filters for the required data. Complete setup of the backend with Django, GraphQL, Django Channels and Celery setup. End to end integration between Frontend (Website UI) and Backend (GraphQL Server).
• Remaining: Bug fixing, Polishing UI, writing tests, documentation.
• Future work: Applying pagination, sorting to the table UI. Adding more documentation. Deploying code to production environment, etc.

6. Blogs:

I have also written blogs to document my GSOC 2022 Journey. Please find the links here:

Conceptual Blogs:

• Core:

  1. Intro: https://medium.com/@sunnytarawade000/sunnys-gsoc-adventure-c13b0f11d61
  2. Basic Data Fetching: https://medium.com/@sunnytarawade000/8bbb71d2b6c8
  3. Filtering Data: https://medium.com/@sunnytarawade000/299947f6cd01
  4. Joins: https://medium.com/@sunnytarawade000/102d6ac164e0

• Additional: 5. Nested Joins: https://medium.com/@sunnytarawade000/71b6fdc435e2

Technical Blogs:

1. Project Setup : https://medium.com/@sunnytarawade000/sunnys-gsoc-adventure-the-technical-d85bc9bb50bf
2. Folder Structure of “graphql_api” app: https://medium.com/@sunnytarawade000/sunnys-gsoc-adventure-the-technical-aa41c6eb5242
3. Fetching single document: https://medium.com/@sunnytarawade000/sunnys-gsoc-adventure-the-technical-8c7be6ca8f4a
4. The function resolve_with_join: https://medium.com/@sunnytarawade000/sunnys-gsoc-adventure-the-technical-3711673988e4

Throughout this program I have learnt so so much… from learning new technologies to Kubernetes deployment to documentation to time management, I would highly recommend GSOC to anyone reading!

And lastly, I would also like to thank Google for this wonderful opportunity! Thank you so much to everyone involved who has made GSOC 2022 a success!!! :)