Skip to content

Instantly share code, notes, and snippets.

@varunbankar

varunbankar/GSoC-2020-Project-Report.md

Last active August 30, 2020 09:19
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save varunbankar/1d292808cd5de204602e68c5888145be to your computer and use it in GitHub Desktop.
Save varunbankar/1d292808cd5de204602e68c5888145be to your computer and use it in GitHub Desktop.
Download ZIP
A quick description of work done during GSoC 2020 under CERN-HSF organisation.
Raw
GSoC-2020-Project-Report.md

Ganga GUI Project

This issue will act as the start point for the Ganga GUI project and will contain the plans and progress of the project. All the future issues will also be mentioned here and this will be a singular base for understanding the project pathway.

About Project

This project aims to implement a Graphical User Interface (GUI) directly into Ganga, which could run alongside the existing text-based command line in Python. The implementation of the GUI will add an additional interface for the user to submit, monitor, and manage the job. Moreover, this project is more than just an implementation of a GUI, the creation of Web APIs will add modularity to the existing software and give users an ability to integrate the tool in other applications, to pull jobs data and perform actions on it, locally as well as remotely. This project has 7 major components along with 1 experimental component. The components are:

  • Integration of a web server into Ganga.
  • Creation of Web APIs for every possible interaction with the frontend.
  • Creation of a clean, minimal looking, mobile responsive frontend UI.
  • Connection of the frontend to the backend using AJAX requests.
  • Improving the server security and implementation of authentication methods.
  • Enhancing the reliability and responsiveness of the GUI and the Web APIs.
  • Creation of unit tests and documentation of the features implemented.
  • (Experimental) Integration of a CLI directly into the frontend UI.

You can check out the proposal here: Ganga Graphical User Interface Proposal.pdf

Summary of Work Done

List of PRs

Summary of PRs

PR #1675

  • Create GangaGUI top level package
  • Understand the CI testing
  • Create start_gui() & stop_gui() functions
  • Set up testing framework

PR #1683

  • Add separate database support to GangaGUI
  • Create token-based authentication
  • Create tests for database and token authentication

PR #1685

  • Provides a fix for a small bug in stop_gui function

PR #1700 - Create Job API which creates a RESTful interface for interactiving with Job data of Ganga

  • Create /api/job/<job_id> - GET
  • Create api/job/<job_id>/<attribute> - GET
  • Create /api/job/create - POST
  • Create api/job/<job_id>/<action> - PUT
  • Create /api/job/<job_id> - DELETE

PR #1702 - Create Subjobs API which creates a RESTful interface for interacting with Subjobs data of Ganga

  • Create /api/job/<job_id>/subjobs - GET
  • Create /api/job/<job_id>/subjob/<subjob_id> - GET
  • Create /api/job/<job_id>/subjob/<subjob_id>/<attribute> - GET

PR #1703 - Create Jobs API which creates a RESTful interface for interacting with Jobs data of Ganga

  • Create /api/jobs - GET
  • Create /api/jobs/ids - GET
  • Create /api/jobs/incomplete_ids - GET

PR #1704 - Create Config API for interacting with Config data of Ganga

  • /api/config - GET

PR #1705 - Create Templates API for interaction with Templates data of Ganga

  • /api/templates - GET
  • /api/template/<template_id> - DELETE

PR #1706 - Create Credential Store API for interacting with Credentials data of Ganga

  • /api/credential_store - GET
  • /api/credential_store/renew - PUT

PR #1707 - Create Jobtree API for interacting with Jobtree data of Ganga

  • /api/jobtree - GET

PR #1716 - Start working on the view routes

  • Create a dashboard view with recent jobs info and quick statistics

PR #1717 - Continue working on the view routes

  • Create a config view to see each config section, their options, and effective values

PR #1728 - Restructure the whole project and change the app server which used to run on a GangaThread to two servers. One internal server which now runs on a GangaThread and has access to all the Ganga GPI functions and data. Another one gui, with which the user interacts and manages the rendering and serving of the HTML, CSS, JS files. Please check the PR to know more as it essentially changes many of the earlier implementations of the APIs.

Following endpoints were implemented in the internal server:

  • Job API

    • /internal/jobs/<int:job_id> - GET
    • /internal/jobs/<int:job_id>/<attribute> - GET
    • /internal/jobs/<int:job_id>/full-print - GET
    • /internal/jobs/create - POST
    • /internal/jobs/<int:job_id>/copy - PUT
    • /internal/jobs/<int:job_id>/<action> - PUT
    • /internal/jobs/<int:job_id> - DELETE
    • /internal/jobs/<int:job_id>/export - GET

  • Subjobs API

    • /internal/jobs/<int:job_id>/subjobs - GET
    • /internal/jobs/<int:job_id>/subjobs/length - GET
    • /internal/jobs/<int:job_id>/subjobs/<int:subjob_id> - GET
    • /internal/jobs/<int:job_id>/subjobs/<int:subjob_id>/<attribute> - GET
    • /internal/jobs/<int:job_id>/subjobs/<int:subjob_id>/copy - PUT
    • /internal/jobs/<int:job_id>/subjobs/<int:subjob_id>/full-print - GET

  • Jobs API

    • /internal/jobs - GET
    • /internal/jobs/length - GET
    • /internal/jobs/statistics - GET
    • /internal/jobs/incomplete-ids - GET
    • /internal/jobs/recent - GET

  • Config API

    • /internal/config - GET
    • /internal/config/<section> - GET

  • Templates API

    • /internal/templates - GET
    • /internal/templates/<int:template_id>/full-print - GET
    • /internal/templates/length - GET
    • /internal/templates/<int:template_id> - DELETE

  • Credentials API

    • /internal/credentials - GET
    • /internal/credentials/renew - PUT

  • Jobtree API

    • /internal/jobtree - GET

  • Plugins API

    • /internal/plugins - GET
    • /internal/plugins/<plugin_name> - GET

  • Other API

    • /internal/jobs/actions - GET
    • /internal/load - GET
    • /internal/runfile - GET
    • /ping - GET

The following endpoints were created in the gui server:

  • View Routes

    • /login
    • /logout
    • /
    • /config
    • /create
    • /create/runfile
    • /templates
    • /jobs
    • /jobs/<int:job_id>
    • /jobs/<int:job_id>/export
    • /jobs/<int:job_id>/edit
    • /job/<int:job_id>/browse
    • /job/<int:job_id>/browse/<path:path>
    • /jobs/<int:job_id>/subjobs
    • /jobs/<int:job_id>/subjobs/<int:subjob_id>
    • /credentials
    • /plugins
    • /plugin/<plugin_name>
    • /logs
    • /storage
    • /storage/<path:path>
    • /cli
    • connect - Websocket
    • pty-input - Websocket
    • resize - Websocket
    • /shutdown

  • Web APIs

    • Token
      • /token - POST
    • Job Web API
      • /api/jobs/<int:job_id> - GET
      • /api/jobs/<int:job_id>/<attribute> - GET
      • /api/jobs/<int:job_id>/full-print - GET
      • /api/jobs/create - POST
      • /api/jobs/<int:job_id>/copy - PUT
      • /api/jobs/<int:job_id>/<action> - PUT
      • /api/jobs/<int:job_id> - DELETE
      • /api/jobs/<int:job_id>/pin - GET
      • /api/jobs/<int:job_id>/unpin - GET
    • Subjobs Web API
      • /api/jobs/<int:job_id>/subjobs - GET
      • /api/jobs/<int:job_id>/subjobs/<int:subjob_id> - GET
      • /api/jobs/<int:job_id>/subjobs/<int:subjob_id>/<attribute> - GET
      • /api/jobs/<int:job_id>/subjobs/<int:subjob_id>/full-print - GET
      • /api/jobs/<int:job_id>/subjobs/<int:subjob_id>/copy - PUT
    • Jobs Web API
      • /api/jobs - GET
      • /api/jobs/statistics - GET
      • /api/jobs/incomplete_ids - GET
    • Config Web API
      • /api/config - GET
      • /api/config/<section> - GET
    • Tempaltes Web API
      • /api/templates - GET
      • /api/templates/<int:template_id>/full-print - GET
      • /api/templates/<int:template_id> - DELETE
    • Credentails Web API
      • /api/credentials - GET
      • /api/credentials/renew - PUT
    • Jobtree Web API
      • /api/jobtree - GET
    • Plugins Web API
      • /api/plugins - GET
      • /api/plugins/<plugin_name> - GET

Screenshots of View Routes

Dashboard View

Screenshot 2020-08-30 at 1 47 46 PM

Create View

Screenshot 2020-08-30 at 1 47 53 PM

Templates View

Screenshot 2020-08-30 at 1 48 00 PM

Jobs View

Screenshot 2020-08-30 at 1 48 08 PM

Job View

Screenshot 2020-08-30 at 1 56 38 PM

Subjobs View

Screenshot 2020-08-30 at 1 56 43 PM

Subjob View

Screenshot 2020-08-30 at 1 56 50 PM

Plugins View

Screenshot 2020-08-30 at 1 48 14 PM

Config View

Screenshot 2020-08-30 at 1 48 28 PM

Web CLI View

Screenshot 2020-08-30 at 1 48 49 PM

TODO

  • Create internal documentation for future Ganga developer to understand the code
  • Write more tests
  • Make few aesthetic changes to the GUI

Challenges & Learnings

Since I started contributing to Open Source my jouney has been full of challenges and constant learning from them. While I was working on this project I faced a lot of challenges while designing the project and how it should be approached so it will be finished in time as well as be easier to understand for other people. There have been a lot of mistakes I did while completing this project. One of the major challenge was initially I had used only one server which ran on a single GangaThread, essentially meaning it was a very weak server and not very secure as well. Later, I had to restructure the whole project where I made use to two webserver, one was a weak server running on a GangaThread but it's only job to act as a RESTful interface between the main gui server (with which the user interacted. The gui server handled all the load of rendering and servering of web pages. It ran on a production grade gunicorn server. Another major challenge was intergrating a Web CLI inside of the GUI. As it needed to invert the approach of starting Ganga session first and then the GUI server.

All these challanges have only helped my grow more and become a better thinker and problem solver. I am grateful to have tackled these problems and implement a workable solution for them.

Acknowledgement

I have learnt great many things about web development through this project. I also got a hands on experience on create a project this big end to end. I am grateful to CERN-HSF and Google for giving me this wonderful opportunity to be able to contribute to such massive project. I would also like to thank my amazing mentors Prof. Ulrik Egede and Prof. Alexander Richards for guiding me throught this summer and helping me out whereever I need. They have been an absolute essential part of this journey.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment