This summer, I participated as a student developer in the Google Summer of Code Program. In this program, university students contribute towards an open source project between May and July, under the guidance of the project mentors. I was selected with CERN-HSF as my mentoring organisation.
CERN-HSF is an umbrella organization for CERN, some international labs and universities for development of software pertaining to particle physics and computation/analysis of data related to it. There are a plethora of open source projects in CERN-HSF.
My work during this summer was on DIRAC which is a software framework for distributed computing. DIRAC builds a layer between the users and the computing grids, offering a common interface to a number of heterogeneous providers, integrating them in a seamless manner, providing interoperability, at the same time an optimized, transparent and reliable usage of the resources.
TLDR : If you or your organization owns millions of computer servers/grids, which are located across the globe, DIRAC would help you in distributing resources, monitoring your jobs across these.
My work was focussed on the ResourceStatusSystem(RSS) of DIRAC. This subsystem is responsible for maintaining the status of each computing resource (think storage, compute units, memory etc.). The aim of my project as a whole was to revamp the RSS to be more robust. This would involve the following:
- Unifying the RSS Backend into SQLAlchemy by providing a common base database connection module
- Porting certain modules from SQLite to SQLAlchemy + MySQL
- Writing performance tests.
- Moving RSS from Python 2.7 to Python 3
A big thanks to my mentors and fellow GSoC students who helped me solve countless doubts no matter how silly they were 😄
- Federico Stagni : Mentor
- Zoltan Mathe : Mentor
- Christophe Haen : Mentor
- Pujan Mehta : GSoC Student
The SQLAlchemy python library is an object relational mapper. In simple terms, it provides an abstraction over the SQL database, allowing us to read/write objects to the database without explicitly writing the SQL statements. Another benefit to using SQLAlchemy is the fact that the database is decoupled from the backend code, making any future database migration (say moving from MySQL to Postgres) very easy.
RSS has 2 database connector modules (ResourceStatusDB and ResourceManagementDB) written in SQLAlchemy. These 2 modules share a lot of common code for various database level operations/checks. Therefore my first task was to remove this redundancy by introudcing a common SQLAlchemy base module which the ResourceStatusDB and ResourceManagementDB can extend for their specific usecases. This task has been completed in #4121.
Before understanding this task, let me first explain 2 DIRAC components :
- Agent : An Agent is an independent process responsible for a specific task.
- Action : An action is a program which is triggered on some state change.
Often times, if a resource's status changes, users are to be notified of the same, via an email, chat application or some other medium. All notifications are sent via actions (which are triggered by status changes).
Incase of an email, it is not a good idea to send an email for every status change. Rather, its better to group the notifications and publish them only once per hour or so. This is achieved using the following mechanism:
- Maintain a local DB table for Email notifications.
- On an action trigger the notification is appended to the DB table.
- An agent executes once per hour and queries the DB table for Email notifications,groups them and sends an Email, followed by deleting the notifications in the DB table.
The DB table was maintained using SQLite, with queries explicitly written, whereas the rest of the DIRAC uses SQLAlchemy + MySQL. Therefore my goal was to change the EmailAction and EmailAgent to use SQLAlchemy and the ResoursceStatusDB for the notification DB table. This has been completed in #4121.
Once the EmailAction was successfuly modified to use ResourceStatusDB, the next task was to provide some more ways to notify users of any resource's status change. Slack is a prominent work chat application. Mattermost is an open source alternative to Slack. In this task I developed the mechanism to post DIRAC resource notification to Slack and Mattermost.
Both of them provide nice integrations using webhooks. Webhooks work by providing a unique URL for your chat room which can be created by the admins. Any message can be sent to the chatroom as a POST query. The following are excellent guides on using webhooks:
This has been completed in #4158.
I implemented performance tests for RSS in Locust, which is a scalable load testing library written in Python. This is the first time DIRAC uses Locust, as opposed to the previously used MultiMechanize. I chose Locust over MultiMechanize because it is supported in Python 3 (DIRAC is moving to Python 3).
Although the performance tests for RSS are complete, I plan to rewrite all existing MultiMechanize tests of the DIRAC framework in Locust. This is currently a work in progress in #4221.
THe tranisition to Python 3 is the current goal across the entire DIRAC framework.
One of my project goals thus was to port RSS from Python 2.7 to Python 3.
Although, due to some dependency issues this was not possible in my GSoC coding period,
in my other tasks, I have taken care of writing Python 3 compatible code as far as possible, while also checking for any future issues using pylint3k code linting.
Currently some DIRAC contributors are working on the dependency issues and when this gets done, I plan to finally transition RSS into Python 3.
- Advanced Python
- Relational databases
- System Design
- Development patterns
- Performance testing