Galaxy (https://galaxyproject.org) is a globally-distributed open source software platform that connects analysis tools, datasets, compute resources, a graphical user interface, and a programmatic API. It enables accessible, reproducible, and collaborative data science and is used by thousands of scientists across the world. It has been in continuous development for 17 years and is among the most active open source projects in the world.
From an engineering standpoint, Galaxy is a large distributed system (400K+ lines of Python code), at the core of which is the data model with objects persisted in a relational database. Our data access layer relies heavily on SQLAlchemy (sqlalchemy.org) - a SQL automation tool kit and object-relational mapper widely-used in the Python ecosystem. SQLAlchemy is about the same age as Galaxy and has recently went through the largest upgrade process in its history. The complexity of upgrading a large system to SQLAlchemy 2.0 is akin to moving to Python 3, only harder. For Galaxy, with thousands of references to the library and extensive use of its features, the migration was a fascinating process of discovery spanning more than two years.
In this poster I will share our experience, summarizing the process we followed, along with the most unexpected, challenging and mind-bending obstacles we encountered along the way, including moving the system from imperative to declarative mapping, rewriting the database migration system, dealing with explicit transactional database access and the evils of autocommit and various implicit magic in a multithreaded context, as well as edge cases that exposed upstream bugs.
Ref: https://github.com/galaxyproject/galaxy, https://usegalaxy.org, https://usegalaxy.eu, https://usegalaxy.org.au, https://training.galaxyproject.org