What happened:
- Detected SSD in an unmounted state. This particular SSD contained PP-related data, so was non-critical.
- In an attempt to repair without any downtime, I force-unmounted this drive.
- This caused MySQL to crash.
- When MySQL crashes, tables of the MyISAM engine require a fair amount of time to recover.
- I have migrated most of osu! away from MyISAM, but a few key tables remain (due to the sheer performance that can be achieved with this engine).
- Scores are part of this data. A full recovery takes around 30 minutes.
- During recovery, MySQL ran out of disk space and crashed, resulting in the repair restarting.
- Total downtime 00:45~01:06 (1 hour, 21 minutes).
What will be done to ensure this doesn't happen again:
- Redirecting all efforts to migrate remaining MyISAM tables to a better engine.
- Allowing slave database servers to be promoted to master (currently not possible due to user permissions restricting WRITE operations).
- Not unmounting disks while MySQL still has open file handles to them in the future. It crashes MySQL ^^;.
Apologies for any inconvenience caused by this downtime. I strive for (as close to) 100% uptime and will continue to do my best to achieve this.
Do your best peppy and get that SSD out of your mouth pippi, here's a cookie for you /me gives a cookie :)