Debugging score import failures when running score spreadsheet via `diffcalc-sheet-generator`

score-import-fail.md

Debugging score import failures when running score spreadsheet via diffcalc-sheet-generator

This guide is intended to tell you what to do when you see one of these:

diffcalc-sheet-generator-importer-1   | System.AggregateException: One or more errors occurred. (Processing legacy score 157224218 failed. (Total score conversion operation returned invalid total of -2344))
diffcalc-sheet-generator-importer-1   |  ---> System.AggregateException: Processing legacy score 157224218 failed. (Total score conversion operation returned invalid total of -2344)
diffcalc-sheet-generator-importer-1   |  ---> System.InvalidOperationException: Total score conversion operation returned invalid total of -2344
diffcalc-sheet-generator-importer-1   |    at osu.Game.Database.StandardisedScoreMigrationTools.convertFromLegacyTotalScore(ScoreInfo score, LegacyBeatmapConversionDifficultyInfo difficulty, LegacyScoreAttributes attributes) in /tmp/tmp.tBMkcXfjq4/osu/osu.Game/Database/StandardisedScoreMigrationTools.cs:line 298
diffcalc-sheet-generator-importer-1   |    at osu.Server.Queues.ScoreStatisticsProcessor.Commands.Queue.BatchInserter.CreateReferenceScore(Ruleset ruleset, HighScore highScore, MySqlConnection connection, MySqlTransaction transaction) in /tmp/tmp.tBMkcXfjq4/osu-queue-score-statistics/osu.Server.Queues.ScoreStatisticsProcessor/Commands/Queue/BatchInserter.cs:line 387
diffcalc-sheet-generator-importer-1   |    at osu.Server.Queues.ScoreStatisticsProcessor.Commands.Queue.BatchInserter.run(HighScore[] scores) in /tmp/tmp.tBMkcXfjq4/osu-queue-score-statistics/osu.Server.Queues.ScoreStatisticsProcessor/Commands/Queue/BatchInserter.cs:line 178
diffcalc-sheet-generator-importer-1   |    --- End of inner exception stack trace ---
diffcalc-sheet-generator-importer-1   |    at osu.Server.Queues.ScoreStatisticsProcessor.Commands.Queue.BatchInserter.run(HighScore[] scores) in /tmp/tmp.tBMkcXfjq4/osu-queue-score-statistics/osu.Server.Queues.ScoreStatisticsProcessor/Commands/Queue/BatchInserter.cs:line 225
diffcalc-sheet-generator-importer-1   |    --- End of inner exception stack trace ---
diffcalc-sheet-generator-importer-1 exited with code 255

This means that your score conversion code is broken. If you are lucky, the happy path is to go to the score page for the given ID on the osu! website - so, in the above example, as I was running catch, it would be https://osu.ppy.sh/scores/fruits/157224218.

However, unfortunately, in this case, there is no replay. What do?

The best idea is to run the score import yourself with the debugger attached. But this means you'll have to do it outside of docker and it's not very clear how to do that. Thus, this guide.

1. In the docker compose output, locate a line like such:

   diffcalc-sheet-generator-importer-1   | Waiting for database 05d9aeefbc9830185f8c28f3a6bca5f0ec0ffea4 to reach step 1. Currently at step 1.
   

   This is important as it gives you your db name to connect to.

2. In the diffcalc-sheet-generator directory, run docker compose ps. Note down the ports for the database and for the redis instance:

   NAME                                   IMAGE                                COMMAND                                               SERVICE     CREATED        STATUS                    PORTS
   diffcalc-sheet-generator-db-1          mysql:8.0                            "docker-entrypoint.sh --innodb-buffer-pool-size=4G"   db          22 hours ago   Up 31 minutes (healthy)   33060/tcp, 0.0.0.0:32771->3306/tcp, :::32771->3306/tcp
   diffcalc-sheet-generator-generator-1   diffcalc-sheet-generator-generator   "./start.sh"                                          generator   22 hours ago   Up 30 minutes             
   diffcalc-sheet-generator-ppcalc-1      diffcalc-sheet-generator-ppcalc      "./start.sh"                                          ppcalc      22 hours ago   Up 30 minutes             
   diffcalc-sheet-generator-redis-1       redis:latest                         "docker-entrypoint.sh redis-server"                   redis       46 hours ago   Up 31 minutes (healthy)   0.0.0.0:32770->6379/tcp, :::32770->6379/tcp
   

   In this instance the db is exposed on port 32771, and the redis server on port 32770.

3. Connect to the database. You can use a mysql shell on your local machine for that, not necessarily in a container, but you will need a magic incantation for this:

   $ mysql -sN --host=localhost --port=32771 --protocol=tcp --user=root --database=05d9aeefbc9830185f8c28f3a6bca5f0ec0ffea4

   The port and protocol specs are especially important. The db name matches value from step 1.

   After that, nuke the partial reimport for a clean slate.

   mysql> TRUNCATE solo_scores_legacy_id_map;
   mysql> TRUNCATE solo_scores;
   mysql> TRUNCATE solo_scores_performance;
   

4. Open osu-queue-score-statistics and run ./UseLocalOsu.sh in the root of that repository so that the score importer uses your broken code.

5. In the run configuration, set:

   • Program arguments: queue import-high-scores --ruleset-id=$YOUR_RULESET_ID
   • Environment variables:

      DB_NAME=05d9aeefbc9830185f8c28f3a6bca5f0ec0ffea4
      DB_PORT=32771
      REDIS_HOST=localhost:32770
     

     The db name must match the value from point 1.

6. Run debug, breakpoint at the point your code breaks, et voila.