I've played a lot of Rimworld, for a long time (since early alphas), and certain specific things have remained as a noticeable, impactful burden on the game engine.
Roof Bit Map Updates
From the chunk of Base-64 encoded data in the save file, and the absolutely atrocious performance, I'm guessing the overhead roof tile state is stored in a packed C structure. Access of indexed tiles within this packed structure is expensive, and updating it even more so. The game begins to noticeably take large fractions of a full second per tile update, and when you've got an army of pawns working away, it stutters and stops and rushes forwards, and seemingly stops again, before going, for quite some time. It's dramatic, and roof construction/deconstruction is the single largest in-game performance impactor I encounter, by frequency.
See the video at the end.