This has updated for 1.20.4, all of the optimization mods mentioned here are compatible with each other and don't affect vanilla behaviour by default.
Lithium - A mod designed to drastically improve the general performance of Minecraft without breaking things.
VMP - A mod designed to improve general server performance at high playercount.