batch norm is bad (td3/sac)

batch_norm_test_final.ipynb

Check out properties whose names start with "running_" (either in your batch norm layer or state_dict). They are "learnable", meaning they change under training but not by gradients. They are not present in parameters().

All learnable parameters are in state_dict(). parameters() are only those that are updated by gradients.

My code is not from stable baselines.

Ahh.... So this misunderstanding spread wider than I thought... Maybe there is a chain of misuse and people never bother checking.
When stable-baseline came out there was no such thing as batch norm by the way. The code is great and should indeed be our implement baseline. But we, "the later generations", really have more responsibilities when working on earlier codes.