Apparently Linux power management can be fairly aggressive and CPU downclocking can have serious effect on PyTorch jobs (even those running on CUDA).
To check the current mode of CPU 0 run the snippet below. On my machine it shows that all cores are in powersave
mode.
for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
[ -f $CPUFREQ ] || continue;
echo $CPUFREQ $(cat $CPUFREQ)
done