Both systems are Raspberry 4 having exactly the same HW configuration, living on the same network. The first is Alpine Linux with OpenRC, the second if Ubuntu with Systemd. Both of the systems run OpenSSH with the same configuration.
➜ ~ time ssh -A 10.10.10.41 'exit 0'
________________________________________________________
Executed in 239.94 millis fish external
usr time 12.44 millis 67.00 micros 12.37 millis
sys time 9.36 millis 448.00 micros 8.91 millis
➜ ~ time ssh -A ubuntu@10.10.10.40 'exit 0'
________________________________________________________
Executed in 2.26 secs fish external
usr time 10.15 millis 92.00 micros 10.06 millis
sys time 9.89 millis 570.00 micros 9.32 millis