Output of `python repro_bucket_rtts.py --bucket_size=10 --use_cuda_tensors --num_buckets=20`

repro_bucket_rtts.txt

run_worker 1 with world size 2
---Warm Up-----
Callback triggered in 7664.990643 ms
Callback triggered in 7664.933709 ms
Callback triggered in 7664.819333 ms
Callback triggered in 7665.318029 ms
Callback triggered in 7668.967457 ms
Callback triggered in 7673.087738 ms
Callback triggered in 7677.450334 ms
Callback triggered in 7684.611007 ms
Callback triggered in 7687.555784 ms
Callback triggered in 7691.241959 ms
Callback triggered in 7698.109258 ms
Callback triggered in 7705.640672 ms
Callback triggered in 7709.827148 ms
Callback triggered in 7724.6322 ms
Callback triggered in 7732.105365 ms
Callback triggered in 7736.347829 ms
Callback triggered in 7744.576978 ms
Callback triggered in 7752.883221 ms
Callback triggered in 7764.845408 ms
Callback triggered in 7768.758176 ms
-----Run-------
Callback triggered in 14.616113 ms
Callback triggered in 22.788813 ms
Callback triggered in 36.881613 ms
Callback triggered in 41.403386 ms
Callback triggered in 54.535876 ms
Callback triggered in 58.846168 ms
Callback triggered in 66.158814 ms
Callback triggered in 74.629088 ms
Callback triggered in 83.069816 ms
Callback triggered in 91.550962 ms
Callback triggered in 100.549081 ms
Callback triggered in 112.74226 ms
Callback triggered in 112.882627 ms
Callback triggered in 112.801712 ms
Callback triggered in 116.806905 ms
Callback triggered in 120.334562 ms
Callback triggered in 121.259876 ms
Callback triggered in 122.223031 ms
Callback triggered in 123.072511 ms
Callback triggered in 123.864412 ms