Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save bmb/0975dcdd1b63639bf41028f2325bfd2b to your computer and use it in GitHub Desktop.
Save bmb/0975dcdd1b63639bf41028f2325bfd2b to your computer and use it in GitHub Desktop.
Using openib BTL:
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 2
# ( 574 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 0.03 0.00
1 1000 0.12 8.08
2 1000 0.12 16.29
4 1000 0.12 32.32
8 1000 0.12 64.65
16 1000 0.12 126.23
32 1000 0.13 238.36
64 1000 0.13 455.52
128 1000 0.16 773.41
256 1000 0.21 1174.31
512 1000 0.29 1689.77
1024 1000 0.47 2056.22
2048 1000 0.90 2158.63
4096 1000 1.99 1962.86
8192 1000 4.57 1708.80
16384 1000 5.39 2896.23
32768 1000 6.75 4626.29
65536 640 9.29 6729.46
131072 320 14.95 8361.02
262144 160 28.28 8839.88
524288 80 57.04 8765.98
1048576 40 115.90 8628.04
2097152 20 236.39 8460.52
4194304 10 485.49 8239.07
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 4
# ( 572 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 0.08 0.00
1 1000 0.32 8.80
2 1000 0.32 18.00
4 1000 0.28 40.71
8 1000 0.28 81.15
16 1000 0.29 160.00
32 1000 0.29 317.88
64 1000 0.32 579.19
128 1000 0.36 1011.86
256 1000 0.45 1641.90
512 1000 0.63 2339.68
1024 1000 1.03 2855.68
2048 1000 2.07 2833.62
4096 1000 9.74 1203.53
8192 1000 6.71 3492.89
16384 1000 8.70 5384.90
32768 1000 13.28 7058.90
65536 640 22.20 8447.74
131072 320 41.34 9070.89
262144 160 81.71 9179.08
524288 80 162.96 9204.60
1048576 40 326.60 9185.61
2097152 20 659.00 9104.69
4194304 10 1410.91 8505.13
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 8
# ( 568 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 0.19 0.00
1 1000 0.60 11.13
2 1000 0.60 22.21
4 1000 0.61 43.85
8 1000 0.60 88.71
16 1000 0.61 173.98
32 1000 0.63 337.99
64 1000 0.68 624.61
128 1000 0.80 1069.21
256 1000 1.02 1667.36
512 1000 1.46 2344.02
1024 1000 2.49 2745.31
2048 1000 8.19 1669.11
4096 1000 49.80 549.07
8192 1000 19.28 2836.07
16384 1000 23.81 4593.68
32768 1000 39.42 5549.20
65536 640 54.32 8053.86
131072 320 98.95 8842.88
262144 160 192.34 9098.27
524288 80 377.61 9268.74
1048576 40 758.15 9233.04
2097152 20 1553.00 9014.82
4194304 10 3221.20 8692.41
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 16
# ( 560 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 0.36 0.00
1 1000 1.33 10.75
2 1000 1.32 21.63
4 1000 1.34 42.67
8 1000 1.31 87.29
16 1000 1.35 169.91
32 1000 1.41 325.37
64 1000 1.56 586.53
128 1000 1.84 995.72
256 1000 2.45 1492.86
512 1000 3.78 1935.12
1024 1000 9.23 1587.02
2048 1000 37.25 786.49
4096 1000 239.49 244.66
8192 1000 71.11 1647.88
16384 1000 79.77 2938.25
32768 1000 105.71 4434.47
65536 640 145.34 6450.56
131072 320 258.52 7252.94
262144 160 500.87 7486.91
524288 80 958.08 7828.19
1048576 40 1890.80 7933.14
2097152 20 3770.35 7956.82
4194304 10 7635.62 7857.91
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 32
# ( 544 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 0.73 0.00
1 1000 2.80 10.58
2 1000 2.78 21.26
4 1000 2.79 42.43
8 1000 2.79 84.65
16 1000 2.86 165.40
32 1000 3.00 315.35
64 1000 3.29 575.28
128 1000 3.91 967.57
256 1000 5.67 1333.84
512 1000 9.65 1569.08
1024 1000 146.75 206.30
2048 1000 774.39 78.19
4096 1000 1144.70 105.79
8192 1000 486.51 497.81
16384 798 21544.11 22.48
32768 582 434.14 2231.45
65536 582 503.58 3847.45
131072 320 784.35 4940.38
262144 160 1406.82 5508.88
524288 80 2323.45 6671.11
1048576 40 3951.23 7845.66
2097152 20 7830.00 7918.26
4194304 10 15892.20 7802.57
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 64
# ( 512 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 1.54 0.00
1 1000 6.14 9.79
2 1000 6.11 19.66
4 1000 6.12 39.30
8 1000 6.11 78.71
16 1000 6.29 152.81
32 1000 6.60 291.39
64 1000 7.44 516.97
128 1000 9.02 852.70
256 1000 17.32 888.03
512 1000 31.04 991.07
1024 1000 599.25 102.67
2048 1000 4042.93 30.44
4096 263 8141.30 30.23
8192 263 4614.31 106.67
16384 263 65368.00 15.06
32768 263 65224.30 30.18
65536 263 57199.22 68.84
131072 186 92369.23 85.26
262144 93 184379.54 85.42
524288 time-out.; Time limit (secs_per_sample * msg_sizes_list_len) is over; use "-time X" or SECS_PER_SAMPLE=X (IMB_settings.h) to increase time limit.
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 128
# ( 448 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 3.14 0.00
1 1000 12.52 9.67
2 1000 12.81 18.91
4 1000 12.48 38.82
8 1000 12.75 75.98
16 1000 13.08 148.11
32 1000 13.76 281.61
64 1000 14.89 520.65
128 1000 21.87 708.96
256 1000 44.15 702.24
512 1000 70.68 877.35
1024 1000 1437.18 86.30
2048 1000 9873.66 25.12
4096 102 4870.17 101.86
8192 102 5201.18 190.76
16384 102 168529.89 11.77
32768 102 168542.87 23.55
65536 time-out.; Time limit (secs_per_sample * msg_sizes_list_len) is over; use "-time X" or SECS_PER_SAMPLE=X (IMB_settings.h) to increase time limit.
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 256
# ( 320 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 6.60 0.00
1 1000 26.67 9.12
2 1000 25.76 18.88
4 1000 25.64 37.93
8 1000 25.63 75.90
16 1000 26.54 146.59
32 1000 28.57 272.34
64 1000 34.81 447.11
128 1000 52.94 588.04
256 1000 97.38 639.30
512 1000 154.89 803.87
1024 1000 3444.74 72.29
2048 418 7826.21 63.64
4096 58 1679.38 593.13
... <no further progress> ...
--------------------------------------------------------
mlnx5 port counter dump
[1] 00:06:10 [SUCCESS] rccomdc1r52-03-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
388
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
1006903
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
116436
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
435203
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
57555660
[2] 00:06:10 [SUCCESS] rccomdc1r52-02-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
591
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
1763979
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
135945
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
436290
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
75391417
[3] 00:06:10 [SUCCESS] rccomdc1r52-04-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
518
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
1031553
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
162071
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
436489
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
57229603
[4] 00:06:10 [SUCCESS] rccomdc1r52-06-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
253
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
888717
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
135895
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
436708
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
40479719
[5] 00:06:10 [SUCCESS] rccomdc1r52-01-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
174
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
1479751
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
116527
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
2093822
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
124120455
[6] 00:06:10 [SUCCESS] rccomdc1r52-05-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
367
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
2265009
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
138774
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
437650
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
40599850
[7] 00:06:10 [SUCCESS] rccomdc1r53-01-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
804
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
484
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
742
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
1364602
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
230488838
[8] 00:06:10 [SUCCESS] rccomdc1r52-07-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
386
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
2693320
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
158336
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
436984
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
40421544
[9] 00:06:10 [SUCCESS] rccomdc1r53-02-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
805
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
463
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
625
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
1336077
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
180177800
[10] 00:06:10 [SUCCESS] rccomdc1r53-04-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
579
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
205
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
1262290
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
81106780
[11] 00:06:10 [SUCCESS] rccomdc1r53-03-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
133
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
202
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
1256846
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
81241582
[12] 00:06:10 [SUCCESS] rccomdc1r53-05-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
168
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
203
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
1264218
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
81228125
[13] 00:06:10 [SUCCESS] rccomdc1r53-08-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
522
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
210
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
1270030
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
81183189
[14] 00:06:10 [SUCCESS] rccomdc1r53-06-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
466
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
228
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
1264548
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
81427750
[15] 00:06:10 [SUCCESS] rccomdc1r53-07-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
1026
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
215
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
1265309
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
81143131
[16] 00:06:10 [SUCCESS] rccomdc1r52-08-dat.erc.monash.edu.au
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/duplicate_request <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/implied_nak_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/local_ack_timeout_err <==
337
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_buffer <==
2514220
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/out_of_sequence <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/packet_seq_err <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rnr_nak_retry_err <==
60789
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_atomic_requests <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_dct_connect <==
0
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_read_requests <==
438450
==> /sys/class/infiniband/mlx5_0/mlx5_ports/1/counters/rx_write_requests <==
40568368
----------------------------------------------------------------
Using TCP BTL:
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 2
# ( 574 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 0.08 0.00
1 1000 0.34 2.83
2 1000 0.34 5.64
4 1000 0.34 11.32
8 1000 0.40 19.02
16 1000 0.38 40.05
32 1000 0.47 64.52
64 1000 0.79 77.36
128 1000 1.00 121.56
256 1000 1.85 131.69
512 1000 3.07 158.83
1024 1000 6.56 148.80
2048 1000 14.76 132.33
4096 1000 43.94 88.89
8192 1000 6.41 1219.37
16384 1000 6.98 2238.25
32768 1000 8.60 3634.13
65536 640 12.26 5096.20
131072 320 20.88 5988.01
262144 160 41.99 5953.17
524288 80 82.92 6029.55
1048576 40 169.55 5897.92
2097152 20 329.29 6073.64
4194304 10 695.71 5749.56
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 4
# ( 572 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 0.18 0.00
1 1000 0.69 4.17
2 1000 0.63 9.07
4 1000 0.66 17.32
8 1000 0.63 36.27
16 1000 0.71 64.30
32 1000 0.77 118.74
64 1000 0.84 216.70
128 1000 1.23 298.19
256 1000 2.16 338.92
512 1000 3.80 385.47
1024 1000 7.17 408.66
2048 1000 16.59 353.10
4096 1000 55.64 210.61
8192 1000 16.38 1431.02
16384 1000 18.62 2518.13
32768 1000 23.34 4016.22
65536 640 36.65 5116.62
131072 320 65.25 5747.39
262144 160 124.55 6021.68
524288 80 242.67 6181.12
1048576 40 479.60 6255.18
2097152 20 948.70 6324.42
4194304 10 1992.89 6021.40
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 8
# ( 568 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 0.21 0.00
1 1000 0.97 6.92
2 1000 0.95 14.08
4 1000 0.96 27.70
8 1000 0.96 55.40
16 1000 0.89 119.88
32 1000 1.16 184.93
64 1000 1.47 291.43
128 1000 1.94 440.94
256 1000 2.77 617.82
512 1000 5.00 683.61
1024 1000 9.48 721.40
2048 1000 25.19 542.67
4096 1000 102.66 266.34
8192 1000 44.58 1226.84
16384 1000 49.10 2227.37
32768 1000 60.09 3640.31
65536 640 89.71 4876.69
131072 320 154.80 5652.43
262144 160 290.54 6023.18
524288 80 556.14 6293.40
1048576 40 1097.60 6377.54
2097152 20 2246.39 6232.22
4194304 10 4558.90 6141.84
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 16
# ( 560 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 0.40 0.00
1 1000 2.00 7.15
2 1000 2.00 14.34
4 1000 2.01 28.40
8 1000 1.98 57.71
16 1000 2.11 108.38
32 1000 2.43 188.38
64 1000 2.83 324.08
128 1000 3.36 545.42
256 1000 4.28 856.43
512 1000 8.30 882.96
1024 1000 18.49 792.41
2048 1000 60.12 487.32
4096 1000 312.85 187.29
8192 1000 129.43 905.41
16384 1000 139.79 1676.58
32768 1000 167.44 2799.46
65536 640 229.02 4093.61
131072 320 379.32 4943.11
262144 160 715.31 5242.46
524288 80 1367.81 5483.20
1048576 40 2700.48 5554.57
2097152 20 5325.00 5633.81
4194304 10 10694.19 5610.52
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 32
# ( 544 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 0.86 0.00
1 1000 3.76 7.85
2 1000 3.73 15.85
4 1000 3.72 31.78
8 1000 3.75 63.09
16 1000 3.94 120.00
32 1000 4.46 212.26
64 1000 5.04 375.19
128 1000 6.26 604.21
256 1000 10.47 723.02
512 1000 18.11 835.86
1024 1000 81.98 369.28
2048 1000 439.81 137.67
4096 1000 3896.32 31.08
8192 826 9482.30 25.54
16384 742 10562.00 45.86
32768 671 10354.66 93.56
65536 335 15582.10 124.34
131072 320 15315.47 253.01
262144 160 8341.48 929.09
524288 80 6009.53 2579.24
1048576 40 7653.10 4050.65
2097152 20 14733.70 4208.04
4194304 10 29353.71 4224.34
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 64
# ( 512 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 1.79 0.00
1 1000 7.69 7.81
2 1000 7.61 15.79
4 1000 7.60 31.64
8 1000 7.61 63.17
16 1000 8.08 119.02
32 1000 8.86 216.93
64 1000 10.22 376.13
128 1000 12.98 592.57
256 1000 22.53 682.78
512 1000 42.21 728.85
1024 1000 302.19 203.59
2048 1000 2035.23 60.46
4096 507 9525.82 25.83
8192 168 8292.85 59.35
16384 168 10614.21 92.74
32768 168 9770.26 201.50
65536 124 21545.42 182.75
131072 time-out.; Time limit (secs_per_sample * msg_sizes_list_len) is over; use "-time X" or SECS_PER_SAMPLE=X (IMB_settings.h) to increase time limit.
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 128
# ( 448 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 4.05 0.00
1 1000 16.55 7.32
2 1000 16.40 14.77
4 1000 16.38 29.58
8 1000 16.42 59.02
16 1000 17.25 112.31
32 1000 18.81 206.03
64 1000 21.65 358.05
128 1000 27.54 563.00
256 1000 45.92 675.22
512 1000 82.19 754.50
1024 1000 648.92 191.12
2048 1000 4480.11 55.37
4096 274 7296.54 67.99
8192 199 19373.17 51.21
16384 197 19156.92 103.59
32768 197 19651.71 201.95
65536 147 43690.91 181.67
131072 147 44923.47 353.38
262144 147 47100.06 674.10
524288 80 33632.01 1888.08
1048576 40 42391.88 2995.86
2097152 20 79267.95 3204.32
4194304 10 158062.48 3213.92
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 256
# ( 320 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 7.50 0.00
1 1000 30.60 7.95
2 1000 29.16 16.68
4 1000 29.22 33.29
8 1000 29.15 66.75
16 1000 30.82 126.26
32 1000 33.85 229.92
64 1000 40.23 386.88
128 1000 51.45 605.07
256 1000 77.55 802.75
512 1000 165.66 751.60
1024 1000 947.42 262.85
2048 1000 6108.66 81.53
4096 380 20832.48 47.81
8192 196 41785.11 47.68
16384 194 41967.78 94.94
32768 194 42057.32 189.47
65536 134 87913.05 181.29
131072 134 91258.47 349.28
262144 134 94789.34 672.54
524288 80 72568.70 1756.96
1048576 40 92119.43 2768.15
2097152 20 168131.55 3033.34
4194304 10 353418.52 2886.10
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 512
# ( 64 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 14.76 0.00
1 1000 67.19 7.25
2 1000 63.60 15.32
4 1000 63.50 30.70
8 1000 63.65 61.25
16 1000 66.08 118.00
32 1000 72.97 213.71
64 1000 85.87 363.23
128 1000 109.92 567.51
256 1000 165.12 755.55
512 1000 328.50 759.54
1024 1000 2038.14 244.84
2048 1000 12518.90 79.72
4096 389 43649.32 45.73
8192 192 80484.20 49.60
16384 192 83803.16 95.28
32768 192 85793.00 186.13
65536 187 275627.10 115.87
131072 187 357477.68 178.68
262144 time-out.; Time limit (secs_per_sample * msg_sizes_list_len) is over; use "-time X" or SECS_PER_SAMPLE=X (IMB_settings.h) to increase time limit.
#---------------------------------------------------
# Benchmarking One_put_all
# #processes = 576
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 16.56 0.00
1 1000 75.61 7.25
2 1000 74.22 14.78
4 1000 74.15 29.58
8 1000 74.14 59.17
16 1000 77.10 113.79
32 1000 84.10 208.64
64 1000 99.31 353.38
128 1000 127.36 551.11
256 1000 187.88 747.19
512 1000 380.96 736.98
1024 1000 3066.41 183.12
2048 1000 19534.95 57.49
4096 289 48621.77 46.20
8192 162 106560.91 42.16
16384 159 107016.12 83.95
32768 159 106560.65 168.62
65536 159 349116.84 102.94
131072 159 356619.63 201.55
262144 time-out.; Time limit (secs_per_sample * msg_sizes_list_len) is over; use "-time X" or SECS_PER_SAMPLE=X (IMB_settings.h) to increase time limit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment