Skip to content

Instantly share code, notes, and snippets.

@macros

macros/24 drive hardware raid6

Created April 22, 2011 10:37
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save macros/936429 to your computer and use it in GitHub Desktop.
Save macros/936429 to your computer and use it in GitHub Desktop.
Download ZIP
48 drive x-25m tests
Raw
24 drive hardware raid6
oot@file-s4:/mnt# tiobench --size 196608 --threads 1 --threads 4 --threads 16 --threads 64 --random 20000
Run #1: /usr/bin/tiotest -t 64 -f 3072 -r 312 -b 4096 -d . -T-TT
Unit information
================
File size = megabytes
Blk Size = bytes
Rate = megabytes per second
CPU% = percentage of CPU used during the test
Latency = milliseconds
Lat% = percent of requests that took longer than X seconds
CPU Eff = Rate divided by CPU% - throughput per cpu load
Sequential Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 358.91 23.81% 0.011 10.47 0.00000 0.00000 1507
2.6.38-8-generic 196608 4096 4 1381.76 344.7% 0.011 11.70 0.00000 0.00000 401
2.6.38-8-generic 196608 4096 16 1999.58 1717.% 0.031 10.30 0.00000 0.00000 116
2.6.38-8-generic 196608 4096 64 2022.15 6729.% 0.123 32.57 0.00000 0.00000 30
Random Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 22.82 4.965% 0.171 1.06 0.00000 0.00000 460
2.6.38-8-generic 196608 4096 4 101.99 0% 0.152 0.86 0.00000 0.00000 0
2.6.38-8-generic 196608 4096 16 358.82 3127.% 0.169 0.98 0.00000 0.00000 11
2.6.38-8-generic 196608 4096 64 731.71 22992% 0.317 4.15 0.00000 0.00000 3
Sequential Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 1095.29 97.40% 0.003 340.09 0.00000 0.00000 1124
2.6.38-8-generic 196608 4096 4 1285.60 450.4% 0.011 651.18 0.00000 0.00000 285
2.6.38-8-generic 196608 4096 16 810.38 1206.% 0.074 3546.99 0.00005 0.00000 67
2.6.38-8-generic 196608 4096 64 795.67 5094.% 0.296 5766.58 0.00063 0.00000 16
Random Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 89.83 24.14% 0.003 0.11 0.00000 0.00000 372
2.6.38-8-generic 196608 4096 4 95.09 0% 0.004 4.07 0.00000 0.00000 0
2.6.38-8-generic 196608 4096 16 96.78 0% 0.006 3.79 0.00000 0.00000 0
2.6.38-8-generic 196608 4096 64 97.70 597.4% 0.008 6.50 0.00000 0.00000 16
Raw
24 drive software raid 0
root@file-s4:/mnt# tiobench --size 196608 --threads 1 --threads 4 --threads 16 --threads 64 --random 200000
Run #1: /usr/bin/tiotest -t 64 -f 3072 -r 3125 -b 4096 -d . -T-TT
Unit information
================
File size = megabytes
Blk Size = bytes
Rate = megabytes per second
CPU% = percentage of CPU used during the test
Latency = milliseconds
Lat% = percent of requests that took longer than X seconds
CPU Eff = Rate divided by CPU% - throughput per cpu load
Sequential Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 1928.77 82.84% 0.002 176.84 0.00000 0.00000 2328
2.6.38-8-generic 196608 4096 4 1795.91 285.8% 0.008 105.25 0.00000 0.00000 628
2.6.38-8-generic 196608 4096 16 1803.32 901.7% 0.033 514.53 0.00000 0.00000 200
2.6.38-8-generic 196608 4096 64 1839.61 4348.% 0.128 9869.33 0.00086 0.00000 42
Random Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 24.84 8.076% 0.157 2.06 0.00000 0.00000 308
2.6.38-8-generic 196608 4096 4 96.47 124.0% 0.158 2.74 0.00000 0.00000 78
2.6.38-8-generic 196608 4096 16 344.77 2123.% 0.172 2.81 0.00000 0.00000 16
2.6.38-8-generic 196608 4096 64 672.98 18210% 0.312 7.13 0.00000 0.00000 4
Sequential Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 1026.21 93.55% 0.004 316.07 0.00000 0.00000 1097
2.6.38-8-generic 196608 4096 4 1031.83 357.9% 0.014 346.35 0.00000 0.00000 288
2.6.38-8-generic 196608 4096 16 1034.63 1600.% 0.058 1347.84 0.00000 0.00000 65
2.6.38-8-generic 196608 4096 64 1037.76 6887.% 0.228 2747.09 0.00004 0.00000 15
Random Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 354.48 64.43% 0.003 0.03 0.00000 0.00000 550
2.6.38-8-generic 196608 4096 4 502.28 0% 0.004 22.39 0.00000 0.00000 0
2.6.38-8-generic 196608 4096 16 545.84 0% 0.006 14.43 0.00000 0.00000 0
2.6.38-8-generic 196608 4096 64 552.45 0% 0.024 98.83 0.00000 0.00000 0
Raw
8 drive software raid 0
root@file-s4:/mnt# tiobench --size 196608 --threads 1 --threads 4 --threads 16 --threads 64 --random 20000
Run #1:Run #1: /usr/bin/tiotest -t 64 -f 3072 -r 312 -b 4096 -d . -T-T
Unit information
================
File size = megabytes
Blk Size = bytes
Rate = megabytes per second
CPU% = percentage of CPU used during the test
Latency = milliseconds
Lat% = percent of requests that took longer than X seconds
CPU Eff = Rate divided by CPU% - throughput per cpu load
Sequential Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 1896.71 85.04% 0.002 12.25 0.00000 0.00000 2230
2.6.38-8-generic 196608 4096 4 1672.34 220.0% 0.009 37.24 0.00000 0.00000 760
2.6.38-8-generic 196608 4096 16 1719.56 787.9% 0.035 148.22 0.00000 0.00000 218
2.6.38-8-generic 196608 4096 64 1679.57 3461.% 0.145 6512.71 0.00007 0.00000 49
Random Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 21.21 6.516% 0.184 1.88 0.00000 0.00000 326
2.6.38-8-generic 196608 4096 4 95.74 124.9% 0.155 2.61 0.00000 0.00000 77
2.6.38-8-generic 196608 4096 16 324.86 2253.% 0.178 2.77 0.00000 0.00000 14
2.6.38-8-generic 196608 4096 64 699.11 24289% 0.323 3.26 0.00000 0.00000 3
Sequential Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 784.39 70.64% 0.005 456.68 0.00000 0.00000 1110
2.6.38-8-generic 196608 4096 4 795.53 281.1% 0.019 557.14 0.00000 0.00000 283
2.6.38-8-generic 196608 4096 16 823.82 1218.% 0.072 1446.92 0.00000 0.00000 68
2.6.38-8-generic 196608 4096 64 798.08 4997.% 0.298 3347.30 0.00012 0.00000 16
Random Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 339.79 26.09% 0.003 0.14 0.00000 0.00000 1302
2.6.38-8-generic 196608 4096 4 181.37 0% 0.003 0.20 0.00000 0.00000 0
2.6.38-8-generic 196608 4096 16 553.49 0% 0.006 4.35 0.00000 0.00000 0
2.6.38-8-generic 196608 4096 64 568.47 0% 0.009 6.07 0.00000 0.00000 0
Raw
Final config, 2x24 raid6 with lvm
root@file-s4:/raid# tiobench --size 196608 --threads 1 --threads 4 --threads 16 --threads 64 --random 20000
Run #1: /usr/bin/tiotest -t 64 -f 3072 -r 312 -b 4096 -d . -T-TT
Unit information
================
File size = megabytes
Blk Size = bytes
Rate = megabytes per second
CPU% = percentage of CPU used during the test
Latency = milliseconds
Lat% = percent of requests that took longer than X seconds
CPU Eff = Rate divided by CPU% - throughput per cpu load
Sequential Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 467.17 30.20% 0.008 8.26 0.00000 0.00000 1547
2.6.38-8-generic 196608 4096 4 1517.18 387.8% 0.009 8.57 0.00000 0.00000 391
2.6.38-8-generic 196608 4096 16 3251.97 3105.% 0.018 9.06 0.00000 0.00000 105
2.6.38-8-generic 196608 4096 64 3836.70 15097% 0.065 14.23 0.00000 0.00000 25
Random Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 26.96 7.592% 0.144 1.01 0.00000 0.00000 355
2.6.38-8-generic 196608 4096 4 106.13 0% 0.145 1.00 0.00000 0.00000 0
2.6.38-8-generic 196608 4096 16 346.71 2218.% 0.154 1.06 0.00000 0.00000 16
2.6.38-8-generic 196608 4096 64 690.75 20129% 0.276 10.00 0.00000 0.00000 3
Sequential Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 1075.84 96.92% 0.004 250.55 0.00000 0.00000 1110
2.6.38-8-generic 196608 4096 4 1416.49 492.3% 0.010 945.10 0.00000 0.00000 288
2.6.38-8-generic 196608 4096 16 1409.19 2368.% 0.043 2156.23 0.00000 0.00000 59
2.6.38-8-generic 196608 4096 64 1558.84 12111% 0.153 7176.08 0.00027 0.00000 13
Random Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 81.86 23.05% 0.003 0.11 0.00000 0.00000 355
2.6.38-8-generic 196608 4096 4 95.05 0% 0.003 0.21 0.00000 0.00000 0
2.6.38-8-generic 196608 4096 16 154.47 0% 0.008 6.74 0.00000 0.00000 0
2.6.38-8-generic 196608 4096 64 144.67 740.0% 0.008 6.16 0.00000 0.00000 20
Raw
raid 0+0 ext4
root@file-s4:/mnt# tiobench --size 196608 --threads 1 --threads 4 --threads 16 --threads 64 --random 20000
Run #1: /usr/bin/tiotest -t 64 -f 3072 -r 312 -b 4096 -d . -T-TT
Unit information
================
File size = megabytes
Blk Size = bytes
Rate = megabytes per second
CPU% = percentage of CPU used during the test
Latency = milliseconds
Lat% = percent of requests that took longer than X seconds
CPU Eff = Rate divided by CPU% - throughput per cpu load
Sequential Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 1414.72 60.58% 0.003 367.82 0.00000 0.00000 2335
2.6.38-8-generic 196608 4096 4 3234.35 772.3% 0.005 260.76 0.00000 0.00000 419
2.6.38-8-generic 196608 4096 16 3890.28 3768.% 0.016 87.90 0.00000 0.00000 103
2.6.38-8-generic 196608 4096 64 4092.81 16518% 0.060 249.78 0.00000 0.00000 25
Random Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 25.60 8.520% 0.152 1.60 0.00000 0.00000 300
2.6.38-8-generic 196608 4096 4 113.42 95.81% 0.132 1.96 0.00000 0.00000 118
2.6.38-8-generic 196608 4096 16 405.59 166.1% 0.149 2.07 0.00000 0.00000 244
2.6.38-8-generic 196608 4096 64 956.00 1078.% 0.224 9.10 0.00000 0.00000 89
Sequential Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 835.18 88.35% 0.005 500.46 0.00000 0.00000 945
2.6.38-8-generic 196608 4096 4 1557.56 954.7% 0.009 1244.29 0.00000 0.00000 163
2.6.38-8-generic 196608 4096 16 1365.75 7682.% 0.042 2467.30 0.00002 0.00000 18
2.6.38-8-generic 196608 4096 64 1152.34 37976% 0.202 8123.48 0.00271 0.00000 3
Random Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 106.37 47.65% 0.003 0.09 0.00000 0.00000 223
2.6.38-8-generic 196608 4096 4 125.94 178.9% 0.004 0.18 0.00000 0.00000 70
2.6.38-8-generic 196608 4096 16 68.42 77.94% 0.007 1.03 0.00000 0.00000 88
2.6.38-8-generic 196608 4096 64 55.75 1404.% 0.013 8.18 0.00000 0.00000 4
Raw
raid 0+0 xfs
root@file-s4:/mnt# tiobench --size 196608 --threads 1 --threads 4 --threads 16 --threads 64 --random 20000
Run #1: /usr/bin/tiotest -t 64 -f 3072 -r 312 -b 4096 -d . -T-TT
Unit information
================
File size = megabytes
Blk Size = bytes
Rate = megabytes per second
CPU% = percentage of CPU used during the test
Latency = milliseconds
Lat% = percent of requests that took longer than X seconds
CPU Eff = Rate divided by CPU% - throughput per cpu load
Sequential Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 1180.82 54.16% 0.003 8.66 0.00000 0.00000 2180
2.6.38-8-generic 196608 4096 4 3074.87 738.3% 0.005 8.80 0.00000 0.00000 416
2.6.38-8-generic 196608 4096 16 3830.90 3237.% 0.016 40.95 0.00000 0.00000 118
2.6.38-8-generic 196608 4096 64 4109.27 11623% 0.060 107.60 0.00000 0.00000 35
Random Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 25.99 9.315% 0.150 1.56 0.00000 0.00000 279
2.6.38-8-generic 196608 4096 4 112.36 5.752% 0.136 1.99 0.00000 0.00000 1953
2.6.38-8-generic 196608 4096 16 419.76 204.1% 0.145 2.09 0.00000 0.00000 206
2.6.38-8-generic 196608 4096 64 946.37 30041% 0.225 11.33 0.00000 0.00000 3
Sequential Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 1084.80 98.30% 0.003 0.72 0.00000 0.00000 1103
2.6.38-8-generic 196608 4096 4 1798.55 630.9% 0.008 453.91 0.00000 0.00000 285
2.6.38-8-generic 196608 4096 16 1349.05 2278.% 0.045 1746.53 0.00000 0.00000 59
2.6.38-8-generic 196608 4096 64 1051.33 7846.% 0.229 3276.67 0.00013 0.00000 13
Random Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 113.17 27.52% 0.003 0.12 0.00000 0.00000 411
2.6.38-8-generic 196608 4096 4 100.25 0% 0.004 5.26 0.00000 0.00000 0
2.6.38-8-generic 196608 4096 16 89.31 0% 0.006 4.13 0.00000 0.00000 0
2.6.38-8-generic 196608 4096 64 94.32 425.6% 0.007 4.08 0.00000 0.00000 22
Raw
raid 0+0 xfs write through, drive cache, 64KB stripe, no read ahead
Run #1: /usr/bin/tiotest -t 64 -f 3072 -r 312 -b 4096 -d . -T-TT
Unit information
================
File size = megabytes
Blk Size = bytes
Rate = megabytes per second
CPU% = percentage of CPU used during the test
Latency = milliseconds
Lat% = percent of requests that took longer than X seconds
CPU Eff = Rate divided by CPU% - throughput per cpu load
Sequential Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 920.81 42.01% 0.004 11.13 0.00000 0.00000 2192
2.6.38-8-generic 196608 4096 4 3731.27 963.7% 0.004 18.46 0.00000 0.00000 387
2.6.38-8-generic 196608 4096 16 3856.89 2742.% 0.016 29.24 0.00000 0.00000 141
2.6.38-8-generic 196608 4096 64 3868.02 8893.% 0.064 340.05 0.00000 0.00000 43
Random Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 21.77 8.358% 0.179 1.10 0.00000 0.00000 260
2.6.38-8-generic 196608 4096 4 101.42 1.298% 0.149 3.15 0.00000 0.00000 7813
2.6.38-8-generic 196608 4096 16 377.87 3545.% 0.159 3.54 0.00000 0.00000 11
2.6.38-8-generic 196608 4096 64 872.39 24326% 0.252 9.82 0.00000 0.00000 4
Sequential Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 1123.77 99.55% 0.003 0.81 0.00000 0.00000 1129
2.6.38-8-generic 196608 4096 4 2901.86 1030.% 0.005 1046.95 0.00000 0.00000 282
2.6.38-8-generic 196608 4096 16 2833.56 5411.% 0.021 1544.79 0.00000 0.00000 52
2.6.38-8-generic 196608 4096 64 2788.82 26169% 0.085 1953.44 0.00000 0.00000 11
Random Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- -------- ------ --------- ----------- -------- -------- -----
2.6.38-8-generic 196608 4096 1 339.47 73.86% 0.003 0.12 0.00000 0.00000 460
2.6.38-8-generic 196608 4096 4 501.90 0% 0.004 3.55 0.00000 0.00000 0
2.6.38-8-generic 196608 4096 16 551.85 0% 0.007 6.54 0.00000 0.00000 0
2.6.38-8-generic 196608 4096 64 554.96 1921.% 0.006 3.73 0.00000 0.00000 29
root@file-s4:/mnt#
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment