rados benchmarks

gistfile1.txt

individual bench per OSD:
All tests run from node1

# ceph tell osd.* bench -f plain
osd.0: bench: wrote 1024 MB in blocks of 4096 KB in 20.138275 sec at 52068 KB/sec
osd.1: bench: wrote 1024 MB in blocks of 4096 KB in 21.206290 sec at 49446 KB/sec
osd.2: bench: wrote 1024 MB in blocks of 4096 KB in 21.361698 sec at 49086 KB/sec
osd.3: bench: wrote 1024 MB in blocks of 4096 KB in 20.538838 sec at 51053 KB/sec
osd.4: bench: wrote 1024 MB in blocks of 4096 KB in 21.490303 sec at 48792 KB/sec
osd.5: bench: wrote 1024 MB in blocks of 4096 KB in 21.130678 sec at 49623 KB/sec
osd.6: bench: wrote 1024 MB in blocks of 4096 KB in 20.805201 sec at 50399 KB/sec
osd.7: bench: wrote 1024 MB in blocks of 4096 KB in 21.149284 sec at 49579 KB/sec
osd.8: bench: wrote 1024 MB in blocks of 4096 KB in 20.688483 sec at 50684 KB/sec
osd.9: bench: wrote 1024 MB in blocks of 4096 KB in 20.705826 sec at 50641 KB/sec
osd.10: bench: wrote 1024 MB in blocks of 4096 KB in 20.936271 sec at 50084 KB/sec
osd.11: bench: wrote 1024 MB in blocks of 4096 KB in 20.614813 sec at 50865 KB/sec
osd.12: bench: wrote 1024 MB in blocks of 4096 KB in 20.336421 sec at 51561 KB/sec
osd.13: bench: wrote 1024 MB in blocks of 4096 KB in 19.460655 sec at 53881 KB/sec
osd.14: bench: wrote 1024 MB in blocks of 4096 KB in 19.225516 sec at 54540 KB/sec
osd.15: bench: wrote 1024 MB in blocks of 4096 KB in 21.962512 sec at 47743 KB/sec
osd.16: bench: wrote 1024 MB in blocks of 4096 KB in 24.682518 sec at 42482 KB/sec <<<
osd.17: bench: wrote 1024 MB in blocks of 4096 KB in 21.428410 sec at 48933 KB/sec
osd.18: bench: wrote 1024 MB in blocks of 4096 KB in 21.151572 sec at 49574 KB/sec
osd.19: bench: wrote 1024 MB in blocks of 4096 KB in 20.857247 sec at 50273 KB/sec

we're getting around 50MBps to each OSD/disk.  Running similar tests with raw dd writes locally on a few nodes shows the same:

osd.0
# dd if=/dev/zero of=big oflag=direct bs=4k count=250000
1024000000 bytes (1.0 GB) copied, 20.058 s, 51.1 MB/s

osd.7
# dd if=/dev/zero of=big oflag=direct bs=4k count=250000
1024000000 bytes (1.0 GB) copied, 20.8377 s, 49.1 MB/s

osd.15
# dd if=/dev/zero of=big oflag=direct bs=4k count=250000
1024000000 bytes (1.0 GB) copied, 21.4443 s, 47.8 MB/s

check the network between the test node and a couple of the target OSD nodes using iperf:

# iperf -c 192.168.100.85  -t 10
------------------------------------------------------------
Client connecting to 192.168.100.85, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[  3] local 192.168.100.86 port 51933 connected with 192.168.100.85 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.10 GBytes   944 Mbits/sec <<

Yes, we do have a GigE network. 944Mbits = ~118MB

rados bench:

# rados -p test bench 300 write -t 32 --no-cleanup
---lots of output---
   301      27      7764      7737   102.801         8    1.0173   1.23732
 Total time run:         301.951585
Total writes made:      7764
Write size:             4194304
Bandwidth (MB/sec):     102.851

Stddev Bandwidth:       31.6943
Max bandwidth (MB/sec): 176
Min bandwidth (MB/sec): 0
Average Latency:        1.2427
Stddev Latency:         0.798934
Max latency:            7.74487
Min latency:            0.209529