Skip to content

Instantly share code, notes, and snippets.

@devanshdalal

devanshdalal/leveldb-vs-cassandra-stress-tests.md

Last active December 26, 2017 06:54
Show Gist options
  • Save devanshdalal/0cbfcb6654e8fa3354fe7344dec1e3c2 to your computer and use it in GitHub Desktop.
Save devanshdalal/0cbfcb6654e8fa3354fe7344dec1e3c2 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
leveldb-vs-cassandra-stress-tests.md

Stress-tests

System Config

[genstor@genstor8 apache-cassandra-3.11.1]$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.4.1708 (Core) 
Release:	7.4.1708
Codename:	Core
[genstor@genstor8 apache-cassandra-3.11.1]$ uname -a
Linux genstor8 4.4.91-1.el7.elrepo.x86_64 #1 SMP Sun Oct 8 08:54:00 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

Benchmarks

Leveldb-stress(done with the help of util/leveldb_stress.cpp)

Total payload : 8 GB

-------------------------------------------------
PERF TEST [1024 keys, 8388608 block size]
  SEQUENTIAL WRITE
    payload: 8 GB
    bandwidth: 120.944 MBps
  SEQUENTIAL READ KEYS
    key count: 1024
    time: 0s
    keys/second: 1698
  SEQUENTIAL READ KEY/VALUES
    key count: 1024
    key_hash: 13419855982528054488
    value_hash: 66687336258905910
    time: 37s
    payload: 8GB
    bandwidth: 226.856 MBps

-------------------------------------------------
PERF TEST [8192 keys, 1048576 block size]
  SEQUENTIAL WRITE
    payload: 8 GB
    bandwidth: 120.681 MBps
  SEQUENTIAL READ KEYS
    key count: 8192
    time: 3s
    keys/second: 2402
  SEQUENTIAL READ KEY/VALUES
    key count: 8192
    key_hash: 6261956528663220263
    value_hash: 7400400939896395408
    time: 40s
    payload: 8GB
    bandwidth: 209.926 MBps

-------------------------------------------------
PERF TEST [32768 keys, 262144 block size]
  SEQUENTIAL WRITE
    payload: 8 GB
    bandwidth: 113.296 MBps
  SEQUENTIAL READ KEYS
    key count: 32768
    time: 13s
    keys/second: 2453
  SEQUENTIAL READ KEY/VALUES
    key count: 32768
    key_hash: 4401746994632224256
    value_hash: 14484434486795972475
    time: 49s
    payload: 8GB
    bandwidth: 173.265 MBps

Cassandra

Total payload: 8 GB

----------------------------------------cassandra writes----------------------------------------------

command : ./tools/bin/cassandra-stress write n=128 -node 127.0.0.1 -col size=fixed\(67108864\) n=fixed\(1\) -rate threads=1

Results:
Op rate                   :        1 op/s  [WRITE: 1 op/s]
Partition rate            :        1 pk/s  [WRITE: 1 pk/s]
Row rate                  :        1 row/s [WRITE: 1 row/s]
Latency mean              : 1115.3 ms [WRITE: 1,115.3 ms]
Latency median            : 1054.3 ms [WRITE: 1,054.3 ms]
Latency 95th percentile   : 1723.9 ms [WRITE: 1,723.9 ms]
Latency 99th percentile   : 3745.5 ms [WRITE: 3,745.5 ms]
Latency 99.9th percentile : 4202.7 ms [WRITE: 4,202.7 ms]
Latency max               : 4202.7 ms [WRITE: 4,202.7 ms]
Total partitions          :        128 [WRITE: 128]
Total errors              :          0 [WRITE: 0]
Total GC count            : 34
Total GC memory           : 48.111 GiB
Total GC time             :    3.0 seconds
Avg GC time               :   88.0 ms
StdDev GC time            :   21.7 ms
Total operation time      : 00:02:51

command : ./tools/bin/cassandra-stress read n=256 -node 127.0.0.1 -col size=fixed\(33554432\) n=fixed\(1\) -rate threads=1

Results:
Op rate                   :        2 op/s  [WRITE: 2 op/s]
Partition rate            :        2 pk/s  [WRITE: 2 pk/s]
Row rate                  :        2 row/s [WRITE: 2 row/s]
Latency mean              :  475.9 ms [WRITE: 475.9 ms]
Latency median            :  458.5 ms [WRITE: 458.5 ms]
Latency 95th percentile   :  574.6 ms [WRITE: 574.6 ms]
Latency 99th percentile   :  912.3 ms [WRITE: 912.3 ms]
Latency 99.9th percentile : 1288.7 ms [WRITE: 1,288.7 ms]
Latency max               : 1288.7 ms [WRITE: 1,288.7 ms]
Total partitions          :        256 [WRITE: 256]
Total errors              :          0 [WRITE: 0]
Total GC count            : 35
Total GC memory           : 50.591 GiB
Total GC time             :    2.9 seconds
Avg GC time               :   81.6 ms
StdDev GC time            :   14.9 ms
Total operation time      : 00:02:35

command: ./tools/bin/cassandra-stress write n=1000 -node 127.0.0.1 -col size=fixed\(8388608\) n=fixed\(1\) -rate threads=1 -log level=verbose

Results:
Op rate                   :       11 op/s  [WRITE: 11 op/s]
Partition rate            :       11 pk/s  [WRITE: 11 pk/s]
Row rate                  :       11 row/s [WRITE: 11 row/s]
Latency mean              :   56.5 ms [WRITE: 56.5 ms]
Latency median            :   51.0 ms [WRITE: 51.0 ms]
Latency 95th percentile   :   70.9 ms [WRITE: 70.9 ms]
Latency 99th percentile   :  142.2 ms [WRITE: 142.2 ms]
Latency 99.9th percentile :  775.9 ms [WRITE: 775.9 ms]
Latency max               :  966.3 ms [WRITE: 966.3 ms]
Total partitions          :      1,000 [WRITE: 1,000]
Total errors              :          0 [WRITE: 0]
Total GC count            : 33
Total GC memory           : 47.587 GiB
Total GC time             :    2.8 seconds
Avg GC time               :   85.4 ms
StdDev GC time            :   10.8 ms
Total operation time      : 00:01:26

command: ./tools/bin/cassandra-stress write n=8000 -node 127.0.0.1 -col size=fixed\(1048576\) n=fixed\(1\) -rate threads=1 -log level=verbose

Results:
Results:
Op rate                   :      112 op/s  [WRITE: 112 op/s]
Partition rate            :      112 pk/s  [WRITE: 112 pk/s]
Row rate                  :      112 row/s [WRITE: 112 row/s]
Latency mean              :    6.1 ms [WRITE: 6.1 ms]
Latency median            :    4.9 ms [WRITE: 4.9 ms]
Latency 95th percentile   :    6.9 ms [WRITE: 6.9 ms]
Latency 99th percentile   :   21.7 ms [WRITE: 21.7 ms]
Latency 99.9th percentile :  226.9 ms [WRITE: 226.9 ms]
Latency max               :  768.6 ms [WRITE: 768.6 ms]
Total partitions          :      8,000 [WRITE: 8,000]
Total errors              :          0 [WRITE: 0]
Total GC count            : 32
Total GC memory           : 46.459 GiB
Total GC time             :    2.8 seconds
Avg GC time               :   86.0 ms
StdDev GC time            :   10.1 ms
Total operation time      : 00:01:11

command: ./tools/bin/cassandra-stress write n=32000 -node 127.0.0.1 -col size=fixed\(262144\) n=fixed\(1\) -rate threads=1 -log level=verbose

Results:
Op rate                   :      404 op/s  [WRITE: 404 op/s]
Partition rate            :      404 pk/s  [WRITE: 404 pk/s]
Row rate                  :      404 row/s [WRITE: 404 row/s]
Latency mean              :    1.6 ms [WRITE: 1.6 ms]
Latency median            :    1.4 ms [WRITE: 1.4 ms]
Latency 95th percentile   :    1.8 ms [WRITE: 1.8 ms]
Latency 99th percentile   :    2.5 ms [WRITE: 2.5 ms]
Latency 99.9th percentile :   72.2 ms [WRITE: 72.2 ms]
Latency max               :  715.7 ms [WRITE: 715.7 ms]
Total partitions          :     32,000 [WRITE: 32,000]
Total errors              :          0 [WRITE: 0]
Total GC count            : 13
Total GC memory           : 17.917 GiB
Total GC time             :    1.8 seconds
Avg GC time               :  135.5 ms
StdDev GC time            :   36.9 ms
Total operation time      : 00:01:19

----------------------------------------------cassandra reads----------------------------------------------

command : ./tools/bin/cassandra-stress read n=128 -node 127.0.0.1 -col size=fixed\(67108864\) n=fixed\(1\) -rate threads=1

Results:
Op rate                   :        1 op/s  [READ: 1 op/s]
Partition rate            :        1 pk/s  [READ: 1 pk/s]
Row rate                  :        1 row/s [READ: 1 row/s]
Latency mean              :  691.1 ms [READ: 691.1 ms]
Latency median            :  686.8 ms [READ: 686.8 ms]
Latency 95th percentile   :  735.1 ms [READ: 735.1 ms]
Latency 99th percentile   :  801.1 ms [READ: 801.1 ms]
Latency 99.9th percentile :  814.2 ms [READ: 814.2 ms]
Latency max               :  814.2 ms [READ: 814.2 ms]
Total partitions          :        128 [READ: 128]
Total errors              :          0 [READ: 0]
Total GC count            : 24
Total GC memory           : 37.837 GiB
Total GC time             :    0.7 seconds
Avg GC time               :   28.5 ms
StdDev GC time            :    9.4 ms
Total operation time      : 00:01:49

command : ./tools/bin/cassandra-stress read n=256 -node 127.0.0.1 -col size=fixed\(33554432\) n=fixed\(1\) -rate threads=1

Results:
Op rate                   :        1 op/s  [READ: 1 op/s]
Partition rate            :        1 pk/s  [READ: 1 pk/s]
Row rate                  :        1 row/s [READ: 1 row/s]
Latency mean              : 1224.0 ms [READ: 1,224.0 ms]
Latency median            :  803.7 ms [READ: 803.7 ms]
Latency 95th percentile   : 2592.1 ms [READ: 2,592.1 ms]
Latency 99th percentile   : 3282.0 ms [READ: 3,282.0 ms]
Latency 99.9th percentile : 4676.6 ms [READ: 4,676.6 ms]
Latency max               : 4676.6 ms [READ: 4,676.6 ms]
Total partitions          :        256 [READ: 256]
Total errors              :          0 [READ: 0]
Total GC count            : 57
Total GC memory           : 86.654 GiB
Total GC time             :    3.5 seconds
Avg GC time               :   61.4 ms
StdDev GC time            :    7.8 ms
Total operation time      : 00:05:50

command : ./tools/bin/cassandra-stress read n=1000 -node 127.0.0.1 -col size=fixed\(8388608\) n=fixed\(1\) -rate threads=1 -log level=verbose

Results:
Op rate                   :        7 op/s  [READ: 7 op/s]
Partition rate            :        7 pk/s  [READ: 7 pk/s]
Row rate                  :        7 row/s [READ: 7 row/s]
Latency mean              :  109.3 ms [READ: 109.3 ms]
Latency median            :   80.1 ms [READ: 80.1 ms]
Latency 95th percentile   :  153.5 ms [READ: 153.5 ms]
Latency 99th percentile   : 1162.9 ms [READ: 1,162.9 ms]
Latency 99.9th percentile : 2003.8 ms [READ: 2,003.8 ms]
Latency max               : 2026.9 ms [READ: 2,026.9 ms]
Total partitions          :      1,000 [READ: 1,000]
Total errors              :          0 [READ: 0]
Total GC count            : 50
Total GC memory           : 74.394 GiB
Total GC time             :    3.5 seconds
Avg GC time               :   70.9 ms
StdDev GC time            :    6.4 ms
Total operation time      : 00:02:19

command : ./tools/bin/cassandra-stress read n=8000 -node 127.0.0.1 -col size=fixed\(1048576\) n=fixed\(1\) -rate threads=1 -log level=verbose

Results:
Op rate                   :       58 op/s  [READ: 58 op/s]
Partition rate            :       58 pk/s  [READ: 58 pk/s]
Row rate                  :       58 row/s [READ: 58 row/s]
Latency mean              :   14.8 ms [READ: 14.8 ms]
Latency median            :    9.8 ms [READ: 9.8 ms]
Latency 95th percentile   :   15.9 ms [READ: 15.9 ms]
Latency 99th percentile   :   63.8 ms [READ: 63.8 ms]
Latency 99.9th percentile : 1578.1 ms [READ: 1,578.1 ms]
Latency max               : 2655.0 ms [READ: 2,655.0 ms]
Total partitions          :      8,000 [READ: 8,000]
Total errors              :          0 [READ: 0]
Total GC count            : 52
Total GC memory           : 80.076 GiB
Total GC time             :    3.2 seconds
Avg GC time               :   61.0 ms
StdDev GC time            :    6.4 ms
Total operation time      : 00:02:18

command: ./tools/bin/cassandra-stress read n=32000 -node 127.0.0.1 -col size=fixed\(262144\) n=fixed\(1\) -rate threads=1 -log level=verbose

Results:
Op rate                   :      264 op/s  [READ: 264 op/s]
Partition rate            :      264 pk/s  [READ: 264 pk/s]
Row rate                  :      264 row/s [READ: 264 row/s]
Latency mean              :    3.0 ms [READ: 3.0 ms]
Latency median            :    2.3 ms [READ: 2.3 ms]
Latency 95th percentile   :    4.8 ms [READ: 4.8 ms]
Latency 99th percentile   :    8.8 ms [READ: 8.8 ms]
Latency 99.9th percentile :   45.8 ms [READ: 45.8 ms]
Latency max               :  713.0 ms [READ: 713.0 ms]
Total partitions          :     32,000 [READ: 32,000]
Total errors              :          0 [READ: 0]
Total GC count            : 34
Total GC memory           : 54.011 GiB
Total GC time             :    1.1 seconds
Avg GC time               :   32.4 ms
StdDev GC time            :    9.8 ms
Total operation time      : 00:02:01

summary

SEQUENTIAL wRITE TESTS

bs(all data)        |     256KB(8GB)     |  1MB(8GB)      |  8MB(8GB)       |  32MB(8GB)     | 64MB(8GB)
----------------------------------------------------------------------------------------------------------------
Leveldb(write):     |   113.296 MBps     |  120.681 MBps  |  120.944        |  -             |   -
Cassandra(write)    |   103.424 MBps     |  112 MBps      |  88 MBps        |  67.241 MBps   | 57.38/99.61 Mbps*
----------------------------------------------------------------------------------------------------------
Leveldb(read):      |     173.265 MBps   |  209 MBps      |  226.856 MBps   |  -             |  -
Cassandra(read):    |     66 MBps        |  58 MBps       |  56 Mbps        |  26.14 MBps    | -/92.60 MBps*

* varies too much with runs

Optimization(multiple concurrent read) for cassandra

bs(all data)         |  1MB(8GB)      |
---------------------------------------
write(threads=5):  |  175 MBps      |
write(threads=8):  |  145 MBps      |
reads(threads=4):  |  259 MBps      |
reads(threads=8):  |  382 MBps      |
reads(threads=16): |  361/384 MBps  |

Optimization2

key_cache = 1GB

bs(all data)         |  1MB(8GB)      |
---------------------------------------
cassandra(write):    |  _ MBps      |
cassandra(reads):    |  _ MBps      |

TODO(devansh):

  1. Flushing disk cache before every bm??
  2. Do the tests with varying very big column width(like 64 MB).
  3. What happens with multiple node cluster or multiple thread reads/write in cassandra?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment