Skip to content

Instantly share code, notes, and snippets.

@devanshdalal

devanshdalal/cassandra-prod-stress.md

Last active December 28, 2017 09:25
Show Gist options
  • Save devanshdalal/85d30dd3b762deefb4a847ee48c4e5eb to your computer and use it in GitHub Desktop.
Save devanshdalal/85d30dd3b762deefb4a847ee48c4e5eb to your computer and use it in GitHub Desktop.
Download ZIP
Raw
cassandra-prod-stress.md

Stress-tests

System Config

Distributor ID: Ubuntu
Description:    Ubuntu 12.04.5 LTS
Release:        12.04
Codename:       precise
admin@s006:~/GitTest/apache-cassandra-3.11.1$ uname -a
Linux s006 3.11.0-15-generic #25~precise1-Ubuntu SMP Thu Jan 30 17:39:31 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Local:192.168.2.187
Remote:192.168.2.190

Benchmarks

Total data load in one cycle: 8 GB

Remote To Local

--------------------------------------------------Writes-----------------------------------------------------------

./tools/bin/cassandra-stress write n=8192 -node 192.168.2.187 -col size=fixed\(1048576\) n=fixed\(1\) -rate threads=1 -port native=19042 thrift=19160 jmx=17199

Results:Results:
Op rate                   :       51 op/s  [WRITE: 51 op/s]
Partition rate            :       51 pk/s  [WRITE: 51 pk/s]
Row rate                  :       51 row/s [WRITE: 51 row/s]
Latency mean              :   17.3 ms [WRITE: 17.3 ms]
Latency median            :    4.8 ms [WRITE: 4.8 ms]
Latency 95th percentile   :    8.2 ms [WRITE: 8.2 ms]
Latency 99th percentile   :   13.2 ms [WRITE: 13.2 ms]
Latency 99.9th percentile : 2910.8 ms [WRITE: 2,910.8 ms]
Latency max               : 18555.6 ms [WRITE: 18,555.6 ms]
Total partitions          :      8,192 [WRITE: 8,192]
Total errors              :          0 [WRITE: 0]
Total GC count            : 0
Total GC memory           : 0.000 KiB
Total GC time             :    0.0 seconds
Avg GC time               :    NaN ms
StdDev GC time            :    0.0 ms
Total operation time      : 00:02:39

./tools/bin/cassandra-stress write n=1024 -node 192.168.2.187 -col size=fixed\(8388608\) n=fixed\(1\) -rate threads=1 -port native=19042 thrift=19160 jmx=17199
Results:
Op rate                   :        4 op/s  [WRITE: 4 op/s]
Partition rate            :        4 pk/s  [WRITE: 4 pk/s]
Row rate                  :        4 row/s [WRITE: 4 row/s]
Latency mean              :  254.1 ms [WRITE: 254.1 ms]
Latency median            :   49.4 ms [WRITE: 49.4 ms]
Latency 95th percentile   :   86.6 ms [WRITE: 86.6 ms]
Latency 99th percentile   : 4479.5 ms [WRITE: 4,479.5 ms]
Latency 99.9th percentile : 26508.0 ms [WRITE: 26,508.0 ms]
Latency max               : 31910.3 ms [WRITE: 31,910.3 ms]
Total partitions          :      1,024 [WRITE: 1,024]
Total errors              :          0 [WRITE: 0]
Total GC count            : 0
Total GC memory           : 0.000 KiB
Total GC time             :    0.0 seconds
Avg GC time               :    NaN ms
StdDev GC time            :    0.0 ms
Total operation time      : 00:04:41

-----------------------------------------------Reads---------------------------------------------------------

./tools/bin/cassandra-stress read n=8192 -node 192.168.2.187 -col size=fixed\(1048576\) n=fixed\(1\) -rate threads=1 -port native=19042 thrift=19160 jmx=17199
Results:
Op rate                   :      102 op/s  [READ: 102 op/s]
Partition rate            :      102 pk/s  [READ: 102 pk/s]
Row rate                  :      102 row/s [READ: 102 row/s]
Latency mean              :    6.5 ms [READ: 6.5 ms]
Latency median            :    6.4 ms [READ: 6.4 ms]
Latency 95th percentile   :    8.3 ms [READ: 8.3 ms]
Latency 99th percentile   :    9.5 ms [READ: 9.5 ms]
Latency 99.9th percentile :   44.4 ms [READ: 44.4 ms]
Latency max               :  433.6 ms [READ: 433.6 ms]
Total partitions          :      8,192 [READ: 8,192]
Total errors              :          0 [READ: 0]
Total GC count            : 0
Total GC memory           : 0.000 KiB
Total GC time             :    0.0 seconds
Avg GC time               :    NaN ms
StdDev GC time            :    0.0 ms
Total operation time      : 00:01:20

./tools/bin/cassandra-stress read n=1024 -node 192.168.2.187 -col size=fixed\(8388608\) n=fixed\(1\) -rate threads=1 -port native=19042 thrift=19160 jmx=17199
Results:
Op rate                   :       15 op/s  [READ: 15 op/s]
Partition rate            :       15 pk/s  [READ: 15 pk/s]
Row rate                  :       15 row/s [READ: 15 row/s]
Latency mean              :   48.1 ms [READ: 48.1 ms]
Latency median            :   46.1 ms [READ: 46.1 ms]
Latency 95th percentile   :   59.2 ms [READ: 59.2 ms]
Latency 99th percentile   :   90.7 ms [READ: 90.7 ms]
Latency 99.9th percentile :  128.8 ms [READ: 128.8 ms]
Latency max               :  197.0 ms [READ: 197.0 ms]
Total partitions          :      1,024 [READ: 1,024]
Total errors              :          0 [READ: 0]
Total GC count            : 0
Total GC memory           : 0.000 KiB
Total GC time             :    0.0 seconds
Avg GC time               :    NaN ms
StdDev GC time            :    0.0 ms
Total operation time      : 00:01:06

Local To Local

--------------------------------------------------Writes-----------------------------------------------------------

./tools/bin/cassandra-stress write n=8192 -node 192.168.2.187 -col size=fixed\(1048576\) n=fixed\(1\) -rate threads=1 -port native=19042 thrift=19160 jmx=17199
Results:
Op rate                   :       53 op/s  [WRITE: 53 op/s]
Partition rate            :       53 pk/s  [WRITE: 53 pk/s]
Row rate                  :       53 row/s [WRITE: 53 row/s]
Latency mean              :   16.5 ms [WRITE: 16.5 ms]
Latency median            :    4.7 ms [WRITE: 4.7 ms]
Latency 95th percentile   :    8.9 ms [WRITE: 8.9 ms]
Latency 99th percentile   :    9.8 ms [WRITE: 9.8 ms]
Latency 99.9th percentile : 3575.6 ms [WRITE: 3,575.6 ms]
Latency max               : 20971.5 ms [WRITE: 20,971.5 ms]
Total partitions          :      8,192 [WRITE: 8,192]
Total errors              :          0 [WRITE: 0]
Total GC count            : 34
Total GC memory           : 48.588 GiB
Total GC time             :    2.4 seconds
Avg GC time               :   69.3 ms
StdDev GC time            :   26.8 ms
Total operation time      : 00:02:35


./tools/bin/cassandra-stress write n=1024 -node 192.168.2.187 -col size=fixed\(8388608\) n=fixed\(1\) -rate threads=1 -port native=19042 thrift=19160 jmx=17199
Results:
Op rate                   :        4 op/s  [WRITE: 4 op/s]
Partition rate            :        4 pk/s  [WRITE: 4 pk/s]
Row rate                  :        4 row/s [WRITE: 4 row/s]
Latency mean              :  223.4 ms [WRITE: 223.4 ms]
Latency median            :   44.1 ms [WRITE: 44.1 ms]
Latency 95th percentile   :   82.5 ms [WRITE: 82.5 ms]
Latency 99th percentile   : 7201.6 ms [WRITE: 7,201.6 ms]
Latency 99.9th percentile : 22246.6 ms [WRITE: 22,246.6 ms]
Latency max               : 26524.8 ms [WRITE: 26,524.8 ms]
Total partitions          :      1,024 [WRITE: 1,024]
Total errors              :          0 [WRITE: 0]
Total GC count            : 36
Total GC memory           : 50.966 GiB
Total GC time             :    1.7 seconds
Avg GC time               :   46.2 ms
StdDev GC time            :    9.9 ms
Total operation time      : 00:04:08

-----------------------------------------------Reads---------------------------------------------------------

./tools/bin/cassandra-stress read n=8192 -node 192.168.2.187 -col size=fixed\(1048576\) n=fixed\(1\) -rate threads=1 -port native=19042 thrift=19160 jmx=17199
Results:
Op rate                   :      100 op/s  [READ: 100 op/s]
Partition rate            :      100 pk/s  [READ: 100 pk/s]
Row rate                  :      100 row/s [READ: 100 row/s]
Latency mean              :    6.2 ms [READ: 6.2 ms]
Latency median            :    6.6 ms [READ: 6.6 ms]
Latency 95th percentile   :    8.1 ms [READ: 8.1 ms]
Latency 99th percentile   :    9.1 ms [READ: 9.1 ms]
Latency 99.9th percentile :   33.8 ms [READ: 33.8 ms]
Latency max               :  119.6 ms [READ: 119.6 ms]
Total partitions          :      8,192 [READ: 8,192]
Total errors              :          0 [READ: 0]
Total GC count            : 27
Total GC memory           : 43.189 GiB
Total GC time             :    0.5 seconds
Avg GC time               :   19.2 ms
StdDev GC time            :    7.3 ms
Total operation time      : 00:01:21


./tools/bin/cassandra-stress read n=1024 -node 192.168.2.187 -col size=fixed\(8388608\) n=fixed\(1\) -rate threads=1 -port native=19042 thrift=19160 jmx=17199

summary

SEQUENTIAL WRITE TESTS


batch size=1 MB
test type           |   remote to local  |  local to local |
------------------------------------------------------------
Cassandra(write)    |         51MBps     |    53MBps  
Cassandra(read)     |         102MBps    |    100MBps  

batch size=8 MB
test type           |   remote to local  |  local to local |
------------------------------------------------------------
Cassandra(write)    |        112MBps     |    112MBps  
Cassandra(read)     |      166.32MBps    |    200.50MBps
  • I got com.datastax.driver.core.exceptions.OperationTimedOutException during writes reapeatedly, due to which writes seem slower. All writes were in general as fast as reads.

TODO(devansh):

  1. Whats the numbers for a distributed cassandra database.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment