Create a gist now

Instantly share code, notes, and snippets.

G1GC / CMS

G1GC v.s. CMS for Cassandra 2.0

Many people have asked me about using the G1 garbage collector with Cassandra. Since most of my customers are running 2.0 in production the test is with 2.0 for now. Once I script it up I'll re-run the numbers with 2.1.

I also need to re-test with Java 7 and the Oracle JDKs.

TL;DR

G1 with "other" settings (see below) can beat CMS on latency and throughput on anemic CPUs if you set -XX:MaxGCPauseMillis=1000.

Settings Op/s latency mean latency median latency p95 latency p99 latency p99.9 latency max
G1+other 74379 2.7 1.7 4.9 7.5 21.7 2901.2
G1-other 72116 2.8 1.7 5.0 7.7 24.3 1088.2
CMS+other 65681 3.0 1.5 4.7 7.7 276.9 1848.0
CMS-other 62196 3.2 1.6 5.1 8.3 185.8 1615.3
CMS-other 64973 3.1 1.5 4.7 7.4 127.6 1645.7

Hardware / Software

  • 6x Intel NUC 16GB RAM, quad-core @ 1.3Ghz, SSD
  • Zulu OpenJDK 8
  • Ubuntu 14.04 LTS Docker image (tobert/cassandra:2.0.13 w/ zulu openjdk)
  • CoreOS 607.0 kernel 3.18.6
  • /data is on an Intel mSATA SSD using ext4

cassandra-stress

cassandra-stress \
                write \
                n=50000000 \
                cl=ONE \
                -mode native cql3 \
                -node 192.168.10.12

Other Settings

In the course of tuning for these workloads I've experimented on and off with the following settings. I'll need to break down which is providing what benefit later.

-XX:+AlwaysPreTouch -XX:+ResizeTLAB -XX:-UseBiasedLocking

stress logs

(http://tobert.org/logs/cms-2Gnew-with-other.log.gz) (http://tobert.org/logs/cms-2Gnew-without-other.log.gz) (http://tobert.org/logs/g1-1000ms-with-other.log.gz) (http://tobert.org/logs/g1-1000ms-without-other.log.gz)

-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=60 -XX:MaxGCPauseMillis=1000 -XX:+AlwaysPreTouch -XX:+ResizeTLAB -XX:-UseBiasedLocking
op rate : 74379 [WRITE:74379]
partition rate : 74379 [WRITE:74379]
row rate : 74379 [WRITE:74379]
latency mean : 2.7 [WRITE:2.7]
latency median : 1.7 [WRITE:1.7]
latency 95th percentile : 4.9 [WRITE:4.9]
latency 99th percentile : 7.5 [WRITE:7.5]
latency 99.9th percentile : 21.7 [WRITE:21.7]
latency max : 2901.2 [WRITE:2901.2]
Total partitions : 50000000 [WRITE:50000000]
Total errors : 0 [WRITE:0]
total gc count : 0
total gc mb : 0
total gc time (s) : 0
avg gc time(ms) : NaN
stdev gc time(ms) : 0
Total operation time : 00:11:12
-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=60 -XX:MaxGCPauseMillis=1000
op rate : 72116 [WRITE:72116]
partition rate : 72116 [WRITE:72116]
row rate : 72116 [WRITE:72116]
latency mean : 2.8 [WRITE:2.8]
latency median : 1.7 [WRITE:1.7]
latency 95th percentile : 5.0 [WRITE:5.0]
latency 99th percentile : 7.7 [WRITE:7.7]
latency 99.9th percentile : 24.3 [WRITE:24.3]
latency max : 1088.2 [WRITE:1088.2]
Total partitions : 50000000 [WRITE:50000000]
Total errors : 0 [WRITE:0]
total gc count : 0
total gc mb : 0
total gc time (s) : 0
avg gc time(ms) : NaN
stdev gc time(ms) : 0
Total operation time : 00:11:33
END
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC
Run 2: I wanted to add links to the stress logs and lost the one from above so I ran again and got better numbers:
op rate : 64973 [WRITE:64973]
partition rate : 64973 [WRITE:64973]
row rate : 64973 [WRITE:64973]
latency mean : 3.1 [WRITE:3.1]
latency median : 1.5 [WRITE:1.5]
latency 95th percentile : 4.7 [WRITE:4.7]
latency 99th percentile : 7.4 [WRITE:7.4]
latency 99.9th percentile : 127.6 [WRITE:127.6]
latency max : 1645.7 [WRITE:1645.7]
Total partitions : 50000000 [WRITE:50000000]
Run 1:
op rate : 62196 [WRITE:62196]
partition rate : 62196 [WRITE:62196]
row rate : 62196 [WRITE:62196]
latency mean : 3.2 [WRITE:3.2]
latency median : 1.6 [WRITE:1.6]
latency 95th percentile : 5.1 [WRITE:5.1]
latency 99th percentile : 8.3 [WRITE:8.3]
latency 99.9th percentile : 185.8 [WRITE:185.8]
latency max : 1615.3 [WRITE:1615.3]
Total partitions : 50000000 [WRITE:50000000]
Total errors : 0 [WRITE:0]
total gc count : 0
total gc mb : 0
total gc time (s) : 0
avg gc time(ms) : NaN
stdev gc time(ms) : 0
Total operation time : 00:13:23
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+AlwaysPreTouch -XX:+ResizeTLAB -XX:-UseBiasedLocking
op rate : 65681 [WRITE:65681]
partition rate : 65681 [WRITE:65681]
row rate : 65681 [WRITE:65681]
latency mean : 3.0 [WRITE:3.0]
latency median : 1.5 [WRITE:1.5]
latency 95th percentile : 4.7 [WRITE:4.7]
latency 99th percentile : 7.7 [WRITE:7.7]
latency 99.9th percentile : 276.8 [WRITE:276.8]
latency max : 1848.0 [WRITE:1848.0]
Total partitions : 50000000 [WRITE:50000000]
Total errors : 0 [WRITE:0]
total gc count : 0
total gc mb : 0
total gc time (s) : 0
avg gc time(ms) : NaN
stdev gc time(ms) : 0
Total operation time : 00:12:41
# G1GC settings for Cassandra 2.0 (or 2.1)
chdir: /data
stdin: /dev/null
stdout: /data/log/console.log
stderr: /data/log/console.log
uid: 1337
gid: 1337
argv:
- /usr/bin/java
- -ea
- -javaagent:{{ glob "" "/opt/cassandra/lib/jamm-*.jar" }}
- -Xmx8G
- -Xms8G
- -Xss256k
- -XX:+UseG1GC # use garbage-first collection
- -XX:InitiatingHeapOccupancyPercent=60 # don't do mixed mode until the heap is this full
- -XX:MaxGCPauseMillis=1000 # default is 200ms
- -XX:+AlwaysPreTouch # allocate and zero (force fault) heap memory on startup
- -XX:+UseTLAB # thread local allocation blocks
- -XX:+ResizeTLAB # auto-optimize TLAB size https://blogs.oracle.com/jonthecollector/entry/the_real_thing
- -XX:-UseBiasedLocking # disable biased locking for cassandra
- -XX:StringTableSize=1000003
- -XX:CompileCommandFile=/data/conf/hotspot_compiler
- -Djava.net.preferIPv4Stack=true
- -Dcom.sun.management.jmxremote.port=7199
- -Dcom.sun.management.jmxremote.rmi.port=7199
- -Dcom.sun.management.jmxremote.ssl=false
- -Dcom.sun.management.jmxremote.authenticate=false
- -Dlog4j.defaultInitOverride=true
- -Dlog4j.configuration=log4j-server.properties # cassandra <= 2.0
- -Dlogback.configurationFile=logback.xml # cassandra >= 2.1
- -Dcassandra.logdir=/data/log
- -Dcassandra-foreground=yes
- -XX:+HeapDumpOnOutOfMemoryError
- -XX:+PrintGCDetails
- -XX:+PrintAdaptiveSizePolicy
- -XX:+PrintGCApplicationStoppedTime
- -XX:+PrintPromotionFailure
- -Xloggc:/data/log/gc.log
- -XX:+UseGCLogFileRotation
- -XX:NumberOfGCLogFiles=10
- -XX:GCLogFileSize=10M
- -cp
- {{ glob ":" "/data/conf" "/data/lib" "/opt/cassandra/lib/*.jar" }}
- org.apache.cassandra.service.CassandraDaemon
{{ range .ExtraArgs }} - {{ . }}
{{ end }}
# G1GC without unrelated settings
chdir: /data
stdin: /dev/null
stdout: /data/log/console.log
stderr: /data/log/console.log
uid: 1337
gid: 1337
argv:
- /usr/bin/java
- -ea
- -javaagent:{{ glob "" "/opt/cassandra/lib/jamm-*.jar" }}
- -Xmx8G
- -Xms8G
- -Xss256k
- -XX:+UseG1GC # use garbage-first collection
- -XX:InitiatingHeapOccupancyPercent=60 # don't do mixed mode until the heap is this full
- -XX:MaxGCPauseMillis=1000 # default is 200ms
- -XX:+UseTLAB # thread local allocation blocks
- -XX:StringTableSize=1000003
- -XX:CompileCommandFile=/data/conf/hotspot_compiler
- -Djava.net.preferIPv4Stack=true
- -Dcom.sun.management.jmxremote.port=7199
- -Dcom.sun.management.jmxremote.rmi.port=7199
- -Dcom.sun.management.jmxremote.ssl=false
- -Dcom.sun.management.jmxremote.authenticate=false
- -Dlog4j.defaultInitOverride=true
- -Dlog4j.configuration=log4j-server.properties # cassandra <= 2.0
- -Dlogback.configurationFile=logback.xml # cassandra >= 2.1
- -Dcassandra.logdir=/data/log
- -Dcassandra-foreground=yes
- -XX:+HeapDumpOnOutOfMemoryError
- -XX:+PrintGCDetails
- -XX:+PrintAdaptiveSizePolicy
- -XX:+PrintGCApplicationStoppedTime
- -XX:+PrintPromotionFailure
- -Xloggc:/data/log/gc.log
- -XX:+UseGCLogFileRotation
- -XX:NumberOfGCLogFiles=10
- -XX:GCLogFileSize=10M
- -cp
- {{ glob ":" "/data/conf" "/data/lib" "/opt/cassandra/lib/*.jar" }}
- org.apache.cassandra.service.CassandraDaemon
{{ range .ExtraArgs }} - {{ . }}
{{ end }}
# default Cassandra 2.0 CMS settings with the 3 other settings from the
# G1 settings that might make a difference
chdir: {{ .DataDir }}
stdin: /dev/null
stdout: /data/log/console.log
stderr: /data/log/console.log
uid: 1337
gid: 1337
argv:
- /usr/bin/java
- -ea
- -javaagent:{{ glob "" "/opt/cassandra/lib/jamm-*.jar" }}
- -XX:+CMSClassUnloadingEnabled
- -XX:+UseThreadPriorities
- -XX:ThreadPriorityPolicy=42
- -Xmx8G
- -Xms8G
- -Xmn2G
- -XX:+HeapDumpOnOutOfMemoryError
- -Xss256k
- -XX:+AlwaysPreTouch # allocate and zero (force fault) heap memory on startup
- -XX:+UseTLAB # thread local allocation blocks
- -XX:+ResizeTLAB # auto-optimize https://blogs.oracle.com/jonthecollector/entry/the_real_thing
- -XX:-UseBiasedLocking # disable biased locking for cassandra
- -XX:StringTableSize=1000003
- -XX:+UseParNewGC
- -XX:+UseConcMarkSweepGC
- -XX:+CMSParallelRemarkEnabled
- -XX:SurvivorRatio=8
- -XX:MaxTenuringThreshold=1
- -XX:CMSInitiatingOccupancyFraction=75
- -XX:+UseCMSInitiatingOccupancyOnly
- -XX:CompileCommandFile={{ .ConfDir }}/hotspot_compiler
- -XX:+UseCondCardMark
- -XX:+CMSParallelInitialMarkEnabled
- -XX:+CMSEdenChunksRecordAlways
- -Djava.net.preferIPv4Stack=true
- -Dcom.sun.management.jmxremote.port={{ .JmxPort }}
- -Dcom.sun.management.jmxremote.rmi.port={{ .JmxPort }}
- -Dcom.sun.management.jmxremote.ssl=false
- -Dcom.sun.management.jmxremote.authenticate=false
- -Dlog4j.configuration=log4j-server.properties
- -Dlog4j.defaultInitOverride=true
- -Dcassandra.logdir={{ .LogDir }}
- -Dcassandra-foreground=yes
- -cp
- {{ glob ":" .ConfDir .LibDir "/opt/cassandra/lib/*.jar" }}
- org.apache.cassandra.service.CassandraDaemon
{{ range .ExtraArgs }} - {{ . }}
{{ end }}
# default Cassandra 2.0 settings
chdir: {{ .DataDir }}
stdin: /dev/null
stdout: /data/log/console.log
stderr: /data/log/console.log
uid: 1337
gid: 1337
argv:
- /usr/bin/java
- -ea
- -javaagent:{{ glob "" "/opt/cassandra/lib/jamm-*.jar" }}
- -XX:+CMSClassUnloadingEnabled
- -XX:+UseThreadPriorities
- -XX:ThreadPriorityPolicy=42
- -Xmx8G
- -Xms8G
- -Xmn2G
- -XX:+HeapDumpOnOutOfMemoryError
- -Xss256k
- -XX:StringTableSize=1000003
- -XX:+UseParNewGC
- -XX:+UseConcMarkSweepGC
- -XX:+CMSParallelRemarkEnabled
- -XX:SurvivorRatio=8
- -XX:MaxTenuringThreshold=1
- -XX:CMSInitiatingOccupancyFraction=75
- -XX:+UseCMSInitiatingOccupancyOnly
- -XX:+UseTLAB
- -XX:CompileCommandFile={{ .ConfDir }}/hotspot_compiler
- -XX:+UseCondCardMark
- -XX:+CMSParallelInitialMarkEnabled
- -XX:+CMSEdenChunksRecordAlways
- -Djava.net.preferIPv4Stack=true
- -Dcom.sun.management.jmxremote.port={{ .JmxPort }}
- -Dcom.sun.management.jmxremote.rmi.port={{ .JmxPort }}
- -Dcom.sun.management.jmxremote.ssl=false
- -Dcom.sun.management.jmxremote.authenticate=false
- -Dlog4j.configuration=log4j-server.properties
- -Dlog4j.defaultInitOverride=true
- -Dcassandra.logdir={{ .LogDir }}
- -Dcassandra-foreground=yes
- -cp
- {{ glob ":" .ConfDir .LibDir "/opt/cassandra/lib/*.jar" }}
- org.apache.cassandra.service.CassandraDaemon
{{ range .ExtraArgs }} - {{ . }}
{{ end }}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment