Cassandra high CPU / LA

profile.txt

 JvmTop 0.8.0 alpha - 06:51:10,  amd64, 16 cpus, Linux 3.14.44-3, load avg 19.35
 http://code.google.com/p/jvmtop

 Profiling PID 9256: org.apache.cassandra.service.CassandraDa

  95.73% (     4.31s) ....google.common.collect.AbstractIterator.tryToComputeN()
   1.39% (     0.06s) com.google.common.base.Objects.hashCode()
   1.26% (     0.06s) io.netty.channel.epoll.Native.epollWait()
   0.85% (     0.04s) net.jpountz.lz4.LZ4JNI.LZ4_compress_limitedOutput()
   0.46% (     0.02s) net.jpountz.lz4.LZ4JNI.LZ4_decompress_fast()
   0.26% (     0.01s) com.google.common.collect.Iterators$7.computeNext()
   0.06% (     0.00s) io.netty.channel.epoll.Native.eventFdWrite()

ttop.txt

2016-02-12T08:20:25.605+0000 Process summary
  process cpu=1565.15%
  application cpu=1314.48% (user=1354.48% sys=-40.00%)
  other: cpu=250.67%
  heap allocation rate 146mb/s
[000405] user=76.25% sys=-0.54% alloc=     0b/s - SharedPool-Worker-9
[000457] user=75.54% sys=-1.26% alloc=     0b/s - SharedPool-Worker-14
[000451] user=73.52% sys= 0.29% alloc=     0b/s - SharedPool-Worker-16
[000311] user=76.45% sys=-2.99% alloc=     0b/s - SharedPool-Worker-4
[000389] user=70.69% sys= 2.62% alloc=     0b/s - SharedPool-Worker-6
[000388] user=86.95% sys=-14.28% alloc=     0b/s - SharedPool-Worker-5
[000404] user=70.69% sys= 0.10% alloc=     0b/s - SharedPool-Worker-8
[000390] user=72.61% sys=-1.82% alloc=     0b/s - SharedPool-Worker-7
[000255] user=87.86% sys=-17.87% alloc=     0b/s - SharedPool-Worker-1
[000444] user=72.21% sys=-2.30% alloc=     0b/s - SharedPool-Worker-12
[000310] user=71.50% sys=-2.31% alloc=     0b/s - SharedPool-Worker-3
[000445] user=69.68% sys=-0.83% alloc=     0b/s - SharedPool-Worker-13
[000406] user=72.61% sys=-4.40% alloc=     0b/s - SharedPool-Worker-10
[000446] user=69.78% sys=-1.65% alloc=     0b/s - SharedPool-Worker-11
[000452] user=66.86% sys= 0.22% alloc=     0b/s - SharedPool-Worker-15
[000256] user=69.08% sys=-2.42% alloc=     0b/s - SharedPool-Worker-2
[004496] user=29.99% sys= 0.59% alloc=   30mb/s - CompactionExecutor:15
[004906] user=29.49% sys= 0.74% alloc=   39mb/s - CompactionExecutor:16
[010143] user=28.58% sys= 0.25% alloc=   26mb/s - CompactionExecutor:17
[000785] user=27.87% sys= 0.70% alloc=   38mb/s - CompactionExecutor:12
[012723] user= 9.09% sys= 2.46% alloc= 2977kb/s - RMI TCP Connection(2673)-127.0.0.1
[000555] user= 5.35% sys=-0.08% alloc=  474kb/s - SharedPool-Worker-24
[000560] user= 3.94% sys= 0.07% alloc=  434kb/s - SharedPool-Worker-22
[000557] user= 3.94% sys=-0.17% alloc=  339kb/s - SharedPool-Worker-25
[000447] user= 2.73% sys= 0.60% alloc=  436kb/s - SharedPool-Worker-19
[000563] user= 3.33% sys=-0.04% alloc=  460kb/s - SharedPool-Worker-20
[000448] user= 2.73% sys= 0.27% alloc=  414kb/s - SharedPool-Worker-21
[000554] user= 1.72% sys= 0.70% alloc=  232kb/s - SharedPool-Worker-26
[000558] user= 1.41% sys= 0.39% alloc=  213kb/s - SharedPool-Worker-23
[000450] user= 1.41% sys=-0.03% alloc=  158kb/s - SharedPool-Worker-17

ttop2.txt

2016-02-12T08:24:35.630+0000 Process summary
  process cpu=1552.48%
  application cpu=1429.75% (user=890.24% sys=539.50%)
  other: cpu=122.74%
  heap allocation rate 137mb/s
[000446] user=72.79% sys=11.24% alloc=     0b/s - SharedPool-Worker-11
[000405] user=123.53% sys=-42.09% alloc=     0b/s - SharedPool-Worker-9
[000310] user= 0.00% sys=80.93% alloc=     0b/s - SharedPool-Worker-3
[000390] user= 0.00% sys=80.74% alloc=     0b/s - SharedPool-Worker-7
[000457] user=61.42% sys=17.69% alloc=     0b/s - SharedPool-Worker-14
[000388] user= 0.00% sys=79.02% alloc=     0b/s - SharedPool-Worker-5
[000404] user= 0.00% sys=78.46% alloc=     0b/s - SharedPool-Worker-8
[000452] user=84.46% sys=-6.20% alloc=     0b/s - SharedPool-Worker-15
[000445] user=80.08% sys=-1.86% alloc=     0b/s - SharedPool-Worker-13
[000311] user= 0.00% sys=77.56% alloc=     0b/s - SharedPool-Worker-4
[000256] user= 0.00% sys=77.46% alloc=     0b/s - SharedPool-Worker-2
[000451] user=89.51% sys=-13.33% alloc=  885kb/s - SharedPool-Worker-16
[000406] user=95.54% sys=-20.62% alloc=     0b/s - SharedPool-Worker-10
[000444] user=110.99% sys=-36.41% alloc=     0b/s - SharedPool-Worker-12
[000389] user= 0.00% sys=71.51% alloc=     0b/s - SharedPool-Worker-6
[000255] user= 0.00% sys=70.60% alloc=     0b/s - SharedPool-Worker-1
[000785] user=30.71% sys= 0.41% alloc=   30mb/s - CompactionExecutor:12
[010143] user=29.84% sys= 0.50% alloc=   25mb/s - CompactionExecutor:17
[004496] user=28.67% sys= 1.15% alloc=   33mb/s - CompactionExecutor:15
[004906] user=28.38% sys= 0.81% alloc=   38mb/s - CompactionExecutor:16
[012723] user= 9.82% sys= 2.47% alloc= 2866kb/s - RMI TCP Connection(2684)-127.0.0.1
[000550] user= 2.82% sys= 0.31% alloc=  262kb/s - SharedPool-Worker-29
[000553] user= 2.92% sys=-0.08% alloc=  249kb/s - SharedPool-Worker-27
[000554] user= 2.43% sys=-0.13% alloc=  202kb/s - SharedPool-Worker-26
[000558] user= 1.56% sys= 0.52% alloc=  161kb/s - SharedPool-Worker-23
[000555] user= 1.85% sys= 0.21% alloc=  182kb/s - SharedPool-Worker-24
[000449] user= 1.56% sys= 0.40% alloc=  144kb/s - SharedPool-Worker-18
[000533] user= 1.85% sys= 0.03% alloc=  144kb/s - SharedPool-Worker-36
[000557] user= 1.75% sys=-0.06% alloc=  119kb/s - SharedPool-Worker-25
[000447] user= 1.46% sys=-0.01% alloc=   97kb/s - SharedPool-Worker-19