Kafka performance Java 8 versus Java 9

gistfile1.txt

I did a quick test on my laptop and produce throughput is significantly higher with Java 9 (2.5x for TLS with stronger encryption and 1.5x for Plaintext). The former is due to AES-GCM being optimised via CPU intrinsics. The latter is less clear, but one theory is that it's due to CRC32C using a CPU intrinsic in Java 9.

./bin/kafka-producer-perf-test.sh --producer.config config/producer.properties --topic=test3 --num-records=100000 --record-size=100000 --throughput=1000000000

Java 8 SSL TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (auto-chosen)
5192 records sent, 1038.4 records/sec (99.03 MB/sec), 323.5 ms avg latency, 368.0 max latency.

Java 8 Plaintext
18900 records sent, 3780.0 records/sec (360.49 MB/sec), 88.7 ms avg latency, 127.0 max latency.

Java 9 SSL TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (auto-chosen, note the stronger encryption)
13563 records sent, 2712.6 records/sec (258.69 MB/sec), 123.9 ms avg latency, 154.0 max latency.

Java 9 PLAINTEXT
29250 records sent, 5850.0 records/sec (557.90 MB/sec), 57.4 ms avg latency, 93.0 max latency.

The numbers for the consumer are similar.

./bin/kafka-consumer-perf-test.sh --topic test2 --consumer.config config/consumer.properties --messages 100000000 --broker-list localhost:9092

Java 8 SSL TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (auto-chosen)
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2017-09-07 19:08:57:390, 2017-09-07 19:12:19:834, 17118.1679, 84.5575, 179497, 886.6501

Java 8 Plaintext
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2017-09-07 19:25:51:112, 2017-09-07 19:26:37:384, 17118.1679, 369.9466, 179497, 3879.1710

Java 9 SSL TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (auto-chosen, note the stronger encryption)
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2017-09-07 19:06:19:943, 2017-09-07 19:07:40:603, 17118.1679, 212.2262, 179497, 2225.3533

Java 9 PLAINTEXT
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2017-09-07 19:05:03:914, 2017-09-07 19:05:45:582, 17118.1679, 410.8229, 179497, 4307.7902