confluent kafka perf testing notes

1_kafka_perf_test_info.txt

references:
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
https://gist.github.com/jkreps/c7ddb4041ef62a900e6c


from a fresh download/untar of confluent platform:

create topics:
bin/kafka-topics --zookeeper localhost:2181 --create --topic test-rep-one --partitions 6 --replication-factor 1
bin/kafka-topics --zookeeper localhost:2181 --create --topic test --partitions 6 --replication-factor 3

run tests:
bin/kafka-producer-perf-test --topic test-rep-one --num-records 50000000 --record-size 100 --throughput 50000000 --producer.config etc/kafka/producer.properties

bin/kafka-producer-perf-test --topic test-rep-one --num-recos 50000000 --record-size 10000 --throughput 10000 --producer.config etc/kafka/producer.properties

2_producer.properties

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see kafka.producer.ProducerConfig for more details

############################# Producer Basics #############################

# list of brokers used for bootstrapping knowledge about the rest of the cluster
# format: host1:port1,host2:port2 ...
bootstrap.servers=localhost:9092

# specify the compression codec for all data generated: none, gzip, snappy, lz4
compression.type=lz4

# name of the partitioner class for partitioning events; default partition spreads data randomly
#partitioner.class=

# the maximum amount of time the client will wait for the response of a request
#request.timeout.ms=

# how long `KafkaProducer.send` and `KafkaProducer.partitionsFor` will block for
#max.block.ms=

# the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together
#linger.ms=

# the maximum size of a request in bytes
#max.request.size=

# the default batch size in bytes when batching multiple records sent to a partition
#batch.size=

# the total bytes of memory the producer can use to buffer records waiting to be sent to the server
#buffer.memory=

max.block.ms=5000
linger.ms=5
max.in.flight.requests.per.connection=1
retries=5
batch.size=100000
max.request.size=10000000
acks=all

# these are the values jay kreps used in his perf testing years ago
buffer.memory=67108864

3_misc_results.txt

single broker on a random macbook prod laptop:

with default producer settings:
[sferguson@GURGI confluent-3.1.1 -]$ bin/kafka-producer-perf-test --topic test-rep-one --num-records 50000000 --record-size 100 --throughput 50000000 --producer.config etc/kafka/producer.properties
1257890 records sent, 251527.7 records/sec (23.99 MB/sec), 48.3 ms avg latency, 490.0 max latency.
1572930 records sent, 314334.5 records/sec (29.98 MB/sec), 11.3 ms avg latency, 127.0 max latency.
2222460 records sent, 444492.0 records/sec (42.39 MB/sec), 10.6 ms avg latency, 172.0 max latency.
2091893 records sent, 418378.6 records/sec (39.90 MB/sec), 4.1 ms avg latency, 68.0 max latency.
4258792 records sent, 839170.8 records/sec (80.03 MB/sec), 3.0 ms avg latency, 154.0 max latency.
5017814 records sent, 1003562.8 records/sec (95.71 MB/sec), 4.5 ms avg latency, 168.0 max latency.
4702674 records sent, 940534.8 records/sec (89.70 MB/sec), 3.0 ms avg latency, 67.0 max latency.
5031558 records sent, 1006311.6 records/sec (95.97 MB/sec), 5.4 ms avg latency, 88.0 max latency.
4816019 records sent, 963203.8 records/sec (91.86 MB/sec), 3.0 ms avg latency, 16.0 max latency.
5850168 records sent, 1170033.6 records/sec (111.58 MB/sec), 3.1 ms avg latency, 12.0 max latency.
5699234 records sent, 1139846.8 records/sec (108.70 MB/sec), 2.8 ms avg latency, 15.0 max latency.
acti3527388 records sent, 705477.6 records/sec (67.28 MB/sec), 4.0 ms avg latency, 172.0 max latency.
3204558 records sent, 640911.6 records/sec (61.12 MB/sec), 7.0 ms avg latency, 130.0 max latency.
50000000 records sent, 759462.907832 records/sec (72.43 MB/sec), 5.48 ms avg latency, 490.00 ms max latency, 3 ms 50th, 9 ms 95th, 71 ms 99th, 431 ms 99.9th.

with our producer settings:
[sferguson@GURGI confluent-3.1.1 -]$ bin/kafka-producer-perf-test --topic test-rep-one --num-records 50000000 --record-size 100 --throughput 50000000 --producer.config etc/kafka/producer.properties
[2016-11-30 11:13:18,805] WARN metadata.fetch.timeout.ms config is deprecated and will be removed soon. Please use max.block.ms (org.apache.kafka.clients.producer.KafkaProducer)
3565358 records sent, 713071.6 records/sec (68.00 MB/sec), 80.9 ms avg latency, 513.0 max latency.
7768086 records sent, 1553306.5 records/sec (148.13 MB/sec), 7.9 ms avg latency, 39.0 max latency.
7590468 records sent, 1517790.0 records/sec (144.75 MB/sec), 7.7 ms avg latency, 32.0 max latency.
7321524 records sent, 1464012.0 records/sec (139.62 MB/sec), 9.2 ms avg latency, 72.0 max latency.
7099728 records sent, 1419945.6 records/sec (135.42 MB/sec), 8.4 ms avg latency, 160.0 max latency.
7693408 records sent, 1538681.6 records/sec (146.74 MB/sec), 7.5 ms avg latency, 74.0 max latency.
8029898 records sent, 1605658.5 records/sec (153.13 MB/sec), 7.7 ms avg latency, 49.0 max latency.
50000000 records sent, 1407023.863125 records/sec (134.18 MB/sec), 13.21 ms avg latency, 513.00 ms max latency, 7 ms 50th, 15 ms 95th, 316 ms 99th, 474 ms 99.9th.