benchmarking_mqtt_brokers.md

Methodology

Benchmark tools used

• EMQ Benchmarking tool

Hardware used

Tests done

Similar benchmarks done by other people:

• http://www.scalagent.com/IMG/pdf/Benchmark_MQTT_servers-v1-1.pdf

• http://rexpie.github.io/2015/08/23/stress-testing-mosquitto.html

• Maximum number of concurrent connections

• Transmission latency with increased concurrent connections

• Message loss (sent using QoS 2) with increased concurrent connections

• Transmission latency with increased size of payload

• Message loss (sent using QoS 2) with increased size of payload

Tuning

Default installations for EMQ, and VerneMQ were pretty non-performant. Used pointers from EMQTT docs, and VerneMQ docs . Kernel variables that might need tuning:

• fs.file-max: Denotes the maximum number of file-handlers that the kernel will allocate
• fs.nr_open: Denotes the maximum number of file-hadlers a process can allocate
• net.core.somaxconn: Need to do more reading on this but related to network connection backlog
• net.ipv4.tcp_max_syn_backlog: Need to do more reading on this but related to network connection backlog
• net.core.netdev_max_backlog: Need to do more reading on this but related to network connection backlog
• net.core.rmem_default: Related to read/write buffers for TCP
• net.core.wmem_default:
• net.core.rmem_max:
• net.core.wmem_max:
• net.core.optmem_max:
• net.ipv4.tcp_rmem:
• net.ipv4.tcp_wmem:
• net.nf_conntrack_max:
• net.netfilter.nf_conntrack_max:
• net.netfilter.nf_conntrack_tcp_timeout_time_wait:
• net.ipv4.tcp_max_tw_buckets:
• net.ipv4.tcp_fin_timeout:

Both VerneMQ, and EMQ recommend tuning the Erlang VM. **** document changes made here

I also tuned a bit the client performing the benchmark tests:

sysctl -w net.ipv4.ip_local_port_range="500 65535"
sysctl -w fs.file-max=1000000

Useful commands

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Tuning_and_Optimizing_Red_Hat_Enterprise_Linux_for_Oracle_9i_and_10g_Databases/chap-Oracle_9i_and_10g_Tuning_Guide-Setting_File_Handles.html To check the max number of file handlers:

cat /proc/sys/fs/file-max

To check current number of open file handlers (three values; total allocated file handlers, number of currently used handlers, maximum file handlers):

cat /proc/sys/fs/file-nr

To change the systems max open file handers while system is still running:

sysctl -w fs.file-max=2097152

To make changes permanent for the above kernel variable:

echo fs.file-max=2097152 >> /etc/sysctl.conf

Notes on tests done

EMQ

Take 1

EMQ installed and all variables set to default. Only changes to the Kernel was to bump up values for fs.file-max=2097152, and fs.nr_open=2097152 since these where the only Kernel variables I was initially aware that needed changing. For the Erlang VM; node.process_limit = 2097152, node.max_ports = 1048576, and for the connection I was using to test (TCP)

listener.tcp.external.acceptors = 64
listener.tcp.external.max_clients = 1000000

After duing the file handler tuning, tried connecting 1000 clients to listen to one topic.

./emqtt_bench_sub -h 192.168.1.105 -p 1883 -c 1000 -q 2 -C false -t test_topic

Ran into the following error when I try to connect more than 992 concurrent connections.

Acceptor on 0.0.0.0:1883 suspend 100ms for 100 emfile errors

Might have been because I was running at 100pc CPU. Bumping up the number of processors from 1 to 2 made no difference. Hmm, did further reading; poor performance might be because I didn't tune the Erlang VM (FML).