mmmries/README.md

## README.md

      
    Raw
  

              README.md
            
          
    I wanted to run another round of performance benchmarks for gnat to see how it's request throughput has changed with the introduction of the ConsumerSupervisor which handles things like processing each request in its own supervised process.
I used a CPU-optimized digital ocean droplet with 16 cores, gnatsd 1.3.0, erlang 21.2.2 and elixir 1.8.0.rc0
You can read the setup instructions below for more details and the results_by_concurrency.md contains details about a lot of different runs.
I'm trying to measure the overhead in the system, so the requests are random byte strings that just get echoed back with processing.
The measurements use byte strings of 4 bytes up to 1024 bytes.
TL/DR; You can do 170k+ synchronous requests/sec of small messages or 192MB+/sec of 1kb messages


## results_by_concurrency.md

      
    Raw
  

              results_by_concurrency.md
            
          
    Gnat Running on Client Machine


client conns
client requesters
server conns
server handlers
message size
req/sec
kb/sec
median
90th %


4
512
4
4
16
93379
2918.1
4103
9972


4
512
8
8
16
93439.82188033953
2919.9944337606103
4269
8743


4
2048
8
8
16
97498.7314572397
3046.8353580387407
19507
25478


4
2048
8
16
16
95559.2278357125
2986.2258698660157
19851
25849


4
2048
16
32
16
95202.40097852077
2975.075030578774
20076
25387


Gnat Running on Server Machine


client conns
client requesters
server conns
server handlers
message size
req/sec
kb/sec
median
90th %


4
1024
8
8
16
108550.8953699757
3392.2154803117405
8335
12124


8
1024
8
8
16
140868.9800177902
4402.1556255559435
6035
11156


8
1024
16
16
16
149550.08281116767
4673.44008784899
5484
10432


8
2048
16
16
16
156817.07885004688
4900.533714063965
10882
18424


16
2048
16
16
16
34329.764239170705
1072.8051324740845
58830
68563


12
2048
16
16
16
50866.57721633351
1589.5805380104223
47780
55717


10
2048
16
16
16
100257.3843993929
3133.0432624810283
14339
43591


8
2048
16
16
16
153525.29842791744
4797.66557587242
11048
18747


6
1024
16
16
16
144757.3148564192
4523.6660892631
5760
9701


8
1024
16
16
16
146443.59617277386
4576.362380399183
5549
10740


8
4096
16
16
16
168454.1685600282
5264.192767500881
20770
30994


8
4096
16
16
256
148215.92498334651
74107.96249167326
24703
35386


8
4096
16
16
1024
98581.15579329274
197162.31158658548
39292
59472


8
4096
16
16
4
170606.2436220505
1332.8612782972696
20249
31148


8
4096
8
8
4
168135.7403836393
1313.560471747182
21324
33085


8
6000
8
8
4
165030.80341203389
1289.3031516565147
31588
45289


8
6000
4
4
4
129725.25596144305
1013.4785621987738
36778
93851


Median and 90th % are reported in microseconds of latency to send request and receive parsed response


## setup_instructions.md

      
    Raw
  

              setup_instructions.md
            
          
    # apt-get install build-essential unzip libssl-dev automake autoconf libncurses5-dev 
# git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.6.0
# echo -e '\n. $HOME/.asdf/asdf.sh' >> ~/.bashrc
# echo -e '\n. $HOME/.asdf/completions/asdf.bash' >> ~/.bashrc
# source ~/.bashrc
# asdf plugin-add erlang
# asdf plugin-add elixir
# asdf install erlang 21.2.2
# asdf global erlang 21.2.2
# asdf install elixir 1.8.0-rc.0
# asdf global elixir 1.8.0-rc.0
# wget https://github.com/nats-io/gnatsd/releases/download/v1.3.0/gnatsd-v1.3.0-linux-amd64.zip
# unzip gnatsd-v1.3.0-linux-amd64.zip
# ./gnatsd-v1.3.0-linux-amd64/gnatsd &
# git clone https://github.com/mmmries/gnat.git
# cd gnat
# mix do deps.get, compile
# mix run bench/server.exs # on the server machine
# mix run bench/client.exs # on the client machine


## straight_line_benchmarks.txt
root@ubuntu-c-16-sfo2-01:~/gnat# mix run bench/parse.exs
Operating System: Linux"
CPU Information: Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
Number of Available Cores: 16
Available memory: 31.42 GB
Elixir 1.8.0
Erlang 21.2.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 0 μs
parallel: 1
inputs: 1024 byte, 128 byte, 16 byte
Estimated total run time: 36 s


Benchmarking parsec with input 1024 byte...
Benchmarking parsec with input 128 byte...
Benchmarking parsec with input 16 byte...

##### With input 1024 byte #####
Name             ips        average  deviation         median         99th %
parsec      438.92 K        2.28 μs  ±1748.29%           2 μs           4 μs

##### With input 128 byte #####
Name             ips        average  deviation         median         99th %
parsec      606.19 K        1.65 μs  ±2093.53%           1 μs           3 μs

##### With input 16 byte #####
Name             ips        average  deviation         median         99th %
parsec      639.57 K        1.56 μs  ±2452.74%           1 μs           3 μs
root@ubuntu-c-16-sfo2-01:~/gnat# mix run bench/publish.exs
Operating System: Linux"
CPU Information: Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
Number of Available Cores: 16
Available memory: 31.42 GB
Elixir 1.8.0
Erlang 21.2.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 0 μs
parallel: 1
inputs: 1024 byte, 128 byte, 16 byte
Estimated total run time: 36 s


Benchmarking pub with input 1024 byte...
Benchmarking pub with input 128 byte...
Benchmarking pub with input 16 byte...

##### With input 1024 byte #####
Name           ips        average  deviation         median         99th %
pub       225.62 K        4.43 μs   ±589.95%           4 μs           9 μs

##### With input 128 byte #####
Name           ips        average  deviation         median         99th %
pub       158.10 K        6.33 μs   ±311.64%           5 μs          11 μs

##### With input 16 byte #####
Name           ips        average  deviation         median         99th %
pub       162.02 K        6.17 μs   ±324.82%           5 μs          12 μs
root@ubuntu-c-16-sfo2-01:~/gnat# mix run bench/request.exs
Operating System: Linux"
CPU Information: Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
Number of Available Cores: 16
Available memory: 31.42 GB
Elixir 1.8.0
Erlang 21.2.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 0 μs
parallel: 1
inputs: 1024 byte, 128 byte, 16 byte
Estimated total run time: 36 s


Benchmarking request with input 1024 byte...
Benchmarking request with input 128 byte...
Benchmarking request with input 16 byte...

##### With input 1024 byte #####
Name              ips        average  deviation         median         99th %
request        8.81 K      113.50 μs    ±38.27%         111 μs         150 μs

##### With input 128 byte #####
Name              ips        average  deviation         median         99th %
request        8.72 K      114.68 μs    ±14.35%         112 μs      154.33 μs

##### With input 16 byte #####
Name              ips        average  deviation         median         99th %
request        8.81 K      113.54 μs    ±15.65%         111 μs         150 μs
client conns	client requesters	server conns	server handlers	message size	req/sec	kb/sec	median	90th %
4	512	4	4	16	93379	2918.1	4103	9972
4	512	8	8	16	93439.82188033953	2919.9944337606103	4269	8743
4	2048	8	8	16	97498.7314572397	3046.8353580387407	19507	25478
4	2048	8	16	16	95559.2278357125	2986.2258698660157	19851	25849
4	2048	16	32	16	95202.40097852077	2975.075030578774	20076	25387
client conns	client requesters	server conns	server handlers	message size	req/sec	kb/sec	median	90th %
4	1024	8	8	16	108550.8953699757	3392.2154803117405	8335	12124
8	1024	8	8	16	140868.9800177902	4402.1556255559435	6035	11156
8	1024	16	16	16	149550.08281116767	4673.44008784899	5484	10432
8	2048	16	16	16	156817.07885004688	4900.533714063965	10882	18424
16	2048	16	16	16	34329.764239170705	1072.8051324740845	58830	68563
12	2048	16	16	16	50866.57721633351	1589.5805380104223	47780	55717
10	2048	16	16	16	100257.3843993929	3133.0432624810283	14339	43591
8	2048	16	16	16	153525.29842791744	4797.66557587242	11048	18747
6	1024	16	16	16	144757.3148564192	4523.6660892631	5760	9701
8	1024	16	16	16	146443.59617277386	4576.362380399183	5549	10740
8	4096	16	16	16	168454.1685600282	5264.192767500881	20770	30994
8	4096	16	16	256	148215.92498334651	74107.96249167326	24703	35386
8	4096	16	16	1024	98581.15579329274	197162.31158658548	39292	59472
8	4096	16	16	4	170606.2436220505	1332.8612782972696	20249	31148
8	4096	8	8	4	168135.7403836393	1313.560471747182	21324	33085
8	6000	8	8	4	165030.80341203389	1289.3031516565147	31588	45289
8	6000	4	4	4	129725.25596144305	1013.4785621987738	36778	93851
	root@ubuntu-c-16-sfo2-01:~/gnat# mix run bench/parse.exs
	Operating System: Linux"
	CPU Information: Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
	Number of Available Cores: 16
	Available memory: 31.42 GB
	Elixir 1.8.0
	Erlang 21.2.2

	Benchmark suite executing with the following configuration:
	warmup: 2 s
	time: 10 s
	memory time: 0 μs
	parallel: 1
	inputs: 1024 byte, 128 byte, 16 byte
	Estimated total run time: 36 s


	Benchmarking parsec with input 1024 byte...
	Benchmarking parsec with input 128 byte...
	Benchmarking parsec with input 16 byte...

	##### With input 1024 byte #####
	Name ips average deviation median 99th %
	parsec 438.92 K 2.28 μs ±1748.29% 2 μs 4 μs

	##### With input 128 byte #####
	Name ips average deviation median 99th %
	parsec 606.19 K 1.65 μs ±2093.53% 1 μs 3 μs

	##### With input 16 byte #####
	Name ips average deviation median 99th %
	parsec 639.57 K 1.56 μs ±2452.74% 1 μs 3 μs
	root@ubuntu-c-16-sfo2-01:~/gnat# mix run bench/publish.exs
	Operating System: Linux"
	CPU Information: Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
	Number of Available Cores: 16
	Available memory: 31.42 GB
	Elixir 1.8.0
	Erlang 21.2.2

	Benchmark suite executing with the following configuration:
	warmup: 2 s
	time: 10 s
	memory time: 0 μs
	parallel: 1
	inputs: 1024 byte, 128 byte, 16 byte
	Estimated total run time: 36 s


	Benchmarking pub with input 1024 byte...
	Benchmarking pub with input 128 byte...
	Benchmarking pub with input 16 byte...

	##### With input 1024 byte #####
	Name ips average deviation median 99th %
	pub 225.62 K 4.43 μs ±589.95% 4 μs 9 μs

	##### With input 128 byte #####
	Name ips average deviation median 99th %
	pub 158.10 K 6.33 μs ±311.64% 5 μs 11 μs

	##### With input 16 byte #####
	Name ips average deviation median 99th %
	pub 162.02 K 6.17 μs ±324.82% 5 μs 12 μs
	root@ubuntu-c-16-sfo2-01:~/gnat# mix run bench/request.exs
	Operating System: Linux"
	CPU Information: Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
	Number of Available Cores: 16
	Available memory: 31.42 GB
	Elixir 1.8.0
	Erlang 21.2.2

	Benchmark suite executing with the following configuration:
	warmup: 2 s
	time: 10 s
	memory time: 0 μs
	parallel: 1
	inputs: 1024 byte, 128 byte, 16 byte
	Estimated total run time: 36 s


	Benchmarking request with input 1024 byte...
	Benchmarking request with input 128 byte...
	Benchmarking request with input 16 byte...

	##### With input 1024 byte #####
	Name ips average deviation median 99th %
	request 8.81 K 113.50 μs ±38.27% 111 μs 150 μs

	##### With input 128 byte #####
	Name ips average deviation median 99th %
	request 8.72 K 114.68 μs ±14.35% 112 μs 154.33 μs

	##### With input 16 byte #####
	Name ips average deviation median 99th %
	request 8.81 K 113.54 μs ±15.65% 111 μs 150 μs