benchmark fluentd vs fluent-agent-lite vs fluent-agent-hydra

readme.md

• AWS EC2 c3.2xlarge
• Amazon Linux AMI release 2014.03
• Linux 3.10.48-55.140.amzn1.x86_64 #1 SMP Wed Jul 9 23:32:19 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
• ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux]
• fluentd 0.10.52
• perl This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi
• fluent-agent-lite 1.0
• go version go1.3 linux/amd64
• fluent-agent-hydra v0.0.2

agent.conf

<source>
  type tail
  path /dev/shm/tmp/dummy.log
  pos_file /var/tmp/_var_log_dummy.pos
  format none
  tag dummy
</source>
<match dummy>
  type copy
  <store>
    type forward
    flush_interval 0
    try_flush_interval 1
    buffer_chunk_limit 8m
    buffer_queue_limit 64
    num_threads 32
    # phi_threshold 35
    <server>
      host 127.0.0.1 # FIX ME
      port 24224
    </server>
  </store>
</match>

result.md

https://github.com/fluent/fluentd-benchmark/tree/master/one_forward

logs/sec	fluentd CPU	RSS	lite CPU	RSS	hydra CPU	RSS	hydra CPU(GOMAXPROCS=2)	RSS
100	0.65	40152	0.18	8424	0.72	10876	1.2	10872
1000	1.7	51836	0.34	8420	1.0	18460	1.5	18188
5000	5.5	87316	1.0	9280	2.2	20804	3.1	25668
10000	11	85468	2.0	9812	3.8	26252	5.7	30468
50000	48	132496	8.9	15216	16	55772	21	54860
100000	107	636892	18	20948	30	75932	42	66920
250000	-	-	45	21048	65	67372	113	90420
500000	-	-	87	21092	98	80208	160	108032
700000	-	-	-	-	-	-	165	106796

• fluentdは100000で emit transaction failed error_class=Fluent::BufferQueueLimitError error=#<Fluent::BufferQueueLimitError: queue size exceeds limit
• fluent-agent-lite max: 580000/sec
• fluent-agent-hydra max: 580000/sec
• fluent-agetn-hydra(GOMAXPROCS=2) max: 700000/sec