td-agent 1.1.17
ログ生成
https://github.com/sonots/dummy_log_generator を使用し、-r 100000 や -r 200000 と指定。実際に20万lines/sec吐き出せていることはもちろん確認済み。
送信側
out_forward に https://gist.github.com/sonots/7646378 と https://gist.github.com/sonots/7662371 のパッチをあててログを出すようにしている。最終品は https://gist.github.com/sonots/7670640 にある。
<source>
type tail
path /var/log/dummy.log
pos_file /var/tmp/_var_log_dummy.pos
format none
tag dummy
</source>
<match dummy>
type forward
flush_interval 0
buffer_queue_limit 10
buffer_chunk_limit 1g
# num_threads 1
retry_wait 0.1
retry_limit 17
max_retry_wait 0
# phi_threshold 35
<server>
host 10.33.145.22
port 20000
</server>
</match>受信側
in_forward にログを出すパッチ https://gist.github.com/sonots/7644438 を作ってみたが、https://github.com/sonots/fluent-plugin-flowcounter-simple で十分計測できそうだったのでそちらを使用。flowcounter-simple は flowcounter の簡易版。
<source>
type forward
port 20000
</source>
<match dummy>
type copy
<store>
type flowcounter_simple
unit second
indicator byte
</store>
<store>
type flowcounter_simple
unit second
indicator num
</store>
</match>送信側
timing:try が #send_data を呼んだタイミング、timing:sent が #send_data が終了したタイミング。
timing:second は毎秒送信したデータ量をログ吐き出しする Thread のログである(0 の場合は出力しない)
内部実装の都合上、行数ではなくバイト数を出力している。
10万 lines / sec の場合は大体毎秒データが流れていて問題なさそう。
2013-11-27 13:09:02 +0900 [info]: plugin:out_forward size:8260000 timing:try
2013-11-27 13:09:02 +0900 [info]: plugin:out_forward size:8260000 timing:sent
2013-11-27 13:09:03 +0900 [info]: plugin:out_forward size:8260000 timing:second
2013-11-27 13:09:03 +0900 [info]: plugin:out_forward size:12980000 timing:try
2013-11-27 13:09:04 +0900 [info]: plugin:out_forward size:12980000 timing:sent
2013-11-27 13:09:04 +0900 [info]: plugin:out_forward size:2360000 timing:try
2013-11-27 13:09:04 +0900 [info]: plugin:out_forward size:2360000 timing:sent
2013-11-27 13:09:04 +0900 [info]: plugin:out_forward size:15340000 timing:second
2013-11-27 13:09:05 +0900 [info]: plugin:out_forward size:11800000 timing:try
2013-11-27 13:09:05 +0900 [info]: plugin:out_forward size:11800000 timing:sent
2013-11-27 13:09:06 +0900 [info]: plugin:out_forward size:11800000 timing:second
2013-11-27 13:09:06 +0900 [info]: plugin:out_forward size:12980000 timing:try
2013-11-27 13:09:06 +0900 [info]: plugin:out_forward size:12980000 timing:sent
2013-11-27 13:09:07 +0900 [info]: plugin:out_forward size:12980000 timing:second
2013-11-27 13:09:07 +0900 [info]: plugin:out_forward size:14160000 timing:try
2013-11-27 13:09:07 +0900 [info]: plugin:out_forward size:14160000 timing:sent
2013-11-27 13:09:08 +0900 [info]: plugin:out_forward size:14160000 timing:second
2013-11-27 13:09:08 +0900 [info]: plugin:out_forward size:14160000 timing:try
2013-11-27 13:09:08 +0900 [info]: plugin:out_forward size:14160000 timing:sent
2013-11-27 13:09:09 +0900 [info]: plugin:out_forward size:14160000 timing:second
2013-11-27 13:09:09 +0900 [info]: plugin:out_forward size:14160000 timing:try
2013-11-27 13:09:10 +0900 [info]: plugin:out_forward size:14160000 timing:sent
2013-11-27 13:09:10 +0900 [info]: plugin:out_forward size:2360000 timing:try
2013-11-27 13:09:10 +0900 [info]: plugin:out_forward size:2360000 timing:sent
2013-11-27 13:09:10 +0900 [info]: plugin:out_forward size:16520000 timing:second
2013-11-27 13:09:11 +0900 [info]: plugin:out_forward size:11800000 timing:try
2013-11-27 13:09:11 +0900 [info]: plugin:out_forward size:11800000 timing:sent
2013-11-27 13:09:12 +0900 [info]: plugin:out_forward size:11800000 timing:second
2013-11-27 13:09:12 +0900 [info]: plugin:out_forward size:12980000 timing:try
2013-11-27 13:09:12 +0900 [info]: plugin:out_forward size:12980000 timing:sent
2013-11-27 13:09:13 +0900 [info]: plugin:out_forward size:12980000 timing:second
2013-11-27 13:09:13 +0900 [info]: plugin:out_forward size:14160000 timing:try
2013-11-27 13:09:13 +0900 [info]: plugin:out_forward size:14160000 timing:sent
2013-11-27 13:09:14 +0900 [info]: plugin:out_forward size:14160000 timing:second
受信側
bytes / sec および lines / sec を出している
2013-11-27 13:09:03 +0900 [info]: plugin:out_flowcounter_simple count:70000 indicator:num unit:second
2013-11-27 13:09:03 +0900 [info]: plugin:out_flowcounter_simple count:7840000 indicator:byte unit:second
2013-11-27 13:09:05 +0900 [info]: plugin:out_flowcounter_simple count:14560000 indicator:byte unit:second
2013-11-27 13:09:05 +0900 [info]: plugin:out_flowcounter_simple count:130000 indicator:num unit:second
2013-11-27 13:09:06 +0900 [info]: plugin:out_flowcounter_simple count:11200000 indicator:byte unit:second
2013-11-27 13:09:06 +0900 [info]: plugin:out_flowcounter_simple count:100000 indicator:num unit:second
2013-11-27 13:09:07 +0900 [info]: plugin:out_flowcounter_simple count:110000 indicator:num unit:second
2013-11-27 13:09:07 +0900 [info]: plugin:out_flowcounter_simple count:12320000 indicator:byte unit:second
2013-11-27 13:09:08 +0900 [info]: plugin:out_flowcounter_simple count:120000 indicator:num unit:second
2013-11-27 13:09:08 +0900 [info]: plugin:out_flowcounter_simple count:13440000 indicator:byte unit:second
2013-11-27 13:09:09 +0900 [info]: plugin:out_flowcounter_simple count:13440000 indicator:byte unit:second
2013-11-27 13:09:10 +0900 [info]: plugin:out_flowcounter_simple count:120000 indicator:num unit:second
2013-11-27 13:09:11 +0900 [info]: plugin:out_flowcounter_simple count:140000 indicator:num unit:second
2013-11-27 13:09:11 +0900 [info]: plugin:out_flowcounter_simple count:15680000 indicator:byte unit:second
2013-11-27 13:09:12 +0900 [info]: plugin:out_flowcounter_simple count:100000 indicator:num unit:second
2013-11-27 13:09:12 +0900 [info]: plugin:out_flowcounter_simple count:11200000 indicator:byte unit:second
2013-11-27 13:09:13 +0900 [info]: plugin:out_flowcounter_simple count:110000 indicator:num unit:second
2013-11-27 13:09:13 +0900 [info]: plugin:out_flowcounter_simple count:12320000 indicator:byte unit:second
2013-11-27 13:09:14 +0900 [info]: plugin:out_flowcounter_simple count:120000 indicator:num unit:second
2013-11-27 13:09:14 +0900 [info]: plugin:out_flowcounter_simple count:13440000 indicator:byte unit:second
2013-11-27 13:09:15 +0900 [info]: plugin:out_flowcounter_simple count:50000 indicator:num unit:second
2013-11-27 13:09:15 +0900 [info]: plugin:out_flowcounter_simple count:5600000 indicator:byte unit:second
10万 lines / sec の場合は、大体毎秒データが流れたが、一旦詰まると try してから sent になるまで2分以上かかり、性能が急激に劣化する。元々捌けていた 10万 lines / sec も捌けていないし、まとめてデータを送る挙動になるのもうれしくない。
なお、最終的には queue size limit exceeds エラーが出る。
なお、flush_interval 0、buffer_chunk_limit 1g および max_retry_wait 0 とし、がんがんデータを送る設定にして実験している。もちろん max_retry_wait の設定ありなしなど試したが大まかな挙動は変わらなかった。
送信側
2013-11-27 13:26:01 +0900 [info]: plugin:out_forward size:6460264 timing:try
2013-11-27 13:26:03 +0900 [info]: plugin:out_forward size:6460264 timing:sent
2013-11-27 13:26:03 +0900 [info]: plugin:out_forward size:37446002 timing:try
2013-11-27 13:26:04 +0900 [info]: plugin:out_forward size:6460264 timing:second
2013-11-27 13:26:15 +0900 [info]: plugin:out_forward size:37446002 timing:sent
2013-11-27 13:26:15 +0900 [info]: plugin:out_forward size:187110122 timing:try
2013-11-27 13:26:15 +0900 [info]: plugin:out_forward size:37446002 timing:second
2013-11-27 13:26:45 +0900 [info]: plugin:out_forward size:187110122 timing:sent
2013-11-27 13:26:45 +0900 [info]: plugin:out_forward size:481892412 timing:try
2013-11-27 13:26:46 +0900 [info]: plugin:out_forward size:187110122 timing:second
2013-11-27 13:28:17 +0900 [info]: plugin:out_forward size:481892412 timing:sent
2013-11-27 13:28:17 +0900 [info]: plugin:out_forward size:481892412 timing:second
2013-11-27 13:28:18 +0900 [info]: plugin:out_forward size:1073729790 timing:try
2013-11-27 13:30:14 +0900 [info]: plugin:out_forward size:1073729790 timing:sent
2013-11-27 13:30:14 +0900 [info]: plugin:out_forward size:1073729790 timing:second
2013-11-27 13:30:15 +0900 [info]: plugin:out_forward size:1073729908 timing:try
2013-11-27 13:33:47 +0900 [info]: plugin:out_forward size:1073729908 timing:sent
2013-11-27 13:33:47 +0900 [info]: plugin:out_forward size:1073729908 timing:second
2013-11-27 13:33:48 +0900 [info]: plugin:out_forward size:1073729908 timing:try
2013-11-27 13:35:52 +0900 [info]: plugin:out_forward size:1073729908 timing:sent
2013-11-27 13:35:52 +0900 [info]: plugin:out_forward size:1073729908 timing:second
2013-11-27 13:35:53 +0900 [info]: plugin:out_forward size:1073729908 timing:try
CPU使用率
100%超えるとこうなる。検証環境では 15万 lines/sec ぐらいでこうなった。
$ top -c
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
43094 td-agent 20 0 1474m 1.1g 2620 S 103.8 4.8 22:17.38 /usr/lib64/fluent/ruby/bin/ruby /usr/sbin/td-agent --group td-agent --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid
NWとか
$ dstat
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
1 0 99 0 0 0| 16k 333k| 0 0 | 49B 58B|1376 1038
4 1 95 0 0 0| 0 0 | 146k 9426k| 0 0 |6231 29k
3 1 96 0 0 0| 0 112k| 133k 8603k| 0 0 |5763 27k
5 1 93 0 0 0| 0 68k| 123k 8015k| 0 0 |7914 33k
3 1 96 0 0 0| 0 0 | 164k 10M| 0 0 |5993 28k
5 2 93 0 0 0| 0 0 | 129k 8365k| 0 0 |5929 28k
3 1 97 0 0 0| 0 0 | 126k 8261k| 0 0 |6322 29k
5 1 94 0 0 0| 0 68k| 100k 9416k| 0 0 |4512 29k
3 1 96 0 0 0| 0 0 | 118k 9970k| 0 0 |7228 33k
6 2 92 0 0 0| 0 0 | 173k 11M| 0 0 |7003 32k
受信側
9099405 lines を３分半かけて受け取っている。9099405 / 210 = 43,330 程度で、10万 lines 行ってない。
2013-11-27 13:26:04 +0900 [info]: plugin:out_flowcounter_simple count:54748 indicator:num unit:second
2013-11-27 13:26:04 +0900 [info]: plugin:out_flowcounter_simple count:6131776 indicator:byte unit:second
2013-11-27 13:26:16 +0900 [info]: plugin:out_flowcounter_simple count:317339 indicator:num unit:second
2013-11-27 13:26:16 +0900 [info]: plugin:out_flowcounter_simple count:35541968 indicator:byte unit:second
2013-11-27 13:26:49 +0900 [info]: plugin:out_flowcounter_simple count:177596048 indicator:byte unit:second
2013-11-27 13:26:50 +0900 [info]: plugin:out_flowcounter_simple count:1585679 indicator:num unit:second
2013-11-27 13:28:25 +0900 [info]: plugin:out_flowcounter_simple count:457389408 indicator:byte unit:second
2013-11-27 13:28:30 +0900 [info]: plugin:out_flowcounter_simple count:4083834 indicator:num unit:second
2013-11-27 13:30:32 +0900 [info]: plugin:out_flowcounter_simple count:1019133360 indicator:byte unit:second
2013-11-27 13:30:43 +0900 [info]: plugin:out_flowcounter_simple count:9099405 indicator:num unit:second
2013-11-27 13:34:06 +0900 [info]: plugin:out_flowcounter_simple count:1019133472 indicator:byte unit:second
2013-11-27 13:34:17 +0900 [info]: plugin:out_flowcounter_simple count:9099406 indicator:num unit:second
2013-11-27 13:36:12 +0900 [info]: plugin:out_flowcounter_simple count:1019133472 indicator:byte unit:second
2013-11-27 13:36:24 +0900 [info]: plugin:out_flowcounter_simple count:9099406 indicator:num unit:second
CPU使用率
$ top -c
4076 td-agent 20 0 4285m 2.6g 2600 S 8.6 11.0 17:11.20 /usr/lib64/fluent/ruby/bin/ruby /usr/sbin/td-agent --group td-agent --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid
NWとか
$ dstat
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
1 0 99 0 0 0| 187B 33k| 0 0 | 0 0 |1340 700
0 0 100 0 0 0| 0 0 | 13M 201k| 0 0 |6838 6523
0 0 99 0 0 0| 0 0 | 13M 201k| 0 0 |6781 6461
1 0 99 0 0 0| 0 396k| 13M 149k| 0 0 |5833 4286
1 1 98 0 0 0| 0 0 | 12M 158k| 0 0 |5998 5210
0 1 99 0 0 0| 0 0 | 13M 207k| 0 0 | 11k 9761
0 0 99 0 0 0| 0 0 | 13M 198k| 0 0 |6761 6445
0 0 100 0 0 0| 0 0 | 12M 192k| 0 0 |6428 6106
0 1 99 0 0 0| 0 144k| 11M 166k| 0 0 |5664 5446
num_threads 2 であまり変わった気がしなかったので思い切って 32 にした。
３分半の詰まりが1分ぐらいには縮まったが。。。
送信側
2013-11-27 14:57:42 +0900 [info]: plugin:out_forward size:1435588 timing:sent
2013-11-27 14:57:42 +0900 [info]: plugin:out_forward size:3828274 timing:try
2013-11-27 14:57:42 +0900 [info]: plugin:out_forward size:1435588 timing:second
2013-11-27 14:57:43 +0900 [info]: plugin:out_forward size:3828274 timing:sent
2013-11-27 14:57:43 +0900 [info]: plugin:out_forward size:24166400 timing:try
2013-11-27 14:57:43 +0900 [info]: plugin:out_forward size:3828274 timing:second
2013-11-27 14:57:48 +0900 [info]: plugin:out_forward size:24166400 timing:sent
2013-11-27 14:57:48 +0900 [info]: plugin:out_forward size:76686548 timing:try
2013-11-27 14:57:48 +0900 [info]: plugin:out_forward size:24166400 timing:second
2013-11-27 14:58:10 +0900 [info]: plugin:out_forward size:76686548 timing:sent
2013-11-27 14:58:10 +0900 [info]: plugin:out_forward size:340961590 timing:try
2013-11-27 14:58:10 +0900 [info]: plugin:out_forward size:76686548 timing:second
2013-11-27 14:59:17 +0900 [info]: plugin:out_forward size:1073729790 timing:try
2013-11-27 14:59:27 +0900 [info]: plugin:out_forward size:340961590 timing:sent
2013-11-27 14:59:28 +0900 [info]: plugin:out_forward size:340961590 timing:second
2013-11-27 15:00:26 +0900 [info]: plugin:out_forward size:1073729908 timing:try
2013-11-27 15:01:35 +0900 [info]: plugin:out_forward size:1073729908 timing:try
2013-11-27 15:02:44 +0900 [info]: plugin:out_forward size:1073729908 timing:try
2013-11-27 15:03:17 +0900 [info]: plugin:out_forward size:1073729790 timing:sent
2013-11-27 15:03:17 +0900 [info]: plugin:out_forward size:1073729790 timing:second
CPU使用率
$ top -c
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
62317 td-agent 20 0 6037m 3.8g 2640 S 105.6 16.0 4:50.71 /usr/lib64/fluent/ruby/bin/ruby /usr/sbin/td-agent --group td-agent --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid
NWとか
$ dstat
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
1 0 99 0 0 0| 25k 358k| 0 0 | 61B 72B|1383 1061
6 0 94 0 0 0| 0 112k| 186k 11M| 0 0 |9509 27k
4 0 96 0 0 0| 0 0 | 144k 8837k| 0 0 |5568 25k
2 0 97 0 0 0| 0 0 | 112k 7185k| 0 0 |2709 18k
4 0 95 0 0 0| 0 12k| 37k 2406k| 0 0 |2648 18k
3 0 97 0 0 0| 0 0 | 36k 2402k| 0 0 |2797 18k
3 0 96 0 0 0| 0 40k| 42k 2890k| 0 0 |2525 17k
3 0 97 0 0 0| 0 0 | 37k 2381k| 0 0 |2729 18k
4 0 96 0 0 0| 0 0 | 41k 2602k| 0 0 |2788 18k
2 0 97 0 0 0| 0 0 | 41k 2711k| 0 0 |2750 18k
4 0 96 0 0 0| 0 0 | 42k 2669k| 0 0 |2668 18k
3 0 97 0 0 0| 0 28k| 40k 2627k| 0 0 |2757 18k
受信側
2013-11-27 14:56:46 +0900 [info]: plugin:out_flowcounter_simple count:11695936 indicator:byte unit:second
2013-11-27 14:56:46 +0900 [info]: plugin:out_flowcounter_simple count:104428 indicator:num unit:second
2013-11-27 14:56:54 +0900 [info]: plugin:out_flowcounter_simple count:68472096 indicator:byte unit:second
2013-11-27 14:56:54 +0900 [info]: plugin:out_flowcounter_simple count:611358 indicator:num unit:second
2013-11-27 14:57:21 +0900 [info]: plugin:out_flowcounter_simple count:100266880 indicator:byte unit:second
2013-11-27 14:57:23 +0900 [info]: plugin:out_flowcounter_simple count:895240 indicator:num unit:second
2013-11-27 14:57:35 +0900 [info]: plugin:out_flowcounter_simple count:414144528 indicator:byte unit:second
2013-11-27 14:57:39 +0900 [info]: plugin:out_flowcounter_simple count:3697719 indicator:num unit:second
2013-11-27 14:57:43 +0900 [info]: plugin:out_flowcounter_simple count:12166 indicator:num unit:second
2013-11-27 14:57:43 +0900 [info]: plugin:out_flowcounter_simple count:1362592 indicator:byte unit:second
2013-11-27 14:57:44 +0900 [info]: plugin:out_flowcounter_simple count:32443 indicator:num unit:second
2013-11-27 14:57:44 +0900 [info]: plugin:out_flowcounter_simple count:3633616 indicator:byte unit:second
2013-11-27 14:57:49 +0900 [info]: plugin:out_flowcounter_simple count:204800 indicator:num unit:second
2013-11-27 14:57:49 +0900 [info]: plugin:out_flowcounter_simple count:22937600 indicator:byte unit:second
2013-11-27 14:58:11 +0900 [info]: plugin:out_flowcounter_simple count:72787232 indicator:byte unit:second
2013-11-27 14:58:12 +0900 [info]: plugin:out_flowcounter_simple count:649886 indicator:num unit:second
2013-11-27 14:59:34 +0900 [info]: plugin:out_flowcounter_simple count:323624560 indicator:byte unit:second
2013-11-27 14:59:37 +0900 [info]: plugin:out_flowcounter_simple count:2889505 indicator:num unit:second
2013-11-27 15:03:35 +0900 [info]: plugin:out_flowcounter_simple count:1019133360 indicator:byte unit:second
2013-11-27 15:03:47 +0900 [info]: plugin:out_flowcounter_simple count:9099405 indicator:num unit:second
chunk のサイズを小さくして、buffer_queue_limit を大きくしてみる。
結果、許容サイズを超えた場合の挙動に、詰まりがなくなりスムースにデータが流れた。ただし、スループットは落ちた。
設定
<source>
type tail
path /var/log/dummy.log
pos_file /var/tmp/_var_log_dummy.pos
format none
tag dummy
</source>
<match dummy>
type forward
flush_interval 0
buffer_queue_limit 10240
buffer_chunk_limit 1m
num_threads 32
retry_wait 0.1
retry_limit 17
max_retry_wait 0
# phi_threshold 35
<server>
host 10.33.145.22
port 20000
</server>
</match>送信側
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:9570862 timing:second
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:956980 timing:try
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:956980 timing:sent
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:956980 timing:try
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:51 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:11485058 timing:second
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:956980 timing:sent
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:956980 timing:try
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:try
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:52 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:53 +0900 [info]: plugin:out_forward size:957098 timing:sent
2013-11-27 17:10:53 +0900 [info]: plugin:out_forward size:957098 timing:try
受信側
2013-11-27 17:11:09 +0900 [info]: plugin:out_flowcounter_simple count:9992640 indicator:byte unit:second
2013-11-27 17:11:09 +0900 [info]: plugin:out_flowcounter_simple count:89220 indicator:num unit:second
2013-11-27 17:11:10 +0900 [info]: plugin:out_flowcounter_simple count:10900960 indicator:byte unit:second
2013-11-27 17:11:10 +0900 [info]: plugin:out_flowcounter_simple count:97330 indicator:num unit:second
2013-11-27 17:11:11 +0900 [info]: plugin:out_flowcounter_simple count:9992640 indicator:byte unit:second
2013-11-27 17:11:11 +0900 [info]: plugin:out_flowcounter_simple count:89220 indicator:num unit:second
CPU使用率とNWとか追記ed