measuring logstash filters performance

logstash_filtersr _performance.md

Measuring logstash filters performance

• given a data file sample named data.log
• modify your config to use the stdin input so that you can pipe you sample log file to logstash

input {
  stdin {
    codec => line
  }
}

• modify your config to use the stdout output with the dots codec

output {
  stdout {
    codec => dots
  }
}

• continously send your sample data to logtash and check pv numbers

$ while true; do cat t.t; done | bin/logstash -f myconfig.conf | pv -Wbart  > /dev/null
10.6KiB 0:01:15 [ 631 B/s] [ 732 B/s]

Here [ 631 B/s] is the current throughput in events per second and [ 732 B/s] is the average throughput for the test duration

It is important to let logstash run for a while, to factor out the statup time and the JVM warmup. Typically the average throughput settles after 30 secs to 1 minute.