logstash 1.1.1 perf testing

00-internal.md

# This tests the raw speed of logstash passing messages around between
# threads/queues without doing much logic.

input {
    generator {
        type => "gen"
        count => 1000000
    }
}

filter {
    noop {
        add_tag => "test"
    }
}

output {
    null { }
}

• with 1.1.0: # no generator input yet
• with 1.1.1-rc2: 58s 60s 58s

01-stdin-out.md

input {
    stdin {
        type => "stdin"
    }
}

filter {
    grep { # 1.1.0 doesn't have a noop filter
        add_tag => "test"
    }
}

output {
    stdout { }
}

date; time seq -f "testing %f" 1 1000000 | 
stdbuf -o 0 java -jar logstash-foo.jar agent -f \
etc/bench-stdin-out.conf |
pv -bt >/dev/null

and wait for pv to hit ~77.1M of output and note the time. This is necessary with 1.1.0's agent doesn't cleanly shutdown when stdin EOFs. Note that when measuring times, there is a ~5 second shutdown overhead when waiting on logstash to exit on stdin EOF (watch pv closely). The times below do not include the shutdown overhead.

• with 1.1.0: 55s 55s
• with 1.1.1-rc2: 60s