Skip to content

Instantly share code, notes, and snippets.

@guyboertje
Last active August 29, 2015 14:25
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save guyboertje/5e3b66134b3e19298b55 to your computer and use it in GitHub Desktop.
Save guyboertje/5e3b66134b3e19298b55 to your computer and use it in GitHub Desktop.
Logstash pipeline bench report
For all below, this applies...
ruby -v
jruby 1.7.20 (1.9.3p551) 2015-05-04 3086e6a on Java HotSpot(TM) 64-Bit Server VM 1.8.0_45-b14 +jit [darwin-x86_64]
1) As a baseline for experimentation I substitute the JSON parser and generator for 'perfect' simulations.
So for parse I simply return a new Hash and for generate I return a new String with 1 interpolation.
yes '{"test": 1}' | bin/logstash -e 'input{stdin{codec => json_lines}} filter{clone{}} output{stdout{codec => json_lines}}' | pv -Wlart > /dev/null
01:00 [89.3k/s] [ 89k/s]
This give me a 'high water' mark. Even with a perfect JSON parser/generator we can't go faster than this.
2) With JrJackson at v0.2.9 (current).
yes '{"test": 1}' | bin/logstash -e 'input{stdin{codec => json_lines}} filter{clone{}} output{stdout{codec => json_lines}}' | pv -Wlart > /dev/null
01:00 [54.2k/s] [50.9k/s]
3) With JrJackson at v0.3.0 (new unreleased)
yes '{"test": 1}' | bin/logstash -e 'input{stdin{codec => json_lines}} filter{clone{}} output{stdout{codec => json_lines}}' | pv -Wlart > /dev/null
01:00 [73.1k/s] [71.4k/s]
BUT Here is the strangeness!
If I remove the JSOM lines codec from either end, I see a reduced thoughput.
For perfect JSON...
yes '{"test": 1}' | bin/logstash -e 'input{stdin{codec => json_lines}} filter{clone{}} output{stdout{}}' | pv -Wlart > /dev/null
01:00 [56.1k/s] [54.6k/s]
and...
yes '{"test": 1}' | bin/logstash -e 'input{stdin{}} filter{clone{}} output{stdout{codec => json_lines}}' | pv -Wlart > /dev/null
01:00 [84.9k/s] [ 85k/s]
I see the same relative numbers for old JrJackson and new JrJackson when removing the JSON lines codec from either end.
Question:
What is the explanation for this?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment