Skip to content

Instantly share code, notes, and snippets.

@jsvd

jsvd/README.md

Last active Apr 7, 2021
Embed
What would you like to do?
logstash grok filter benchmark script

Requirements:

  • JDK 8/9/11 (oracle or openjdk)

Steps to setup the scripts:

mkdir benches
cd benches
curl https://artifacts.elastic.co/downloads/logstash/logstash-7.6.0.tar.gz | tar -zxf - 
cd logstash-7.6.0
wget https://gist.github.com/jsvd/a2613ea1ba00f02926a302781ca62f7b/raw/1a9e97f7c20bf21fefe5b461238074c86859d993/benchmark_grok.rb
sed -i.bak 's/gem \"benchmark-ips\", :group => :development/gem \"benchmark-ips\"/g' Gemfile   
bin/logstash-plugin update

To run the benchmarks:

bin/ruby benchmark_grok.rb

# encoding: utf-8
require_relative "lib/bootstrap/environment"
LogStash::Bundler.setup!({:without => [:build, :development]})
require "logstash-core"
require "logstash/environment"
require "logstash/plugin"
require 'benchmark/ips'
line = '220.181.108.96 - - [13/Jun/2015:21:14:28 +0000] "GET /blog/geekery/xvfb-firefox.html HTTP/1.1" 200 10975 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"'
pattern = '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
# grok plugin setup
grok_plugin = LogStash::Plugin.lookup("filter", "grok").new("match" => ["message", pattern])
grok_plugin.register
# do the benchmark
Benchmark.ips do |x|
x.time = 30
x.warmup = 10
x.report("grok_plugin") do
# we need to create a new event on each iteration since it's modified by grok
event = LogStash::Event.new("message" => line)
grok_plugin.filter(event)
end
# x.report("grok_plugin_2") do
# event = LogStash::Event.new("message" => line)
# grok_plugin_2.filter(event)
# end
x.compare! # necessary only with multiple reports
end
@Budman17r

This comment has been minimized.

Copy link

@Budman17r Budman17r commented Feb 9, 2017

I can't get this to run how do you get Jruby to respond to Ruby?

@guyboertje

This comment has been minimized.

Copy link

@guyboertje guyboertje commented Oct 6, 2017

@Budman17r
You create a symlink in the JRuby bin folder ruby -> jruby

@esg-llachance

This comment has been minimized.

Copy link

@esg-llachance esg-llachance commented Mar 2, 2018

Is this available as a docker image?

@vmavromatis

This comment has been minimized.

Copy link

@vmavromatis vmavromatis commented Jul 10, 2018

Thanks. For me I needed sudo for bundler: sudo gem install bundler

@brianjolly

This comment has been minimized.

Copy link

@brianjolly brianjolly commented Aug 17, 2018

@esg-llachance, fwif I'm running it interactively in docker with this:
docker run -it --rm --name grok -v "$PWD":/benches -w /benches jruby /bin/bash

@phihag

This comment has been minimized.

Copy link

@phihag phihag commented Oct 8, 2019

@brianjolly When I run rake bootstrap, it says bash: rake: command not found.

Running jruby -S gem install rake && jruby -S rake shows another error message, Gem::MissingSpecVersionError: Could not find 'json' (~> 1) - did find: [json-2.2.0-java]. Can I update the Gemfile somehow, or use another docker image?

@brianjolly

This comment has been minimized.

Copy link

@brianjolly brianjolly commented Nov 7, 2019

Hi @phihag, these are my setup steps inside the container:

docker run -it --rm --name grok -v "$PWD":/benches -w /benches jruby /bin/bash
cd logstash
rake bootstrap
cd ..
bundle update --bundler
bundle install

bundle exec ruby benchmark_grok.rb
@jsvd

This comment has been minimized.

Copy link
Owner Author

@jsvd jsvd commented Feb 14, 2020

The script and instructions have been updated to run with logstash 7.6.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment