Skip to content

Instantly share code, notes, and snippets.

@jsvd
Last active June 13, 2024 00:05
Show Gist options
  • Save jsvd/a2613ea1ba00f02926a302781ca62f7b to your computer and use it in GitHub Desktop.
Save jsvd/a2613ea1ba00f02926a302781ca62f7b to your computer and use it in GitHub Desktop.
logstash grok filter benchmark script

Requirements:

  • JDK 8/9/11 (oracle or openjdk)

Steps to setup the scripts:

mkdir benches
cd benches
curl https://artifacts.elastic.co/downloads/logstash/logstash-7.6.0.tar.gz | tar -zxf - 
cd logstash-7.6.0
wget https://gist.github.com/jsvd/a2613ea1ba00f02926a302781ca62f7b/raw/1a9e97f7c20bf21fefe5b461238074c86859d993/benchmark_grok.rb
sed -i.bak 's/gem \"benchmark-ips\", :group => :development/gem \"benchmark-ips\"/g' Gemfile   
bin/logstash-plugin update

To run the benchmarks:

bin/ruby benchmark_grok.rb

# encoding: utf-8
require_relative "lib/bootstrap/environment"
LogStash::Bundler.setup!({:without => [:build, :development]})
require "logstash-core"
require "logstash/environment"
require "logstash/plugin"
require 'benchmark/ips'
line = '220.181.108.96 - - [13/Jun/2015:21:14:28 +0000] "GET /blog/geekery/xvfb-firefox.html HTTP/1.1" 200 10975 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"'
pattern = '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
# grok plugin setup
grok_plugin = LogStash::Plugin.lookup("filter", "grok").new("match" => ["message", pattern])
grok_plugin.register
# do the benchmark
Benchmark.ips do |x|
x.time = 30
x.warmup = 10
x.report("grok_plugin") do
# we need to create a new event on each iteration since it's modified by grok
event = LogStash::Event.new("message" => line)
grok_plugin.filter(event)
end
# x.report("grok_plugin_2") do
# event = LogStash::Event.new("message" => line)
# grok_plugin_2.filter(event)
# end
x.compare! # necessary only with multiple reports
end
@brianjolly
Copy link

Hi @phihag, these are my setup steps inside the container:

docker run -it --rm --name grok -v "$PWD":/benches -w /benches jruby /bin/bash
cd logstash
rake bootstrap
cd ..
bundle update --bundler
bundle install

bundle exec ruby benchmark_grok.rb

@jsvd
Copy link
Author

jsvd commented Feb 14, 2020

The script and instructions have been updated to run with logstash 7.6.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment