Skip to content

Instantly share code, notes, and snippets.

@jordansissel
Created June 22, 2012 03:28
Show Gist options
  • Save jordansissel/2970009 to your computer and use it in GitHub Desktop.
Save jordansissel/2970009 to your computer and use it in GitHub Desktop.
Logstash 1.1.1 + ElasticSearch 0.19.5 compression tests

compression tests of logstash+elasticsearch

  • logstash 1.1.1-pre (unreleased)
  • elasticsearch 0.19.5-SNAPSHOT (unreleased)
wget https://dl.dropbox.com/u/2136051/elasticsearch-0.19.5-SNAPSHOT.zip
unzip elasticsearch-0.19.5-SNAPSHOT.zip

Sample Log Data

250,000 apache log entries.

% wc -l /tmp/sample.log 
250000 /tmp/sample.log

% carrera(~) % du -hs /tmp/sample.log 
52M     /tmp/sample.log

% head -1 /tmp/sample.log 
61.245.165.56 - - [24/May/2012:10:07:26 -0400] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (Windows NT 6.1; rv:13.0) Gecko/20100101 Firefox/13.0"

Logstash

% java -jar build/logstash-1.1.1-pre-monolithic.jar agent -e 'input { tcp { type => "apache" port => 4455 }} filter { grok { pattern => "%{COMBINEDAPACHELOG}" } date { timestamp => "dd/MMM/yyyy:HH:mm:ss Z" } } output { elasticsearch { cluster => "test1"} }'

FEeding data

% nc localhost 4455 < sample.log

Run 1: Without Compression

rm -rf data
ulimit -n 100000
bin/elasticsearch -f -Des.cluster.name=test1

Results

  • input duration: 129.99 seconds (tcp input through netcat)
  • data directory size: 344M
  • documents in elasticsearch: 250,000 (GET /_count?q=*)

Run 2: With Compression

rm -rf data config/elasticsearch.yml
ulimit -n 100000
bin/elasticsearch -f -Des.cluster.name=test1 -Des.index.store.compress.stored=true

Results

  • input duration: 125.56 seconds
  • data directory size: 181M
  • documents in elasticsearch: 250,000
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment