Logstash 1.1.1 + ElasticSearch 0.19.5 compression tests

data.md

compression tests of logstash+elasticsearch

• logstash 1.1.1-pre (unreleased)
• elasticsearch 0.19.5-SNAPSHOT (unreleased)

wget https://dl.dropbox.com/u/2136051/elasticsearch-0.19.5-SNAPSHOT.zip
unzip elasticsearch-0.19.5-SNAPSHOT.zip

Sample Log Data

250,000 apache log entries.

% wc -l /tmp/sample.log 
250000 /tmp/sample.log

% carrera(~) % du -hs /tmp/sample.log 
52M     /tmp/sample.log

% head -1 /tmp/sample.log 
61.245.165.56 - - [24/May/2012:10:07:26 -0400] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (Windows NT 6.1; rv:13.0) Gecko/20100101 Firefox/13.0"

Logstash

% java -jar build/logstash-1.1.1-pre-monolithic.jar agent -e 'input { tcp { type => "apache" port => 4455 }} filter { grok { pattern => "%{COMBINEDAPACHELOG}" } date { timestamp => "dd/MMM/yyyy:HH:mm:ss Z" } } output { elasticsearch { cluster => "test1"} }'

FEeding data

% nc localhost 4455 < sample.log

Run 1: Without Compression

rm -rf data
ulimit -n 100000
bin/elasticsearch -f -Des.cluster.name=test1

Results

• input duration: 129.99 seconds (tcp input through netcat)
• data directory size: 344M
• documents in elasticsearch: 250,000 (GET /_count?q=*)

Run 2: With Compression

rm -rf data config/elasticsearch.yml
ulimit -n 100000
bin/elasticsearch -f -Des.cluster.name=test1 -Des.index.store.compress.stored=true

Results

• input duration: 125.56 seconds
• data directory size: 181M
• documents in elasticsearch: 250,000

how do you enable compression using the configuration file?