ruby processing speeds with the lumberjack (prototype) protocol

readme.md

• https://github.com/jordansissel/lumberjack at revision 2e2c8f4f
• Run 'make' from lumberjack
• run 'build/bin/lumberjack [some large log file]' - I used an old apache log with 2ish gigs of data.
• details on lumberjack protocol https://github.com/jordansissel/lumberjack#readme

Scenario:

• C program reading lines from a file pushing them out a custom protocol over the network.\
• Ruby program consuming that network protocol.

MRI 1.9.3

% ruby --version
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]

# Runs the reader/parser
% ruby lumberjack.rb

# Run the emitter
% build/bin/lumberjack ~/logs/access_log 
Connecting to localhost(127.0.0.1):1234
Connected successfully to localhost(127.0.0.1):1234
Rate: 63031.395709
Rate: 57277.101196
Rate: 57982.996313
Rate: 59372.576960
Rate: 58757.074170
Rate: 58672.095057
Rate: 58214.235940
Rate: 57843.658092
Rate: 58530.891191
Rate: 58611.844784
Rate: 58823.076509
Rate: 58438.315439
Rate: 58199.337014
...

JRuby 1.6.7

% ruby --version
jruby 1.6.7 (ruby-1.8.7-p357) (2012-02-22 3e82bc8) (OpenJDK 64-Bit Server VM 1.6.0_24) [linux-amd64-java]

% ruby --1.9 lumberjack.rb

% build/bin/lumberjack ~/logs/access_log
Connecting to localhost(127.0.0.1):1234
Connected successfully to localhost(127.0.0.1):1234
Rate: 18839.617430
Rate: 67063.906243
Rate: 135546.777710
Rate: 153908.457677
Rate: 182832.977998
Rate: 190160.582471
Rate: 187394.559042
Rate: 195221.100875
Rate: 191259.522704
...
Rate: 213306.116822
Rate: 216849.445095
Rate: 214213.063491
Rate: 220381.987339
Rate: 219606.818685
Rate: 221823.438912
Rate: 224033.348518
Rate: 222018.387998
Rate: 223325.720615
Rate: 226943.071307

Hi!,

when I "make" I get following error,

cc1: warnings being treated as errors
lumberjack.c: In function ‘set_resource_limits’:
lumberjack.c:84: warning: implicit declaration of function ‘insist’
make: *** [lumberjack.o] Error 1

Please help. Thanks,

/Kambleam