fetep

Keybase proof

I hereby claim:

• I am fetep on github.
• I am fetep (https://keybase.io/fetep) on keybase.
• I have a public key whose fingerprint is ABBC E079 C8DC 883A 5697 32DC 8EBD 9865 6367 8B88

To claim this, I am signing this object:

fetep

c0 fetep-vm(~) % cat bin/git-push-n
#!/bin/bash

if [ $# -ne 2 ]; then
    echo "usage: git push-n <remote> <ref>" >&2
    exit 1
fi

output=$(git push -n "$@" 2>&1)
git_rc=$?

fetep

{
    "template": "logstash-*",
    "order" : 0,
    "settings" : {
        "number_of_shards" : 6,
        "number_of_replicas" : 1,
        "index" : {
          "routing" : { "allocation" : { "total_shards_per_node" : "2" } },
          "query" : { "default_field" : "@message" },
          "store" : { "compress" : { "stored" : true, "tv" : true, "throttle" : { "type" : "merge", "max_bytes_per_sec" : "5mb" } } },

fetep

over here, https://gist.github.com/3243702, i proposed extending the text protocol by supporting updating a timer with multiple values at once (to help statsd clients that do a bit of batching for efficiency). this makes the update string a bit expensive to parse (especially because you have to get to the end of a potentially long string to even know what kind of update you have).

so here are some thoughts on a different way to encode the statsd update.

if we chose one of these, so we could call this protocol version "2" in our zeromq input (e.g. zmq message payload would be "2;").

fetep

#!/usr/bin/env ruby

require 'benchmark'

N=1000    # number of parse iterations
T=10000   # number of timer update samples

# build some sample timer update strings w/#{T} values
samples = []
T.times { samples << ((Kernel.rand(1000) + 100.0) / 100.0) }

fetep

make statsd better for high throughput

In the interest of shipping high volumes of updates in the shortest amount of time and work, I'm proposing two statsd changes:

1. enhance the timer update format
2. add a zeromq input (in addition to UDP)

Original UDP protocol

fetep

# This tests the raw speed of logstash passing messages around between
# threads/queues without doing much logic.

input {
    generator {
        type => "gen"
        count => 1000000
    }
}

fetep

0 fetep-x201(~/l/logstash) % cat etc/bench.conf
input {
    generator {
        type => "gen"
        count => 100000
    }
}

filter {
    noop {

fetep

logstash config snippet:

output {
    elasticsearch {
        host => "localhost"
        node_name => "logstash-test1"
    }
}

fetep

input {
    stdin {
        type => "zeus"
    }
}

filter {
    grok {
        type => "zeus"
        pattern => "%{IP:ip} %{HOST:host} %{USER:user} \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) \"(?:%{URI:referrer}|-)\" %{QS:agent} \"%{GREEDYDATA:cookie}\" %{QS:ssl} node_s:(?:%{NUMBER:node_time}|-) req_s:(?:%{NUMBER:req_time}|-) retries:(?:%{NUMBER:retries}|-)"