Skip to content

Instantly share code, notes, and snippets.

View untergeek's full-sized avatar

Aaron Mildenstein untergeek

View GitHub Profile
@untergeek
untergeek / crontab
Created July 22, 2014 14:43
curator script info
30 2 * * * ~/bin/curator.sh &> /dev/null
@untergeek
untergeek / logstash.excerpt
Last active August 29, 2015 14:05
apache pruning
if [type] == "apache_json" {
geoip { source => "clientip" }
if [useragent] != "" { useragent { source => "useragent" } }
if [auth] == "-" { mutate { remove_field => "auth" } }
if [ident] == "-" { mutate { remove_field => "ident" } }
if [referer] == "-" { mutate { remove_field => "referer" } }
}
@untergeek
untergeek / segment_count.sh
Created September 16, 2012 13:42
Segment count per index
for index in $(ls /data/ES/elasticsearch/nodes/0/indices/); do echo -n "$index " ; curl -s -XGET "http://localhost:9200/$index/_segments?pretty=true" | grep committed_segments | awk -F: '{print $2}' | awk -F, '{sum += $1}; END {print sum}' | tail -1; done
@untergeek
untergeek / rspec.output.txt
Created October 26, 2015 18:25
The plot thickens... This is on a newly installed box, with no other deps installed anywhere.
failures in bulk class expected behavior
should retry an event until max_retries reached
RSpec::Mocks::MockExpectationError: (LogStash::Outputs::ElasticSearch: {"manage_template"=>true, "index"=>"logstash-2014.11.17", "template_overwrite"=>true, "hosts"=>["127.0.0.1"], "retry_max_items"=>10, "retry_max_interval"=>1, "max_retries"=>3, "codec"=><LogStash::Codecs::Plain charset=>"UTF-8">, "workers"=>1, "template_name"=>"logstash", "flush_size"=>500, "idle_flush_time"=>1, "action"=>"index", "path"=>"/", "ssl"=>false, "ssl_certificate_verification"=>true, "sniffing"=>false, "sniffing_delay"=>5, "doc_as_upsert"=>false, "upsert"=>""}).submit([["index", {:_id=>nil, :_index=>"logstash-2014.11.17", :_type=>"logs", :_routing=>nil}, #<LogStash::Event:0x6ca5d544 @metadata_accessors=#<LogStash::Util::Accessors:0x360bb9b1 @store={"retry_count"=>3}, @lut={}>, @cancelled=false, @data={"somevalue"=>100, "@timestamp"=>"2014-11-17T20:37:17.223Z", "@version"=>"1"}, @metadata={"retry_count"=>3}, @accessors=#<LogStash::Util::Acce
@untergeek
untergeek / ls-es-mapping.json
Last active December 31, 2015 12:49
Logstash v1.3.2+ Elasticsearch Mapping Template
{
"template" : "logstash-*",
"settings" : {
"index.refresh_interval" : "5s",
"analysis" : {
"analyzer" : {
"default" : {
"type" : "standard",
"stopwords" : "_none_"
}
@untergeek
untergeek / event.json
Last active January 1, 2016 16:39
Logstash events...
{
"@version":"1",
"@timestamp":"2013-12-29T15:41:02.123Z",
"host":"host.example.com",
"collectd_type":"df_complex",
"plugin":"df",
"plugin_instance":"Volumes-Time Machine Backups",
"type_instance":"used",
"value":2604146343936.0,
"tags":["collectd"]
@untergeek
untergeek / stdout.json
Created January 6, 2016 23:23
Beats eats my timestamp with pre-formatted JSON
2016/01/06 23:15:27.708044 publish.go:100: DBG Publish: {
"@timestamp": "2016-01-06T23:15:23.210Z",
"beat": {
"hostname": "ip-172-31-46-141",
"name": "ip-172-31-46-141"
},
"count": 1,
"fields": null,
"input_type": "log",
"message": "{ \"@timestamp\": \"2016-01-06T23:15:20+00:00\", \"@version\": \"1\", \"vhost\": \"untergeek.com\", \"clientip\": \"80.240.139.191\", \"bytes\": 60375, \"duration\": 0.114, \"status\": 200, \"request\": \"/feed/\", \"method\": \"GET\", \"referrer\": \"-\", \"useragent\": \"Fever/1.39 (Feed Parser; http://feedafever.com; Allow like Gecko)\" }",
@untergeek
untergeek / fix_logstash-1.3.1_template.sh
Created December 18, 2013 20:49
This script will replace the elasticsearch mapping template named "logstash" (see line 3) and replace it with the below template from v1.3.2, which is currently at https://github.com/logstash/logstash/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template.json This template will fix the error LOGSTASH-1711 and the "indexed without…
#!/bin/bash
curl -XDELETE http://localhost:9200/_template/logstash?pretty=true
echo
echo "Deleted current logstash template..."
echo "Creating new logstash template..."
sleep 2
@untergeek
untergeek / build_error
Created May 4, 2016 21:23
rake bootstrap error
buh@Elasticbox (03:18 PM) ~/git/logstash
» rm -rf vendor
buh@Elasticbox (03:18 PM) ~/git/logstash
» rake bootstrap
mkdir -p vendor
mkdir vendor/_
Downloading http://jruby.org.s3.amazonaws.com/downloads/1.7.25/jruby-bin-1.7.25.tar.gz
Installing minitar >= 0 because the build process needs it.
[bootstrap] Fetching and installing gem: minitar (>= 0)
Fetching: minitar-0.5.4.gem (100%)
@untergeek
untergeek / system-install.sh
Created May 11, 2016 19:24
System installer script
#!/bin/sh
unset CDPATH
. "$(cd `dirname $0`/..; pwd)/bin/logstash.lib.sh"
setup
for file in /etc/default/logstash /etc/sysconfig/logstash; do
if [ -f "$file" ]; then
. $file
fi