This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
30 2 * * * ~/bin/curator.sh &> /dev/null |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
if [type] == "apache_json" { | |
geoip { source => "clientip" } | |
if [useragent] != "" { useragent { source => "useragent" } } | |
if [auth] == "-" { mutate { remove_field => "auth" } } | |
if [ident] == "-" { mutate { remove_field => "ident" } } | |
if [referer] == "-" { mutate { remove_field => "referer" } } | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
for index in $(ls /data/ES/elasticsearch/nodes/0/indices/); do echo -n "$index " ; curl -s -XGET "http://localhost:9200/$index/_segments?pretty=true" | grep committed_segments | awk -F: '{print $2}' | awk -F, '{sum += $1}; END {print sum}' | tail -1; done |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
failures in bulk class expected behavior | |
should retry an event until max_retries reached | |
RSpec::Mocks::MockExpectationError: (LogStash::Outputs::ElasticSearch: {"manage_template"=>true, "index"=>"logstash-2014.11.17", "template_overwrite"=>true, "hosts"=>["127.0.0.1"], "retry_max_items"=>10, "retry_max_interval"=>1, "max_retries"=>3, "codec"=><LogStash::Codecs::Plain charset=>"UTF-8">, "workers"=>1, "template_name"=>"logstash", "flush_size"=>500, "idle_flush_time"=>1, "action"=>"index", "path"=>"/", "ssl"=>false, "ssl_certificate_verification"=>true, "sniffing"=>false, "sniffing_delay"=>5, "doc_as_upsert"=>false, "upsert"=>""}).submit([["index", {:_id=>nil, :_index=>"logstash-2014.11.17", :_type=>"logs", :_routing=>nil}, #<LogStash::Event:0x6ca5d544 @metadata_accessors=#<LogStash::Util::Accessors:0x360bb9b1 @store={"retry_count"=>3}, @lut={}>, @cancelled=false, @data={"somevalue"=>100, "@timestamp"=>"2014-11-17T20:37:17.223Z", "@version"=>"1"}, @metadata={"retry_count"=>3}, @accessors=#<LogStash::Util::Acce |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"template" : "logstash-*", | |
"settings" : { | |
"index.refresh_interval" : "5s", | |
"analysis" : { | |
"analyzer" : { | |
"default" : { | |
"type" : "standard", | |
"stopwords" : "_none_" | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"@version":"1", | |
"@timestamp":"2013-12-29T15:41:02.123Z", | |
"host":"host.example.com", | |
"collectd_type":"df_complex", | |
"plugin":"df", | |
"plugin_instance":"Volumes-Time Machine Backups", | |
"type_instance":"used", | |
"value":2604146343936.0, | |
"tags":["collectd"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2016/01/06 23:15:27.708044 publish.go:100: DBG Publish: { | |
"@timestamp": "2016-01-06T23:15:23.210Z", | |
"beat": { | |
"hostname": "ip-172-31-46-141", | |
"name": "ip-172-31-46-141" | |
}, | |
"count": 1, | |
"fields": null, | |
"input_type": "log", | |
"message": "{ \"@timestamp\": \"2016-01-06T23:15:20+00:00\", \"@version\": \"1\", \"vhost\": \"untergeek.com\", \"clientip\": \"80.240.139.191\", \"bytes\": 60375, \"duration\": 0.114, \"status\": 200, \"request\": \"/feed/\", \"method\": \"GET\", \"referrer\": \"-\", \"useragent\": \"Fever/1.39 (Feed Parser; http://feedafever.com; Allow like Gecko)\" }", |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
curl -XDELETE http://localhost:9200/_template/logstash?pretty=true | |
echo | |
echo "Deleted current logstash template..." | |
echo "Creating new logstash template..." | |
sleep 2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
buh@Elasticbox (03:18 PM) ~/git/logstash | |
» rm -rf vendor | |
buh@Elasticbox (03:18 PM) ~/git/logstash | |
» rake bootstrap | |
mkdir -p vendor | |
mkdir vendor/_ | |
Downloading http://jruby.org.s3.amazonaws.com/downloads/1.7.25/jruby-bin-1.7.25.tar.gz | |
Installing minitar >= 0 because the build process needs it. | |
[bootstrap] Fetching and installing gem: minitar (>= 0) | |
Fetching: minitar-0.5.4.gem (100%) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/sh | |
unset CDPATH | |
. "$(cd `dirname $0`/..; pwd)/bin/logstash.lib.sh" | |
setup | |
for file in /etc/default/logstash /etc/sysconfig/logstash; do | |
if [ -f "$file" ]; then | |
. $file | |
fi |