Skip to content

Instantly share code, notes, and snippets.

Avatar

Aaron Mildenstein untergeek

View GitHub Profile
@untergeek
untergeek / README.rst
Created Sep 27, 2018 — forked from HonzaKral/README.rst
CLI for elaasticsearch-py helpers
View README.rst

Elasticsearch CLI

Experimental CLI interface for the helpers in the `python library`_.

Main purpose is to expose the bulk functionality to enable rapid loading of data into an elasticsearch cluster. Combined with the scan command it can also be used to reindex data from elasticsearch into a different index or cluster.

@untergeek
untergeek / doc.md
Last active Apr 2, 2018
Rollover, Snapshot, and Curator
View doc.md

Snapshot, Rollover, and Curator

Snapshot

Create a repository

Name the repository whatever you like. In this case, testrepository.

PUT /_snapshot/testrepository
@untergeek
untergeek / system-install.sh
Created May 11, 2016
System installer script
View system-install.sh
#!/bin/sh
unset CDPATH
. "$(cd `dirname $0`/..; pwd)/bin/logstash.lib.sh"
setup
for file in /etc/default/logstash /etc/sysconfig/logstash; do
if [ -f "$file" ]; then
. $file
fi
@untergeek
untergeek / build_error
Created May 4, 2016
rake bootstrap error
View build_error
buh@Elasticbox (03:18 PM) ~/git/logstash
» rm -rf vendor
buh@Elasticbox (03:18 PM) ~/git/logstash
» rake bootstrap
mkdir -p vendor
mkdir vendor/_
Downloading http://jruby.org.s3.amazonaws.com/downloads/1.7.25/jruby-bin-1.7.25.tar.gz
Installing minitar >= 0 because the build process needs it.
[bootstrap] Fetching and installing gem: minitar (>= 0)
Fetching: minitar-0.5.4.gem (100%)
@untergeek
untergeek / stdout.json
Created Jan 6, 2016
Beats eats my timestamp with pre-formatted JSON
View stdout.json
2016/01/06 23:15:27.708044 publish.go:100: DBG Publish: {
"@timestamp": "2016-01-06T23:15:23.210Z",
"beat": {
"hostname": "ip-172-31-46-141",
"name": "ip-172-31-46-141"
},
"count": 1,
"fields": null,
"input_type": "log",
"message": "{ \"@timestamp\": \"2016-01-06T23:15:20+00:00\", \"@version\": \"1\", \"vhost\": \"untergeek.com\", \"clientip\": \"80.240.139.191\", \"bytes\": 60375, \"duration\": 0.114, \"status\": 200, \"request\": \"/feed/\", \"method\": \"GET\", \"referrer\": \"-\", \"useragent\": \"Fever/1.39 (Feed Parser; http://feedafever.com; Allow like Gecko)\" }",
@untergeek
untergeek / rspec.output.txt
Created Oct 26, 2015
The plot thickens... This is on a newly installed box, with no other deps installed anywhere.
View rspec.output.txt
failures in bulk class expected behavior
should retry an event until max_retries reached
RSpec::Mocks::MockExpectationError: (LogStash::Outputs::ElasticSearch: {"manage_template"=>true, "index"=>"logstash-2014.11.17", "template_overwrite"=>true, "hosts"=>["127.0.0.1"], "retry_max_items"=>10, "retry_max_interval"=>1, "max_retries"=>3, "codec"=><LogStash::Codecs::Plain charset=>"UTF-8">, "workers"=>1, "template_name"=>"logstash", "flush_size"=>500, "idle_flush_time"=>1, "action"=>"index", "path"=>"/", "ssl"=>false, "ssl_certificate_verification"=>true, "sniffing"=>false, "sniffing_delay"=>5, "doc_as_upsert"=>false, "upsert"=>""}).submit([["index", {:_id=>nil, :_index=>"logstash-2014.11.17", :_type=>"logs", :_routing=>nil}, #<LogStash::Event:0x6ca5d544 @metadata_accessors=#<LogStash::Util::Accessors:0x360bb9b1 @store={"retry_count"=>3}, @lut={}>, @cancelled=false, @data={"somevalue"=>100, "@timestamp"=>"2014-11-17T20:37:17.223Z", "@version"=>"1"}, @metadata={"retry_count"=>3}, @accessors=#<LogStash::Util::Acce
@untergeek
untergeek / 1_logstash.conf
Last active Jan 24, 2017
Logstash heartbeat plugin -> Zabbix monitoring
View 1_logstash.conf
input {
heartbeat {
message => "epoch"
interval => 10
add_field => { "zabbix_host" => "host.example.tld" "zabbix_key" => "ls_heartbeat" }
tags => [ "heartbeat" ]
}
}
filter { }
@untergeek
untergeek / ls-es-template-20150831.json
Created Aug 31, 2015
Logstash -> Elasticsearch Template 2015-08-31
View ls-es-template-20150831.json
{
"template" : "logstash-*",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"_default_" : {
"_all" : {"enabled" : true, "omit_norms" : true},
"dynamic_templates" : [ {
"message_field" : {
@untergeek
untergeek / h.conf
Last active Feb 20, 2017
Logstash Debugging #1
View h.conf
input {
# file {
# path => ["C:/logs/Proj/*/*.log"]
# start_position => beginning
# }
stdin { }
}
@untergeek
untergeek / curator_windows_executable.md
Last active Oct 30, 2017
Build a Windows binary for Curator with Nuitka
View curator_windows_executable.md

Curator Binary Creation (Windows Edition)

Prerequisites

Please install in this order.

Install Python

  • Download 2.7.9 (64-bit)
  • Install with these options:
    • Install for all users
    • Customize Python 2.7.9:
      • Select Add python.exe to Path