Skip to content

Instantly share code, notes, and snippets.

View markwalkom's full-sized avatar

Mark Walkom markwalkom

57 followers · 8 following
  • @warkolm
View GitHub Profile
@markwalkom
markwalkom / missing-fields-query.json
Created May 18, 2016 05:13
Via Kibana, only show documents that have a missing field
# Via https://smelloworld.wordpress.com/2016/05/17/missing-fields-search-in-elasticsearch/
{“query”:{“filtered”:{“query”:{“match_all”:{}},”filter”:{“missing”:{“field”:”FIELDNAME”}}}}}
@markwalkom
markwalkom / jqtips.md
Last active May 21, 2016 09:27
jq tips

Elasticsearch

Sum number of docs in a cluster

cat nodes_stats.json|jq '.nodes[].indices.docs.count'|awk '{s+=$0} END {print s}'

Sum total store size

cat nodes_stats.json|jq '.nodes[].indices.store.size_in_bytes'|awk '{s+=$0} END {print s}'

Working with the swapi data

Get a list of planets + key for translate lookup

cat people.json | jq -r '.[]|"\"\(.pk)\"" + ": " + "\"\(.fields.name)\""'

@markwalkom
markwalkom / gist:3766250c5f6b6206bafcc6c23562b3fc
Created May 4, 2016 08:39
Logstash Grok Pattern for Windows DNS Log files - via https://discuss.elastic.co/t/windows-dns-incompatible-character-encodings/48961/5
WINDNS %{NUMBER:log_date} %{TIME:log_time} %{WORD:dns_thread_id} %{WORD:dns_context}%{SPACE}%{WORD:dns_packet_id} %{WORD:dns_ip_protocol} %{WORD:dns_direction} %{IP:dns_client_address}%{SPACE}%{WORD:dns_xid}%{SPACE}(?:Q|R|U) ?(Q|R|U)?%{SPACE}[%{GREEDYDATA:dns_hex_flags}%{SPACE}%{WORD:dns_response}]%{SPACE}%{WORD:dns_recordtype}%{SPACE}([1-9][0-9]?)%{GREEDYDATA:dns_query_name}
@markwalkom
markwalkom / Logstash all
Created February 5, 2016 03:44
$ logstash-2.2.0/bin/plugin list
logstash-codec-avro
logstash-codec-cef
logstash-codec-cloudfront
logstash-codec-cloudtrail
logstash-codec-collectd
logstash-codec-compress_spooler
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
@markwalkom
markwalkom / gist:cd8b4a9f82c442079284
Created December 28, 2015 21:48
fail2ban patterns
F2B_DATE %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[ ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})
F2B_ACTION (\w+)\.(?:\w+)(\s+)?\:
F2B_JAIL \[(?<jail>\w+\-?\w+?)\]
F2B_LEVEL (?<level>\w+)\s+
@markwalkom
markwalkom / README.md
Last active July 25, 2016 14:46
CollectD to ELK

This is an example of using ELK to parse and view collectd data.

Caveat - I haven't fully tested this mapping yet, it doesn't take into account any other fields that may be added with other collectd plugins, just the ones I have specified below.

@markwalkom
markwalkom / gist:f47a30e37cd402f2dc5d
Last active August 29, 2015 14:21
Export from ES to a json file
input {
elasticsearch {
hosts => [ "HOSTNAME_HERE" ]
port => "9200"
index => "INDEXNAME_HERE"
size => 500
scroll => "5m"
}
}
output {
@markwalkom
markwalkom / logstash.conf
Last active April 29, 2022 10:23
Reindexing Elasticsearch with Logstash 2.0
input {
elasticsearch {
hosts => [ "HOSTNAME_HERE" ]
port => "9200"
index => "INDEXNAME_HERE"
size => 1000
scroll => "5m"
docinfo => true
scan => true
}
@markwalkom
markwalkom / es-1.2-settings.md
Last active August 29, 2015 14:08 — forked from jprante/es-1.2-settings.md
@markwalkom
markwalkom / keybase.md
Created August 17, 2014 08:08
keybase.md

Keybase proof

I hereby claim:

  • I am markwalkom on github.
  • I am markwalkom (https://keybase.io/markwalkom) on keybase.
  • I have a public key whose fingerprint is 3624 D73D 1018 8785 6475 F84D 5CA5 78CB 1845 5C92

To claim this, I am signing this object: