Skip to content

Instantly share code, notes, and snippets.

View markwalkom's full-sized avatar

Mark Walkom markwalkom

57 followers · 8 following
  • @warkolm
View GitHub Profile
@markwalkom
markwalkom / gist:3766250c5f6b6206bafcc6c23562b3fc
Created May 4, 2016 08:39
Logstash Grok Pattern for Windows DNS Log files - via https://discuss.elastic.co/t/windows-dns-incompatible-character-encodings/48961/5
WINDNS %{NUMBER:log_date} %{TIME:log_time} %{WORD:dns_thread_id} %{WORD:dns_context}%{SPACE}%{WORD:dns_packet_id} %{WORD:dns_ip_protocol} %{WORD:dns_direction} %{IP:dns_client_address}%{SPACE}%{WORD:dns_xid}%{SPACE}(?:Q|R|U) ?(Q|R|U)?%{SPACE}[%{GREEDYDATA:dns_hex_flags}%{SPACE}%{WORD:dns_response}]%{SPACE}%{WORD:dns_recordtype}%{SPACE}([1-9][0-9]?)%{GREEDYDATA:dns_query_name}
@markwalkom
markwalkom / jqtips.md
Last active May 21, 2016 09:27
jq tips

Elasticsearch

Sum number of docs in a cluster

cat nodes_stats.json|jq '.nodes[].indices.docs.count'|awk '{s+=$0} END {print s}'

Sum total store size

cat nodes_stats.json|jq '.nodes[].indices.store.size_in_bytes'|awk '{s+=$0} END {print s}'

Working with the swapi data

Get a list of planets + key for translate lookup

cat people.json | jq -r '.[]|"\"\(.pk)\"" + ": " + "\"\(.fields.name)\""'

@markwalkom
markwalkom / missing-fields-query.json
Created May 18, 2016 05:13
Via Kibana, only show documents that have a missing field
# Via https://smelloworld.wordpress.com/2016/05/17/missing-fields-search-in-elasticsearch/
{“query”:{“filtered”:{“query”:{“match_all”:{}},”filter”:{“missing”:{“field”:”FIELDNAME”}}}}}
@markwalkom
markwalkom / Step 1 - Elasticsearch and Kibana.md
Last active April 26, 2017 06:46
Monitoring Your Elastic Stack with Beats

Download

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.3.1.tar.gz
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.3.1-darwin-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.3.1.zip

Extract

@markwalkom
markwalkom / Step 1 - Elasticsearch and Kibana.md
Last active April 26, 2017 23:51
Monitoring Your Elastic Stack, with Beats

Download

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.3.1.tar.gz
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.3.1-darwin-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.3.1.zip

Extract

@markwalkom
markwalkom / paypal-transactions.conf
Created July 22, 2017 02:07
Logstash config to process PayPal transaction history (downloaded in CSV)
input {
stdin {}
}
filter {
csv {
columns => ["Date","Time","Time Zone","Name","Type","Status","Currency","Gross","Fee","Net","From Email Address","To Email Address","Transaction ID","Counterparty Status","Shipping address","Address Status","Item Title","Item ID","Shipping and Handling Amount","Compensation Amount","GST","Option 1 Name","Option 1 Value","Option 2 Name","Option 2 Value","Auction Site","Buyer ID","Item URL","Closing Date","Escrow ID","Invoice ID","Reference Txn ID","Invoice Number","Custom Number","Quantity","Receipt ID","Balance","Contact Phone Number"]
add_field => [ "timestamp", "%{Date} %{Time}" ]
remove_field => [ "Date", "Time", "Time Zone" ]
}
date {
@markwalkom
markwalkom / Elasticsearch-5.6.2.xml
Last active September 20, 2019 15:50
Elasticsearch+Kibana 5.6.2 on unRAID 6.3.5
<?xml version="1.0"?>
<Container version="2">
<Name>Elasticsearch-5.6.2</Name>
<Repository>59b11c02b218</Repository>
<Registry>https://docker.elastic.co/</Registry>
<Network>bridge</Network>
<Privileged>false</Privileged>
<Support>https://discuss.elastic.co/c/elasticsearch</Support>
<Overview>Elasticsearch is a open source, distributed, RESTful search and analytics engine.</Overview>
<Category>Tools:</Category>
@markwalkom
markwalkom / 1 - kibana.yml
Created November 10, 2017 10:08
Custom Kibana Regionmap with sample data
# Custom Region Maps
regionmap:
layers:
- name: "Australian States"
url: "http://localhost:8000/aus_state.geojson"
attribution: "exploratory.io"
fields:
- name: "STATE_NAME"
description: "State Name"
@markwalkom
markwalkom / gist:a96e8b40220872610d1098d891e7cff0
Last active January 4, 2020 04:26
Vegemite + Twitter
@elastic OR @logstash OR @elasticsearch OR Elasticsearch OR Logstash OR Kibana OR packetbeat OR "elastic stack" OR "elastic search" OR elasticbeats OR filebeat OR elasticon OR "elk stack” OR swiftype OR auditbeat OR “elastic apm” OR “open source apm” OR elkstack OR belkstack OR opbeat OR “Elastic APM” OR elastic.co OR “elk stack” OR “elastic cloud” OR elastalert OR Swiftype OR Swifttype OR skedlr OR lifeatelastic OR searchguard OR “elastic endpoint” -@Kibana_DRAGON_ -@kibana_love
@markwalkom
markwalkom / README.md
Last active April 21, 2020 20:50
PiHole + Elasticsearch Ingest processing

NOTE - this was specifically built for a docker instance, with the Filebeat docker module collecting the logs.