Skip to content

Instantly share code, notes, and snippets.

View markwalkom's full-sized avatar

Mark Walkom markwalkom

View GitHub Profile
@markwalkom
markwalkom / logstash.conf
Last active April 29, 2022 10:23
Reindexing Elasticsearch with Logstash 2.0
View logstash.conf
input {
elasticsearch {
hosts => [ "HOSTNAME_HERE" ]
port => "9200"
index => "INDEXNAME_HERE"
size => 1000
scroll => "5m"
docinfo => true
scan => true
}
View gist:3766250c5f6b6206bafcc6c23562b3fc
WINDNS %{NUMBER:log_date} %{TIME:log_time} %{WORD:dns_thread_id} %{WORD:dns_context}%{SPACE}%{WORD:dns_packet_id} %{WORD:dns_ip_protocol} %{WORD:dns_direction} %{IP:dns_client_address}%{SPACE}%{WORD:dns_xid}%{SPACE}(?:Q|R|U) ?(Q|R|U)?%{SPACE}[%{GREEDYDATA:dns_hex_flags}%{SPACE}%{WORD:dns_response}]%{SPACE}%{WORD:dns_recordtype}%{SPACE}([1-9][0-9]?)%{GREEDYDATA:dns_query_name}
@markwalkom
markwalkom / README.md
Last active April 21, 2020 20:50
PiHole + Elasticsearch Ingest processing
View README.md

NOTE - this was specifically built for a docker instance, with the Filebeat docker module collecting the logs.

@markwalkom
markwalkom / Beats.md
Last active March 9, 2020 19:05
LCA2019 - Awesome Monitoring Infrastructure Using the Elastic Stack
View Beats.md
View gist:a96e8b40220872610d1098d891e7cff0
@elastic OR @logstash OR @elasticsearch OR Elasticsearch OR Logstash OR Kibana OR packetbeat OR "elastic stack" OR "elastic search" OR elasticbeats OR filebeat OR elasticon OR "elk stack” OR swiftype OR auditbeat OR “elastic apm” OR “open source apm” OR elkstack OR belkstack OR opbeat OR “Elastic APM” OR elastic.co OR “elk stack” OR “elastic cloud” OR elastalert OR Swiftype OR Swifttype OR skedlr OR lifeatelastic OR searchguard OR “elastic endpoint” -@Kibana_DRAGON_ -@kibana_love
@markwalkom
markwalkom / Elasticsearch-5.6.2.xml
Last active September 20, 2019 15:50
Elasticsearch+Kibana 5.6.2 on unRAID 6.3.5
View Elasticsearch-5.6.2.xml
<?xml version="1.0"?>
<Container version="2">
<Name>Elasticsearch-5.6.2</Name>
<Repository>59b11c02b218</Repository>
<Registry>https://docker.elastic.co/</Registry>
<Network>bridge</Network>
<Privileged>false</Privileged>
<Support>https://discuss.elastic.co/c/elasticsearch</Support>
<Overview>Elasticsearch is a open source, distributed, RESTful search and analytics engine.</Overview>
<Category>Tools:</Category>
@markwalkom
markwalkom / crlf.csv
Created February 17, 2019 23:46
CSV file with CRLF
View crlf.csv
field0 field1 fieldcrlf field3 field4
hello 1234 this is a line feed 3.0 AYX
View BOM-json.py
from ftplib import FTP
import os
import xml.etree.ElementTree as ET
import xmltodict
import json
url = 'ftp.bom.gov.au'
filename = 'IDN65068.xml'
def writeline(data):
@markwalkom
markwalkom / 1 - kibana.yml
Created November 10, 2017 10:08
Custom Kibana Regionmap with sample data
View 1 - kibana.yml
# Custom Region Maps
regionmap:
layers:
- name: "Australian States"
url: "http://localhost:8000/aus_state.geojson"
attribution: "exploratory.io"
fields:
- name: "STATE_NAME"
description: "State Name"
@markwalkom
markwalkom / paypal-transactions.conf
Created July 22, 2017 02:07
Logstash config to process PayPal transaction history (downloaded in CSV)
View paypal-transactions.conf
input {
stdin {}
}
filter {
csv {
columns => ["Date","Time","Time Zone","Name","Type","Status","Currency","Gross","Fee","Net","From Email Address","To Email Address","Transaction ID","Counterparty Status","Shipping address","Address Status","Item Title","Item ID","Shipping and Handling Amount","Compensation Amount","GST","Option 1 Name","Option 1 Value","Option 2 Name","Option 2 Value","Auction Site","Buyer ID","Item URL","Closing Date","Escrow ID","Invoice ID","Reference Txn ID","Invoice Number","Custom Number","Quantity","Receipt ID","Balance","Contact Phone Number"]
add_field => [ "timestamp", "%{Date} %{Time}" ]
remove_field => [ "Date", "Time", "Time Zone" ]
}
date {