Skip to content

Instantly share code, notes, and snippets.

View markwalkom's full-sized avatar

Mark Walkom markwalkom

View GitHub Profile
markwalkom / logstash.conf
Last active April 29, 2022 10:23
Reindexing Elasticsearch with Logstash 2.0
input {
elasticsearch {
hosts => [ "HOSTNAME_HERE" ]
port => "9200"
size => 1000
scroll => "5m"
docinfo => true
scan => true
WINDNS %{NUMBER:log_date} %{TIME:log_time} %{WORD:dns_thread_id} %{WORD:dns_context}%{SPACE}%{WORD:dns_packet_id} %{WORD:dns_ip_protocol} %{WORD:dns_direction} %{IP:dns_client_address}%{SPACE}%{WORD:dns_xid}%{SPACE}(?:Q|R|U) ?(Q|R|U)?%{SPACE}[%{GREEDYDATA:dns_hex_flags}%{SPACE}%{WORD:dns_response}]%{SPACE}%{WORD:dns_recordtype}%{SPACE}([1-9][0-9]?)%{GREEDYDATA:dns_query_name}
markwalkom /
Last active April 21, 2020 20:50
PiHole + Elasticsearch Ingest processing

NOTE - this was specifically built for a docker instance, with the Filebeat docker module collecting the logs.

markwalkom /
Last active March 9, 2020 19:05
LCA2019 - Awesome Monitoring Infrastructure Using the Elastic Stack
@elastic OR @logstash OR @elasticsearch OR Elasticsearch OR Logstash OR Kibana OR packetbeat OR "elastic stack" OR "elastic search" OR elasticbeats OR filebeat OR elasticon OR "elk stack” OR swiftype OR auditbeat OR “elastic apm” OR “open source apm” OR elkstack OR belkstack OR opbeat OR “Elastic APM” OR OR “elk stack” OR “elastic cloud” OR elastalert OR Swiftype OR Swifttype OR skedlr OR lifeatelastic OR searchguard OR “elastic endpoint” -@Kibana_DRAGON_ -@kibana_love
markwalkom / Elasticsearch-5.6.2.xml
Last active September 20, 2019 15:50
Elasticsearch+Kibana 5.6.2 on unRAID 6.3.5
<?xml version="1.0"?>
<Container version="2">
<Overview>Elasticsearch is a open source, distributed, RESTful search and analytics engine.</Overview>
markwalkom / crlf.csv
Created February 17, 2019 23:46
CSV file with CRLF
field0 field1 fieldcrlf field3 field4
hello 1234 this is a line feed 3.0 AYX
from ftplib import FTP
import os
import xml.etree.ElementTree as ET
import xmltodict
import json
url = ''
filename = 'IDN65068.xml'
def writeline(data):
markwalkom / 1 - kibana.yml
Created November 10, 2017 10:08
Custom Kibana Regionmap with sample data
# Custom Region Maps
- name: "Australian States"
url: "http://localhost:8000/aus_state.geojson"
attribution: ""
- name: "STATE_NAME"
description: "State Name"
markwalkom / paypal-transactions.conf
Created July 22, 2017 02:07
Logstash config to process PayPal transaction history (downloaded in CSV)
input {
stdin {}
filter {
csv {
columns => ["Date","Time","Time Zone","Name","Type","Status","Currency","Gross","Fee","Net","From Email Address","To Email Address","Transaction ID","Counterparty Status","Shipping address","Address Status","Item Title","Item ID","Shipping and Handling Amount","Compensation Amount","GST","Option 1 Name","Option 1 Value","Option 2 Name","Option 2 Value","Auction Site","Buyer ID","Item URL","Closing Date","Escrow ID","Invoice ID","Reference Txn ID","Invoice Number","Custom Number","Quantity","Receipt ID","Balance","Contact Phone Number"]
add_field => [ "timestamp", "%{Date} %{Time}" ]
remove_field => [ "Date", "Time", "Time Zone" ]
date {