Skip to content

Instantly share code, notes, and snippets.

Avatar

Mark Walkom markwalkom

View GitHub Profile
@markwalkom
markwalkom / crlf.csv
Created Feb 17, 2019
CSV file with CRLF
View crlf.csv
field0 field1 fieldcrlf field3 field4
hello 1234 this is a line feed 3.0 AYX
@markwalkom
markwalkom / Beats.md
Last active Mar 9, 2020
LCA2019 - Awesome Monitoring Infrastructure Using the Elastic Stack
View Beats.md
View BOM-json.py
from ftplib import FTP
import os
import xml.etree.ElementTree as ET
import xmltodict
import json
url = 'ftp.bom.gov.au'
filename = 'IDN65068.xml'
def writeline(data):
@markwalkom
markwalkom / README.md
Last active Apr 21, 2020
PiHole + Elasticsearch Ingest processing
View README.md

NOTE - this was specifically built for a docker instance, with the Filebeat docker module collecting the logs.

View gist:a96e8b40220872610d1098d891e7cff0
@elastic OR @logstash OR @elasticsearch OR Elasticsearch OR Logstash OR Kibana OR packetbeat OR "elastic stack" OR "elastic search" OR elasticbeats OR filebeat OR elasticon OR "elk stack” OR swiftype OR auditbeat OR “elastic apm” OR “open source apm” OR elkstack OR belkstack OR opbeat OR “Elastic APM” OR elastic.co OR “elk stack” OR “elastic cloud” OR elastalert OR Swiftype OR Swifttype OR skedlr OR lifeatelastic OR searchguard OR “elastic endpoint” -@Kibana_DRAGON_ -@kibana_love
@markwalkom
markwalkom / 1 - kibana.yml
Created Nov 10, 2017
Custom Kibana Regionmap with sample data
View 1 - kibana.yml
# Custom Region Maps
regionmap:
layers:
- name: "Australian States"
url: "http://localhost:8000/aus_state.geojson"
attribution: "exploratory.io"
fields:
- name: "STATE_NAME"
description: "State Name"
@markwalkom
markwalkom / Elasticsearch-5.6.2.xml
Last active Sep 20, 2019
Elasticsearch+Kibana 5.6.2 on unRAID 6.3.5
View Elasticsearch-5.6.2.xml
<?xml version="1.0"?>
<Container version="2">
<Name>Elasticsearch-5.6.2</Name>
<Repository>59b11c02b218</Repository>
<Registry>https://docker.elastic.co/</Registry>
<Network>bridge</Network>
<Privileged>false</Privileged>
<Support>https://discuss.elastic.co/c/elasticsearch</Support>
<Overview>Elasticsearch is a open source, distributed, RESTful search and analytics engine.</Overview>
<Category>Tools:</Category>
@markwalkom
markwalkom / paypal-transactions.conf
Created Jul 22, 2017
Logstash config to process PayPal transaction history (downloaded in CSV)
View paypal-transactions.conf
input {
stdin {}
}
filter {
csv {
columns => ["Date","Time","Time Zone","Name","Type","Status","Currency","Gross","Fee","Net","From Email Address","To Email Address","Transaction ID","Counterparty Status","Shipping address","Address Status","Item Title","Item ID","Shipping and Handling Amount","Compensation Amount","GST","Option 1 Name","Option 1 Value","Option 2 Name","Option 2 Value","Auction Site","Buyer ID","Item URL","Closing Date","Escrow ID","Invoice ID","Reference Txn ID","Invoice Number","Custom Number","Quantity","Receipt ID","Balance","Contact Phone Number"]
add_field => [ "timestamp", "%{Date} %{Time}" ]
remove_field => [ "Date", "Time", "Time Zone" ]
}
date {
@markwalkom
markwalkom / expressvpn+billion-setup.asciidoc
Last active Nov 13, 2019
ExpressVPN + Billion Router Setup
View expressvpn+billion-setup.asciidoc

ExpressVPN doesn’t document how to do this, but here’s what worked for me on a bipac-8700vax.

First we need to grab some details to let us set things up

  1. Login to your account on the ExpressVPN site

  2. Go to My Subscriptions > Set up ExpressVPN, it should take you to https://www.expressvpn.com/setup#manual

  3. Go to Manual Config > PPTP & L2TP-IPSec

  4. Copy the Username, Password

  5. Select the region you are in/want to connect to and copy the hostname. eg la3-ubuntu-l2tp.expressprovider.com

@markwalkom
markwalkom / Step 1 - Elasticsearch and Kibana.md
Last active Apr 26, 2017
Monitoring Your Elastic Stack, with Beats
View Step 1 - Elasticsearch and Kibana.md

Download

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.3.1.tar.gz
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.3.1-darwin-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.3.1.zip

Extract