Skip to content

Instantly share code, notes, and snippets.


Mathieu Martin webmat

View GitHub Profile
webmat /
Last active Mar 20, 2021
Understanding Filebeat modules

Filebeat modules are all either open source, or provided via the Elastic License. You can look at them all, to understand how the parsing, the conversion and the mapping to ECS are done.

Looking for the modules

Looking at the code of the pipelines


The file schema-explorer.ndjson is a dashboard called "Schema Explorer". This dashboard lets you view ECS field definitions from within Kibana.

This requires that you import the ECS csv (found here generated/csv/fields.csv) into Elasticsearch via the Machine Learning Data Visualizer.

webmat / grok_fields.rb
Created Mar 13, 2019
Script to list all fields in Logstash grok patterns
View grok_fields.rb
PATTERNS_GLOB = ENV['PATTERNS_GLOB'] || '~/work/elastic/plugins/logstash-patterns-core/patterns/*'
FIELD_MATCHER = /{\w+:([^}]+)}/
ECS = %w(labels agent client cloud container destination ecs error event file
geo group host http log network observer organization os process
related server service source url user user_agent)
field_names = {}
puts "File name\tField\tLine\tPosition\tConflict"
Dir[PATTERNS_GLOB].each do |file|
webmat / ecs-example.json
Last active Jan 29, 2022
Example ECS event
View ecs-example.json
"@timestamp": "2018-12-07T11:05:07.000Z",
"agent" : {
"type": "filebeat",
"version": "7.0.0-beta1"
"ecs": { "version": "1.0.0" },
"event": {
"action": "get",
"dataset": "apache.access",
webmat /
Last active Apr 28, 2018

Keybase proof

I hereby claim:

  • I am webmat on github.
  • I am webmat ( on keybase.
  • I have a public key ASBxf8fg0XaY8to2x3Ujd9jBhZIDzp_fjMMI33Dlg9Vq-Qo

To claim this, I am signing this object:

View gist:4b8a65cf865fd69fafc8
Understanding the concepts
- The Git Parable, but Tom Preston-Werner
- git is a bucket of commits
- most commits have one parent, and a blob
- some commits have 2 parents (merges)
- some commits have 1 parents (first commit!)
- refs
- branches: pointers to a commit - moves over time
webmat /
Last active Aug 29, 2015
Disk representation of the different MySQL engines

I recently had to detect whether a MySQL server had any tables that didn't support transactions (mainly MyISAM).

The only problem is that this server had close to 5000 indivitual databases, and my usual query didn't cut it:

SELECT  information_schema.TABLES.TABLE_SCHEMA,
FROM information_schema.TABLES
WHERE information_schema.TABLES.TABLE_SCHEMA not in ('performance_schema', 'mysql', 'information_schema')
webmat / Attendees-Example.csv
Last active Dec 29, 2015
Pick a winner out of EventBrite attendees (in CSV format)
View Attendees-Example.csv
Attendee # Date Last Name First Name Email QTY Ticket Type Date Attending Order # Order Type Total Paid (USD) Fees Paid (USD) Eventbrite Fees (USD) CC Processing (USD) Attendee Status Home Address 1 Home Address 2 Home City Home State/Province Home Postcode Home Country
277877635 21 Oct 2013 Martin Mathieu 1 DevOps Enthusiast (80) 4 Nov 2013 218440551 Free Order 0.00 0.00 0.0 0.00 Attending
webmat / steps.txt
Last active Dec 27, 2015
Deploying the Railsbridge message board to Heroku
View steps.txt
# Gemfile
group :production do
gem 'pg'
group :development, :test do
gem 'sqlite3'
# config/application.rb
alias j='jump'
alias m='mark'
alias mm='marks'
alias um='unmark'