Skip to content

Instantly share code, notes, and snippets.

Avatar

Mathieu Martin webmat

View GitHub Profile
@webmat
webmat / README.md
Last active Mar 20, 2021
Understanding Filebeat modules
View README.md

Filebeat modules are all either open source, or provided via the Elastic License. You can look at them all, to understand how the parsing, the conversion and the mapping to ECS are done.

Looking for the modules

Looking at the code of the pipelines

View README.md

The file schema-explorer.ndjson is a dashboard called "Schema Explorer". This dashboard lets you view ECS field definitions from within Kibana.

This requires that you import the ECS csv (found here generated/csv/fields.csv) into Elasticsearch via the Machine Learning Data Visualizer.

@webmat
webmat / grok_fields.rb
Created Mar 13, 2019
Script to list all fields in Logstash grok patterns
View grok_fields.rb
PATTERNS_GLOB = ENV['PATTERNS_GLOB'] || '~/work/elastic/plugins/logstash-patterns-core/patterns/*'
FIELD_MATCHER = /{\w+:([^}]+)}/
ECS = %w(labels agent client cloud container destination ecs error event file
geo group host http log network observer organization os process
related server service source url user user_agent)
field_names = {}
puts "File name\tField\tLine\tPosition\tConflict"
Dir[PATTERNS_GLOB].each do |file|
@webmat
webmat / ecs-example.json
Last active Apr 8, 2021
Example ECS event
View ecs-example.json
{
"@timestamp": "2018-12-07T11:05:07.000Z",
"agent" : {
"type": "filebeat",
"version": "7.0.0-beta1"
},
"ecs": { "version": "1.0.0" },
"event": {
"action": "get",
"dataset": "apache.access",
@webmat
webmat / keybase.md
Last active Apr 28, 2018
keybase.md
View keybase.md

Keybase proof

I hereby claim:

  • I am webmat on github.
  • I am webmat (https://keybase.io/webmat) on keybase.
  • I have a public key ASBxf8fg0XaY8to2x3Ujd9jBhZIDzp_fjMMI33Dlg9Vq-Qo

To claim this, I am signing this object:

View gist:4b8a65cf865fd69fafc8
Understanding the concepts
- The Git Parable, but Tom Preston-Werner
http://tom.preston-werner.com/2009/05/19/the-git-parable.html
- git is a bucket of commits
- most commits have one parent, and a blob
- some commits have 2 parents (merges)
- some commits have 1 parents (first commit!)
- refs
- branches: pointers to a commit - moves over time
@webmat
webmat / readme.md
Last active Aug 29, 2015
Disk representation of the different MySQL engines
View readme.md

I recently had to detect whether a MySQL server had any tables that didn't support transactions (mainly MyISAM).

The only problem is that this server had close to 5000 indivitual databases, and my usual query didn't cut it:

SELECT  information_schema.TABLES.TABLE_SCHEMA,
        information_schema.TABLES.TABLE_NAME,
        information_schema.TABLES.ENGINE,
FROM information_schema.TABLES
WHERE information_schema.TABLES.TABLE_SCHEMA not in ('performance_schema', 'mysql', 'information_schema')
@webmat
webmat / Attendees-Example.csv
Last active Dec 29, 2015
Pick a winner out of EventBrite attendees (in CSV format)
View Attendees-Example.csv
Attendee # Date Last Name First Name Email QTY Ticket Type Date Attending Order # Order Type Total Paid (USD) Fees Paid (USD) Eventbrite Fees (USD) CC Processing (USD) Attendee Status Home Address 1 Home Address 2 Home City Home State/Province Home Postcode Home Country
277877635 21 Oct 2013 Martin Mathieu webmat@gmail.com 1 DevOps Enthusiast (80) 4 Nov 2013 218440551 Free Order 0.00 0.00 0.0 0.00 Attending
@webmat
webmat / steps.txt
Last active Dec 27, 2015
Deploying the Railsbridge message board to Heroku
View steps.txt
# Gemfile
group :production do
gem 'pg'
end
group :development, :test do
gem 'sqlite3'
end
# config/application.rb
View jump-zsh-shortcuts.sh
alias j='jump'
alias m='mark'
alias mm='marks'
alias um='unmark'