webmat

Filebeat modules are all either open source, or provided via the Elastic License. You can look at them all, to understand how the parsing, the conversion and the mapping to ECS are done.

Looking for the modules

• All Filebeat modules are listed here: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html
• The code for Apache 2 open source modules is here: https://github.com/elastic/beats/tree/master/filebeat
• The code for Elastic Licensed modules is here: https://github.com/elastic/beats/tree/master/x-pack/filebeat/module

Looking at the code of the pipelines

webmat

{
  "@timestamp": "2018-12-07T11:05:07.000Z",
  "agent" : {
    "type": "filebeat",
    "version": "7.0.0-beta1"
  },
  "ecs": { "version": "1.0.0" },
  "event": {
    "action": "get",
    "dataset": "apache.access",

webmat

#!/usr/bin/env ruby

# Temporarily add this dir to your path
#   export PATH=~/gists/bundle_benchmark:$PATH
# cd to any of your project and run the benchmark
#   cd ~/my-project
#   benchmark.rb

require 'benchmark'
require 'bundler'

webmat

PATTERNS_GLOB = ENV['PATTERNS_GLOB'] || '~/work/elastic/plugins/logstash-patterns-core/patterns/*'

FIELD_MATCHER = /{\w+:([^}]+)}/
ECS = %w(labels agent client cloud container destination ecs error event file
         geo group host http log network observer organization os process
         related server service source url user user_agent)

field_names = {}
puts "File name\tField\tLine\tPosition\tConflict"
Dir[PATTERNS_GLOB].each do |file|

webmat

ActiveAdmin::Dashboards.build do
  # Add this section in your dashboard...

  section "Background Jobs" do
    now = Time.now.getgm
    ul do
      li do
        jobs = Delayed::Job.where('failed_at is not null').count(:id)
        link_to "#{jobs} failing jobs", admin_jobs_path(q: {failed_at_is_not_null: true}), style: 'color: red'
      end

webmat

The file schema-explorer.ndjson is a dashboard called "Schema Explorer". This dashboard lets you view ECS field definitions from within Kibana.

This requires that you import the ECS csv (found here generated/csv/fields.csv) into Elasticsearch via the Machine Learning Data Visualizer.

webmat

Keybase proof

I hereby claim:

• I am webmat on github.
• I am webmat (https://keybase.io/webmat) on keybase.
• I have a public key ASBxf8fg0XaY8to2x3Ujd9jBhZIDzp_fjMMI33Dlg9Vq-Qo

To claim this, I am signing this object:

webmat

webmat

# Gemfile
group :production do
  gem 'pg'
end

group :development, :test do
  gem 'sqlite3'
end

# config/application.rb

webmat

alias   j='jump'
alias   m='mark'
alias  mm='marks'
alias  um='unmark'