Skip to content

Instantly share code, notes, and snippets.

Avatar

Ry Biesemeyer yaauie

View GitHub Profile
@yaauie
yaauie / USAGE.md
Last active Oct 28, 2021
Proof-of-concept high-precision timestamp parser for Logstash, using the Logstash Ruby Filter and Ruby's Time object.
View USAGE.md
filter {
  ruby {
    path => "${PWD}/precision-timestamp-parse.logstash-filter-ruby.rb"
    script_params => {
      source => "precise-timestamp-field"
      format => "ISO8601"
    }
  }
}
@yaauie
yaauie / pin-racc-pending-release-of-1.6-artifact-for-java.patch
Created Oct 19, 2021
RACC 1.6 was released without an accompanying java artifact for jruby. This patch pins racc in logstash-core.
View pin-racc-pending-release-of-1.6-artifact-for-java.patch
diff --git a/logstash-core/logstash-core.gemspec b/logstash-core/logstash-core.gemspec
index e22a9f289..a4563498a 100644
--- a/logstash-core/logstash-core.gemspec
+++ b/logstash-core/logstash-core.gemspec
@@ -78,4 +78,8 @@ Gem::Specification.new do |gem|
gem.add_dependency 'down', '~> 5.2.0' #(MIT license)
gem.add_dependency 'tzinfo-data' #(MIT license)
gem.add_dependency 'rufus-scheduler' #(MIT license)
+
+ # TEMPORARY: racc-1.6.0 doesn't have JAVA counterpart (yet)
@yaauie
yaauie / transform-field-names.logstash-filter-ruby.rb
Created Sep 28, 2021
A script for the Logstash Ruby Filter to transform field names, possibly recursively.
View transform-field-names.logstash-filter-ruby.rb
###############################################################################
# transform-field-names.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to transform field names, possibly
# recursively.
###############################################################################
#
# Copyright 2020 Ry Biesemeyer
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
@yaauie
yaauie / estimate-serialized-size.logstash-filter-ruby.rb
Last active Sep 27, 2021
Logstash Ruby Filter Script for estimating the serialized size of an event or one of its fields.
View estimate-serialized-size.logstash-filter-ruby.rb
###############################################################################
# estimate-serialized-size.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to estimate the serialized size of an
# event or one of its fields.
###############################################################################
#
# Copyright 2021 Ry Biesemeyer
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
View noisy-proxy.rb
require 'time' # Time#iso8601
# A NoisyProxy will log all method calls on the wrapped object and their results to the given IO.
# It will optionally _infect_ any object returned by the real method.
class NoisyProxy < BasicObject
def initialize(inner, infect:false, io:$stderr)
@inner = inner
@infect = infect
@io = io
end
View USAGE.md

Suppose you had events with the following structure:

{
  "books": [
    {"author":"Alice",  "title":"Fields" },
    {"author":"Bob",    "title":"Oceans" },
    {"author":"Connie", "title":"Rivers" },
    {"author":"Connie", "title":"Lakes"  },
    {"author":"David",  "title":"Streams"},
@yaauie
yaauie / USAGE.md
Created Mar 3, 2021
Logstash script for ruby filter plugin to destructively merge two array fields
View USAGE.md

This script has three parameters:

  • source: a field reference to the source array
  • target: a field reference to the target array
  • coerce: (optional: default false) when encountering non-array values in either source or target fields, setting coerce => true causes these fields to first be converted to a single-entry array.
filter {
@yaauie
yaauie / logstash-codec-csv.auto-flush-on-evict.patch
Last active Sep 28, 2021
Patch Logstash CSV Codec 1.0.0 to ensure the Logstash File Input re-detects column names per file (NOT NECESSARY with logstash-codec-multiline >= 3.1.1)
View logstash-codec-csv.auto-flush-on-evict.patch
diff --git a/lib/logstash/codecs/csv.rb b/lib/logstash/codecs/csv.rb
index 07d6416..66cd6ed 100644
--- a/lib/logstash/codecs/csv.rb
+++ b/lib/logstash/codecs/csv.rb
@@ -133,12 +133,19 @@ class LogStash::Codecs::CSV < LogStash::Codecs::Base
rescue CSV::MalformedCSVError => e
@logger.error("CSV parse failure. Falling back to plain-text", :error => e, :data => data)
yield LogStash::Event.new("message" => data, "tags" => ["_csvparsefailure"])
end
end
@yaauie
yaauie / downcase-field-names.logstash-filter-ruby.rb
Created Oct 14, 2020
A script for a Logstash Ruby Filter to transform field names to all lowercase
View downcase-field-names.logstash-filter-ruby.rb
###############################################################################
# downcase-field-names.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to transform field names to all lowercase
###############################################################################
#
# Copyright 2020 Ry Biesemeyer
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
View stringify-timestamp.logstash-filter-ruby.rb
###############################################################################
# stringify-timestamp.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to stringify logstash timestamps using
# ISO-8601.
###############################################################################
#
# Copyright 2020 Ry Biesemeyer
#
# Permission is hereby granted, free of charge, to any person obtaining a copy