yaauie

We have had some success using LS-to-LS over HTTP(S), which supports an HTTP(s) Load Balancer or Proxy in the middle, and can be secured with TLS/SSL. It can be made to be quite performant, but doing so requires some specific tuning.

Upstream (HTTP Output)

The upstream pipelie would contain a single HTTP output plugin aimed either directly at a downstream Logstash or at a Load Balancer, importantly configured with:

• format => json_batch (for performance; without this one event will be sent at a time) and
• retry_non_idempotent => true (for resilience; without this, some failures cannot be safely retried).

Depending on whether we ar sending directly to another Logstash or through an SSL-terminating Load Balancer or proxy, the output may need to be configured

• with HTTP Basic credentials (user/password),

yaauie

package org.logstash.instrument.metrics.timer;

import org.logstash.instrument.metrics.AbstractMetric;

import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.LongSupplier;
import java.util.function.Supplier;

/**

yaauie

###############################################################################
# replace-with-serialzied.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to replace the event's contents with a
# single field containing a string JSON-encoded representation of the event.
#
# This filter _MUTATES_ the event, removing all DATA-keys while leaving METADATA
# in-tact.
#
###############################################################################

yaauie

The included apply-template.rb provides a way generate Logstash config fragments from a shared template.

This can be useful for shared verbose configuration that is shared across multiple pipelines.

For example, if we are using multiple pipelines with pipelines.yml

 - pipeline.id: one
   path.config: "${LOGSTASH_HOME}/pipelines/one/*.conf"
 - pipeline.id: two

yaauie

╭─{ yaauie@limbo:~/src/elastic/ls (✔ 8.0) }
╰─● (rbenv shell jruby-9.2.19.0; ruby kwargs-splat.rb)
ArgumentError: wrong number of arguments calling `initialize` (given 1, expected 0)
     new at kwargs-splat.rb:7
  <main> at kwargs-splat.rb:17
( rbenv shell jruby-9.2.19.0; ruby kwargs-splat.rb; )  9.23s user 0.53s system 405% cpu 2.412 total
[error: 1]

yaauie

###############################################################################
# json-sanitize-field-names.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to transform a JSON string so that the
# resulting JSON string's decoded representation does not contain square
# brackets in keys.
#
# This filter does NOT parse the JSON string into an Object, and has undefined
# behaviour when the string is not valid JSON.
#

yaauie

filter {
  ruby {
    path => "${PWD}/precision-timestamp-parse.logstash-filter-ruby.rb"
    script_params => {
      source => "precise-timestamp-field"
      format => "ISO8601"
    }
  }
}

yaauie

diff --git a/logstash-core/logstash-core.gemspec b/logstash-core/logstash-core.gemspec
index e22a9f289..a4563498a 100644
--- a/logstash-core/logstash-core.gemspec
+++ b/logstash-core/logstash-core.gemspec
@@ -78,4 +78,8 @@ Gem::Specification.new do |gem|
   gem.add_dependency 'down', '~> 5.2.0' #(MIT license)
   gem.add_dependency 'tzinfo-data' #(MIT license)
   gem.add_dependency 'rufus-scheduler' #(MIT license)
+
+  # TEMPORARY: racc-1.6.0 doesn't have JAVA counterpart (yet)

yaauie

###############################################################################
# transform-field-names.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to transform field names, possibly
# recursively.
###############################################################################
#
# Copyright 2020 Ry Biesemeyer
#
# Permission is hereby granted, free of charge, to any person obtaining a copy

yaauie

###############################################################################
# estimate-serialized-size.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to estimate the serialized size of an
# event or one of its fields.
###############################################################################
#
# Copyright 2021 Ry Biesemeyer
#
# Permission is hereby granted, free of charge, to any person obtaining a copy