yaauie

###############################################################################
# utf8-coerce.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to forcefully coerce string-value field
# to valid UTF-8, preferring a _representational_ transcode operation, and
# falling back to the use of UTF8 replacement characters when encountering byte
# sequences that cannot be represented in unicode, optionally stashing a base64
# encoded copy of the original when such lossy replacements are made.
###############################################################################
#

yaauie

Logstash Ruby Filter Scripts

• Transpose: transpose an array of two-value maps in a source field into an unordered key/value map, optionally storing the result in a target instead of overwriting (e.g., from [{"key":"this","value":"that"}] to {"this" => "that"})
• Untranspose: transpose an unordered key/value map in a source field into an array of two-value maps, optionally storing the result in a target instead of overwriting (e.g., from {"this" => "that"} to [{"key":"this","value":"that"}])
• Flatten Structure: transforms the deeply nested structure of an event or a source field into a flat key-value map by joining nested keys on a configurable separator, optionally storing the result in a target field

yaauie

# encoding: utf-8

require 'thread'
require 'monitor'

##
# The FairEnoughRouter is a generic "fair enough" router. When instantiated
# with a collection of objects, it can be used to select an arbitrary value,
# prioritising ones for which it hasn't recently been exceptional and
# those that are currently less-concurrently used.

yaauie

	\xA7	\xE9
ASCII-8BIT	(Encoding::UndefinedConversionError) "\xA7" from ASCII-8BIT to UTF-8	(Encoding::UndefinedConversionError) "\xE9" from ASCII-8BIT to UTF-8
Big5	(Encoding::InvalidByteSequenceError) incomplete "\xA7" on Big5	(Encoding::InvalidByteSequenceError) incomplete "\xE9" on Big5
Big5-HKSCS	(Encoding::InvalidByteSequenceError) incomplete "\xA7" on Big5-HKSCS	(Encoding::InvalidByteSequenceError) incomplete "\xE9" on Big5-HKSCS
Big5-UAO	(Encoding::InvalidByteSequenceError) incomplete "\xA7" on Big5-UAO	(Encoding::InvalidByteSequenceError) incomplete "\xE9" on Big5-UAO
CESU-8	(Encoding::InvalidByteSequenceError) "\xA7" on CESU-8	(Encoding::InvalidByteSequenceError) incomplete "\xE9" on CESU-8
CP51932	(Encoding::InvalidByteSequenceError) incomplete "\xA7" on CP51932	(Encoding::InvalidByteSequenceError) incomplete "\xE9" on CP51932
CP850	º	Ú
CP852	ž	Ú

yaauie

Implicit AWS Credentials Test

The provided script is capable of invoking the AWS SDK directly from within the Logstash environment, and is useful for validating the SDK's ability to use implicit credentials (such shared credentials from a discoverable file on disk or credentials made available to an EC2 instance by IMDS or ECS).

It does so by writing an object to an S3 bucket, which the implicit credentials it finds needs to have write access to.

USAGE

yaauie

###############################################################################
# determine-field-type.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to determine a field's type
###############################################################################
#
# Copyright 2022 Ry Biesemeyer
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal

yaauie

literal: 👍
u-encoded: \u001c\u{1F44D}
inspected: "\u001C👍"
dumped: "\x1C\u{1F44D}"
json-encoced: "\u001c👍"

yaauie

#!/bin/sh
#
# Utility for determining why we cannot list the
# contents of a deeply-nested directory.
#
# Usage:
#  listup.sh /deeply/nested/path
#
##############################################################################
# Copyright 2022 Ry Biesemeyer

yaauie

###############################################################################
# flatten-structure.logstash-filter-ruby.rb
# ---------------------------------
# A script for a Logstash Ruby Filter to flatten a nested structure to produce
# flat structure whose keys are the paths of the previous structure
###############################################################################
#
# Copyright 2022 Ry Biesemeyer
#
# Permission is hereby granted, free of charge, to any person obtaining a copy

yaauie

We have had some success using LS-to-LS over HTTP(S), which supports an HTTP(s) Load Balancer or Proxy in the middle, and can be secured with TLS/SSL. It can be made to be quite performant, but doing so requires some specific tuning.

Upstream (HTTP Output)

The upstream pipelie would contain a single HTTP output plugin aimed either directly at a downstream Logstash or at a Load Balancer, importantly configured with:

• format => json_batch (for performance; without this one event will be sent at a time) and
• retry_non_idempotent => true (for resilience; without this, some failures cannot be safely retried).

Depending on whether we ar sending directly to another Logstash or through an SSL-terminating Load Balancer or proxy, the output may need to be configured

• with HTTP Basic credentials (user/password),