Implicit-AWS-Credentials-Via-Logstash.md

Implicit AWS Credentials Test

The provided script is capable of invoking the AWS SDK directly from within the Logstash environment, and is useful for validating the SDK's ability to use implicit credentials (such shared credentials from a discoverable file on disk or credentials made available to an EC2 instance by IMDS or ECS).

It does so by writing an object to an S3 bucket, which the implicit credentials it finds needs to have write access to.

USAGE

Set up your enviroment with at minimum T_AWS_BUCKET and T_AWS_REGION, then invoke the ruby executable that comes with Logstash, providing the above script by path:

export T_AWS_BUCKET="my-test-bucket-name"
export T_AWS_REGION="ca-central-1"
${LOGSTASH_HOME}/bin/ruby -e implicit-aws-s3-credentials-test.rb

Example output:

ubuntu@ip-172-31-0-138:~/wd$ logstash-8.4.3/bin/ruby implicit-aws-s3-credentials-test.rb
Using bundled JDK: /home/ubuntu/wd/logstash-8.4.3/jdk
Loading AWS SDK...
--LOADED AWS SDK version 3.131.6
using ENV[T_AWS_BUCKET] : `my-test-bucket-name`
using ENV[T_AWS_REGION] : `ca-central-1`
using ENV[T_AWS_ENDPOINT] : `https://s3.ca-central-1.amazonaws.com/` (default)
using ENV[T_AWS_SSEKMS_KEY_ID] : <UNSET>
using ENV[T_AWS_SERVER_SIDE_ENCRYPTION] : <UNSET>
using ENV[T_OBJECT_NAME] :  `test-1672269598` (default)
using ENV[T_OBJECT_CONTENTS] : `Test file generated at 2022-12-28 23:19:58 +0000` (default)
SUCCESS: wrote test-1672269598 to my-test-bucket-name@ca-central-1

ADVANCED

It may be helpful to use an environment-initializing shell script to export the variables:

# REQUIRED SETTINGS
export T_AWS_BUCKET="my-test-bucket-name"
export T_AWS_REGION="ca-central-1"

# OPTIONAL SETTINGS (uncomment to activate)
# export T_AWS_ENDPOINT="https://my.aws.endpoint/"
# export T_AWS_SSEKMS_KEY_ID="arn:aws:kms:ca-central-1:REDACTED:key/REDACTED"
# export T_AWS_SERVER_SIDE_ENCRYPTION="aws:kms"
# export T_OBJECT_NAME="my-test"
# export T_OBJECT_CONTENTS="test file contents"

-- test-env.sh

This can be sourced into your current shell before executing the test:

(source test-env.sh && ${LOGSTASH_HOME}/bin/ruby -e implicit-aws-s3-credentials-test.rb)

implicit-aws-s3-credentials-test.rb

##
# Validates implicit credentials by using them to upload a test file to an S3 bucket
#
# Control by setting environment variables.
# DO NOT edit this script directly.
#
# SEE: https://gist.github.com/yaauie/63e7bac0674eb5074341e0a7578fc785 for usage
##

# REQUIRED ENVIRONMENT VARIABLES
aws_bucket =             env_fetch("T_AWS_BUCKET")
aws_region =             env_fetch("T_AWS_REGION")

# OPTIONAL ENVIRONMENT VARIABLES (will use sensible defaults if unset or empty)
aws_endpoint =           env_fetch("T_AWS_ENDPOINT") { "https://s3.#{aws_region}.amazonaws.com/" }
ssekms_key_id =          env_fetch("T_AWS_SSEKMS_KEY_ID") { nil }
server_side_encryption = env_fetch("T_AWS_SERVER_SIDE_ENCRYPTION") { ssekms_key_id ? "aws:kms" : nil }
test_object_name =       env_fetch("T_OBJECT_NAME") { "test-#{Time.now.to_i}" }
test_object_contents =   env_fetch("T_OBJECT_CONTENTS") { "Test file generated at #{Time.now}" }

client_options = {
        endpoint: aws_endpoint,
        region:   aws_region,
}

upload_options = {
  acl:                    "private",
  storage_class:          "STANDARD",
  multipart_threshold:    15 * 1024 * 1024, # 15MB
  ssekms_key_id:          ssekms_key_id,
  server_side_encryption: server_side_encryption,
}.reject {|_,v| v.nil? }

begin
  result = Tempfile.create("test-object-contents") do |tf|
    tf.write(test_object_contents)
    Aws::S3::Bucket.new(aws_bucket, client_options).object(test_object_name).upload_file(tf.path, upload_options)
  end

  if result
    $stderr.puts("SUCCESS: wrote #{test_object_name} to #{aws_bucket}@#{aws_region}")
    exit 0
  else
    $stderr.puts("ERROR: failed to write #{test_object_name} to #{aws_bucket}@#{aws_region}")
    exit 17
  end
rescue => e
  $stderr.puts("ERROR: failed to write #{test_object_name} to #{aws_bucket}@#{aws_region}")
  $stderr.puts("#{e.message} at #{e.backtrace.join}")
  exit 27
end

BEGIN {
  require 'rubygems'
  $stderr.puts("Loading AWS SDK...")
  begin
    require 'aws-sdk-s3'
    $stderr.puts("--LOADED AWS SDK version `#{Aws::CORE_GEM_VERSION}`");
  rescue LoadError
    $stderr.puts("--FAILED to load AWS SDK for S3; loading entire unversioned AWS SDK instead...")
    require "aws-sdk"
    Aws.eager_autoload!
    $stderr.puts("--LOADED AWS SDK version `#{Aws::VERSION}`")
  end


  def env_fetch(name)
    if ENV.include?(name) && !ENV.fetch(name).empty?
      ENV.fetch(name).tap { |value| $stderr.puts("using ENV[#{name}] : `#{value}`") }
    elsif block_given?
      yield.tap do |value|
        $stderr.puts("using ENV[#{name}] : #{value.nil? ? "<UNSET>" : "`#{value}` (default)"}")
      end
    else
      fail "no value for REQUIRED environment variable `#{name}`"
    end
  end
}