imweijh

require "java"
require "rubygems"
require "sequel"
require "sqljdbc4.jar" #this files path may need to be relevate to file location

DB = Sequel.connect(
  	:adapter => "jdbc",
		:url => "jdbc:sqlserver://doric-server5;database=ADCData_Doric;integratedSecurity=true"
    # This is using integrated security, might want to pass username and password
    # More Info at http://sequel.rubyforge.org/rdoc/files/doc/opening_databases_rdoc.html

imweijh

http {
  log_format bodylog '$remote_addr - $remote_user [$time_local] '
      '"$request" $status $body_bytes_sent '
      '"$http_referer" "$http_user_agent" $request_time '
      '<"$request_body" >"$resp_body"';

  lua_need_request_body on;

  set $resp_body "";
  body_filter_by_lua '

imweijh

We have had some success using LS-to-LS over HTTP(S), which supports an HTTP(s) Load Balancer or Proxy in the middle, and can be secured with TLS/SSL. It can be made to be quite performant, but doing so requires some specific tuning.

Upstream (HTTP Output)

The upstream pipelie would contain a single HTTP output plugin aimed either directly at a downstream Logstash or at a Load Balancer, importantly configured with:

• format => json_batch (for performance; without this one event will be sent at a time) and
• retry_non_idempotent => true (for resilience; without this, some failures cannot be safely retried).

Depending on whether we ar sending directly to another Logstash or through an SSL-terminating Load Balancer or proxy, the output may need to be configured

• with HTTP Basic credentials (user/password),

imweijh

Logstash internals (Queues and Threading)

The logstash agent is 3 parts: inputs -> filters -> outputs.

Each '->' is an internal messaging system. It is implemented with a 'SizedQueue' in Ruby. SizedQueue allows a bounded maximum of items in the queue such that any writes to the queue will block if the queue is full at maximum capacity.

Logstash sets the queue size to 20. This means only 20 events can be pending into the next phase - this helps reduce any data loss and in general avoids logstash trying to act as a data storage system. These internal queues are not for storing messages long-term.

In reverse, here's what happens with a queue fills.

imweijh

# How to install Check MK Agent on ubuntu 16.04

# Install check_mk_agent:
#   - sudo apt-get install check-mk-agent (will install older version)
#   - On your Check_MK dashboard, go to "Monitoring Agents", click the link for "Check_MK Agent for Linux", save the raw text
#     on your server:

sudo vi /usr/bin/check_mk_agent
# paste Check_MK dashboard > Monitoring Agents > Check_MK Agent for Linux

imweijh

Electron is tricky to get set up on Windows Subsystem for Linux, but it can work!

Four things needed overall:

1. you need WSL2, not WSL1
2. you need node, of course, and that part isn't so bad
3. you need to apt install several dependencies
4. you need an X Server so it can display the electron GUI over in Windows-land

Setup instructions, in order:

imweijh

Building C apps on Windows in 2021

This document was written on April 3, 2021. The procedure may change over time. This is a companion gist to the youtube video here, where I go through every step of both options

imweijh

::
:: This Script install the check-mk-agent and adds a Firewall Rule to the local Windows Firewall to allow Access to it!
:: https://gist.github.com/Ham5ter/2a8526e843c72ff9343bc3a38cdac97e
::

:: Allow ICMP Echo (Ping)
netsh advfirewall firewall delete rule name="ICMP Allow incoming V4 echo request"
netsh advfirewall firewall add rule name="ICMP Allow incoming V4 echo request" protocol=icmpv4:8,any dir=in action=allow

:: Allow TCP Port 6556 (check-mk-agent)

imweijh

Measuring logstash filters performance

• given a data file sample named data.log
• modify your config to use the stdin input so that you can pipe you sample log file to logstash

input {
  stdin {
    codec => line
  }
}

imweijh

input {
	beats {
		port => 5301
	}
}

filter {
	if [fields][type] == "monthly-indexed.r2web.bsd.alljob" {
		mutate {
            gsub => ["message","/"," "]