Skip to content

Instantly share code, notes, and snippets.

@GaryRogers
GaryRogers / VagrantCentOS.md
Last active Feb 8, 2020
Provisioning Vagrant CentOS hosts
View VagrantCentOS.md
@GaryRogers
GaryRogers / RSyslogSetup.md
Last active Jun 27, 2021
RSyslog Forwarding Setup
View RSyslogSetup.md

RSyslog Forwarding Setup

Overview

I'm looking to centralize logging for our dev team into Elasticsearch via Logstash. The wrinkle is that we aren't a Java shop, so installing java on our hosts just to ship logs back to a central Logstash indexer is something we'd like to avoid. So, I'm approaching things as a chance to understand RSyslog and its capabilities as a log shipper.

Procedure

Set up TCP listening on the log index host

Uncomment the following lines in /etc/rsyslog.conf. This will enable the rsyslog daemon to listen for incoming requests on TCP port 514. We're using TCP here so that we can have some confidence that the messages from the agent hosts reach the indexer. (More on this below)

@GaryRogers
GaryRogers / gist:1ff32f3b27f4c4e52d55
Created Aug 12, 2014
Using Forms with Bootstrap-Dialog
View gist:1ff32f3b27f4c4e52d55

Using forms with Bootstrap-Dialog

var alertMessage = "";
var formTemplate = Backbone.Marionette.TemplateCache.get('#myTemplate');
var formString = formTemplate({ defaultDate: moment().add(90, 'days').format('YYYYMMDD') });

// remove the newlines. Bootstrap-Dialog doesn't like them (turns them into <br>'s)
formString = formString.replace(/[\n]/g, '');
@GaryRogers
GaryRogers / bootstrap.sh
Created Aug 16, 2014
Vagrant/CentOS Node Install with Oracle Libraries
View bootstrap.sh
#!/usr/bin/env bash
if [ $(yum repolist | grep -c ^epel) -eq 0 ]; then
printf "[bootstrap] Adding EPEL Repo\n";
yum --quiet -y install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
fi
# ===================================================================
# Install python if it doesn't exist.
# ===================================================================
@GaryRogers
GaryRogers / gist:ad17907ddad83769b1ac
Last active Aug 22, 2018
CentOS/Elasticsearch Dockerfile
View gist:ad17907ddad83769b1ac
# Docker file to create a CentOS Elasticsearch host.
FROM centos:centos6
MAINTAINER Gary Rogers <gary-rogers@uiowa.edu>
# Install things as roo
USER root
RUN \
yum update -y --quiet && \
yum install -y --quiet wget && \
View gist:0c5f4fa9baa0f3a45a76
# Docker file to create a CentOS StatsD host.
# This uses Elasticsearch as a backend rather than Graphite/Carbon.
# Depends on having an Elasticsearch container.
FROM centos:centos6
MAINTAINER Gary Rogers <gary-rogers@uiowa.edu>
# Install things as root
USER root
RUN \
View gist:7f7035edf40a679ff9c4
# ==[ printSlack ]=============================================================
# Function to send output from the commandline to Slack.
# (wants SLACK_TOKEN to be defined in .bashrc or other ENV method, or you can set it here.)
#
# @parameter string $LEVEL INFO/ERROR/WARNING message. Changes emoji
# @parameter string $MESSAGE Message to send to slack.
printSlack()
{
SLACK_HOSTNAME=${SLACK_HOSTNAME-'mycompany.slack.com'};
SLACK_TOKEN=${SLACK_TOKEN-'oops'};
View gist:1b549b783e909d546eec

Remove Syslog line headers from multi-line logs in logstash

Overview

Rather than run a log shipper on hosts, we use Syslog when shipping logs out of monolog. This works great for single-line logs. It breaks when a log message gets split up by syslog. When syslog does this, it duplicates the line header, like so:

2015-06-09T05:39:31.457042-05:00 host.example.edu : This is a really really really
2015-06-09T05:39:31.475414-05:00 host.example.edu : really long message
@GaryRogers
GaryRogers / gist:8ccc6a9e711ee229efa6
Last active Sep 24, 2016
Setting up InfluxDB on CentOS/RHEL
View gist:8ccc6a9e711ee229efa6

Setting up InfluxDB on CentOS/RHEL

The InfluxDB Docs give you a very brief overview of installing InfluxDB on a host. It boils down to 'here's the RPM, install it.' That's fine for looking at the software, but you'll probably want to adjust the configuration a bit for a production environment.

Basic Install

https://influxdb.com/docs/v0.9/introduction/installation.html

Config changes

Modify /etc/opt/influxdb/influxdb.conf

View LogstashReplay.md

Replaying logs to logstash

  • Copy comprssed log files to a work area.
  • Uncompress them, remove date part of file name.
  • Copy /etc/logstash/conf.d/*.conf to a work location.
  • Modify conf files to change output to stdout { codev => "rubydebug" }
  • You want to do this to make sure things are working before you push logs into ElasticSearch.
  • Modify conf files to change path in the input/file section