GaryRogers

Provisioning Vagrant CentOS Hosts

Overview

Process

Install Vagrant and VirtualBox Vagrant Download https://www.vagrantup.com/downloads.html

Virtual Box Download https://www.virtualbox.org/wiki/Downloads

GaryRogers

RSyslog Forwarding Setup

Overview

I'm looking to centralize logging for our dev team into Elasticsearch via Logstash. The wrinkle is that we aren't a Java shop, so installing java on our hosts just to ship logs back to a central Logstash indexer is something we'd like to avoid. So, I'm approaching things as a chance to understand RSyslog and its capabilities as a log shipper.

Procedure

Set up TCP listening on the log index host

Uncomment the following lines in /etc/rsyslog.conf. This will enable the rsyslog daemon to listen for incoming requests on TCP port 514. We're using TCP here so that we can have some confidence that the messages from the agent hosts reach the indexer. (More on this below)

GaryRogers

Using forms with Bootstrap-Dialog

var alertMessage = "";
var formTemplate = Backbone.Marionette.TemplateCache.get('#myTemplate');
var formString = formTemplate({ defaultDate: moment().add(90, 'days').format('YYYYMMDD') });

// remove the newlines. Bootstrap-Dialog doesn't like them (turns them into <br>'s)
formString = formString.replace(/[\n]/g, '');

GaryRogers

#!/usr/bin/env bash

if [ $(yum repolist | grep -c ^epel) -eq 0 ]; then
  printf "[bootstrap] Adding EPEL Repo\n";
  yum --quiet -y install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
fi

# ===================================================================
# Install python if it doesn't exist.
# ===================================================================

GaryRogers

# Docker file to create a CentOS Elasticsearch host.
FROM centos:centos6
MAINTAINER Gary Rogers <gary-rogers@uiowa.edu>

# Install things as roo
USER root

RUN \
  yum update -y --quiet && \
  yum install -y --quiet wget && \

GaryRogers

# Docker file to create a CentOS StatsD host.
# This uses Elasticsearch as a backend rather than Graphite/Carbon.
# Depends on having an Elasticsearch container.
FROM centos:centos6
MAINTAINER Gary Rogers <gary-rogers@uiowa.edu>

# Install things as root
USER root

RUN \

GaryRogers

# ==[ printSlack ]=============================================================
# Function to send output from the commandline to Slack.
# (wants SLACK_TOKEN to be defined in .bashrc or other ENV method, or you can set it here.)
#
# @parameter string $LEVEL   INFO/ERROR/WARNING message. Changes emoji
# @parameter string $MESSAGE Message to send to slack.
printSlack()
{
  SLACK_HOSTNAME=${SLACK_HOSTNAME-'mycompany.slack.com'};
  SLACK_TOKEN=${SLACK_TOKEN-'oops'};

GaryRogers

Remove Syslog line headers from multi-line logs in logstash

Overview

Rather than run a log shipper on hosts, we use Syslog when shipping logs out of monolog. This works great for single-line logs. It breaks when a log message gets split up by syslog. When syslog does this, it duplicates the line header, like so:

2015-06-09T05:39:31.457042-05:00 host.example.edu : This is a really really really
2015-06-09T05:39:31.475414-05:00 host.example.edu : really long message

GaryRogers

Setting up InfluxDB on CentOS/RHEL

The InfluxDB Docs give you a very brief overview of installing InfluxDB on a host. It boils down to 'here's the RPM, install it.' That's fine for looking at the software, but you'll probably want to adjust the configuration a bit for a production environment.

Basic Install

https://influxdb.com/docs/v0.9/introduction/installation.html

Config changes

Modify /etc/opt/influxdb/influxdb.conf

GaryRogers

Replaying logs to logstash

• Copy comprssed log files to a work area.
• Uncompress them, remove date part of file name.
• Copy /etc/logstash/conf.d/*.conf to a work location.
• Modify conf files to change output to stdout { codev => "rubydebug" }
• You want to do this to make sure things are working before you push logs into ElasticSearch.
• Modify conf files to change path in the input/file section