Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@GaryRogers
GaryRogers / elasticsearch
Created May 16, 2014 17:14
Logstash patterns for elasticsearch.log
ELASTICSEARCHTIME \[%{TIMESTAMP_ISO8601:timestamp}\]
ELASTICSEARCHLEVEL \[%{LOGLEVEL:level}\s+\]
ELASTICSEARCHSERVICE \[%{DATA:service}\s+\]
ELASTICSEARCHVERSION \[%{DATA:version}\]
ELASTICSEARCHLOG %{ELASTICSEARCHTIME}%{ELASTICSEARCHLEVEL}%{ELASTICSEARCHSERVICE} %{ELASTICSEARCHVERSION} %{GREEDYDATA:mymessage}
@GaryRogers
GaryRogers / monolog
Created May 16, 2014 21:57
Logstash pattersn for monolog
MONOLOGTIME \[%{TIMESTAMP_ISO8601:timestamp}\]
MONOLOGLEVEL \[%{LOGLEVEL:level}\]
MONOLOGCONTEXT \[context:%{DATA:context}\]
MONOLOGEXTRA \[extra:%{DATA:extra}\]
MONOLOG %{MONOLOGTIME} %{MONOLOGLEVEL} %{MONOLOGCONTEXT}
@GaryRogers
GaryRogers / config.xml
Created May 19, 2014 19:37
ant build script for a PHAR Archive
<project name="myproject" default="build" basedir=".">
<property name="build" value="${basedir}/build"/>
<property name="build.src" value="${build}/src"/>
<target name="clean">
<delete dir="${build}" />
</target>
<target name="init">
<mkdir dir="${build}" />
@GaryRogers
GaryRogers / monolog_table.sql
Created May 28, 2014 21:06
Monolog Oracle Table
CREATE TABLE
MONOLOG
(
ID NUMBER(11) NOT NULL,
LOG_TIME TIMESTAMP(6),
LOG_LEVEL VARCHAR2(20),
LOG_HOSTNAME VARCHAR2(512),
LOG_USER VARCHAR2(30),
LOG_IP VARCHAR2(45),
LOG_MESSAGE CLOB,
@GaryRogers
GaryRogers / OracleHander.php
Created May 28, 2014 21:07
Monolog Oracle Handler
<?php
namespace Monolog\Handler;
use Monolog\Logger;
use Monolog\Formatter\LineFormatter;
// https://github.com/Seldaek/monolog/blob/master/doc/extending.md
class OracleHandler extends AbstractProcessingHandler {
@GaryRogers
GaryRogers / bootstrap.sh
Created August 16, 2014 00:27
Vagrant/CentOS Node Install with Oracle Libraries
#!/usr/bin/env bash
if [ $(yum repolist | grep -c ^epel) -eq 0 ]; then
printf "[bootstrap] Adding EPEL Repo\n";
yum --quiet -y install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
fi
# ===================================================================
# Install python if it doesn't exist.
# ===================================================================
@GaryRogers
GaryRogers / gist:0c5f4fa9baa0f3a45a76
Created October 2, 2014 21:01
StatsD/CentOS Dockerfile
# Docker file to create a CentOS StatsD host.
# This uses Elasticsearch as a backend rather than Graphite/Carbon.
# Depends on having an Elasticsearch container.
FROM centos:centos6
MAINTAINER Gary Rogers <gary-rogers@uiowa.edu>
# Install things as root
USER root
RUN \
# ==[ printSlack ]=============================================================
# Function to send output from the commandline to Slack.
# (wants SLACK_TOKEN to be defined in .bashrc or other ENV method, or you can set it here.)
#
# @parameter string $LEVEL INFO/ERROR/WARNING message. Changes emoji
# @parameter string $MESSAGE Message to send to slack.
printSlack()
{
SLACK_HOSTNAME=${SLACK_HOSTNAME-'mycompany.slack.com'};
SLACK_TOKEN=${SLACK_TOKEN-'oops'};

Remove Syslog line headers from multi-line logs in logstash

Overview

Rather than run a log shipper on hosts, we use Syslog when shipping logs out of monolog. This works great for single-line logs. It breaks when a log message gets split up by syslog. When syslog does this, it duplicates the line header, like so:

2015-06-09T05:39:31.457042-05:00 host.example.edu : This is a really really really
2015-06-09T05:39:31.475414-05:00 host.example.edu : really long message