Skip to content

Instantly share code, notes, and snippets.

View louiszuckerman's full-sized avatar

Louis Zuckerman louiszuckerman

16 followers · 1 following
View GitHub Profile
@louiszuckerman
louiszuckerman / gist:1346387
Created November 7, 2011 22:18
Logstash parser for ModSecurity/CRS entries in the Apache ErrorLog
Logstash Configuration...
input {
file {
format => "plain"
path => "/var/log/apache2/*error.log"
type => "apacheerror"
}
}
filter {
@louiszuckerman
louiszuckerman / gist:1499710
Created December 20, 2011 01:01
logstash parser for glusterfs logs
input {
file {
path => "/var/log/glusterfs/*.log"
sincedb_path => "/var/tmp/.glusterfs.sincedb"
format => "plain"
type => "glusterfs"
}
file {
path => "/var/log/glusterfs/bricks/*.log"
sincedb_path => "/var/tmp/.glusterfs.bricks.sincedb"
@louiszuckerman
louiszuckerman / gist:2229570
Created March 28, 2012 19:10
Logstash CloudWatch Output Documentation
Logstash CloudWatch Output Plugin
Code: https://github.com/semiosis/logstash/blob/master/lib/logstash/outputs/cloudwatch.rb
Jira Issue: https://logstash.jira.com/browse/LOGSTASH-461
Architecture overview...
The CloudWatch output plugin simply aggregates events & calls the CloudWatch API to push data. All event routing & processing is done using conventional Logstash configuration.
To send events to CloudWatch you'll need to add at least one field to the event which is the name of the CloudWatch metric the event belongs to. This is done in Logstash using the add_field option of inputs & filters like so:
@louiszuckerman
louiszuckerman / gist:2377958
Created April 13, 2012 16:03
Logstash load average monitor
input {
exec {
command => "echo `date +'%z %Y-%m-%d'``uptime`"
type => "loadaverages"
interval => 10
}
}
filter {
grok {
type => "loadaverages"
@louiszuckerman
louiszuckerman / gist:3209901
Created July 30, 2012 20:30
Semiosis' logstash upstart job
description "Logstash producer"
start on runlevel [2345]
stop on runlevel [016]
respawn
script
cd /opt/logstash
export HOME=/opt/logstash
@louiszuckerman
louiszuckerman / gist:3795948
Created September 27, 2012 19:28
savn -- brings version controlled order to chaotic system configurations
#!/bin/bash
# This script is meant to make it easy for a system administrator to archive and restore important system files.
# Design:
# A central repository holds archived config files for all servers, with a directory for each server. Individual
# servers check out their subdirectory to a local working copy. Files are archived by hard-linking into the local
# working copy and then committing to the central repo.
# Prerequisites:
@louiszuckerman
louiszuckerman / gist:3908490
Created October 17, 2012 21:46
Logstash grok parser for custom varnishncsa log format
I pass these options to the varnishncsa daemon (this line is from my /etc/init.d/varnishncsa)...
DAEMON_OPTS="-a -F "\''%h "%{X-Forwarded-For}i" %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" %{Varnish:time_firstbyte}x %{Varnish:handling}x'\'" -c -w ${LOGFILE} -D -P $PIDFILE"
I parse that with the following grok pattern...
VARNISHNCSALOG %{IPORHOST:remoteip} %{QUOTEDSTRING:xforwardedfor} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion}" (?:%{NUMBER:status}|\(null\)) (?:%{NUMBER:bytes}|-) "(?:%{NOTSPACE:referrer}|-)" %{QUOTEDSTRING:agent} %{BASE10NUM:berespms} %{WORD:cache}
@louiszuckerman
louiszuckerman / gfid-resolver.sh
Last active November 29, 2023 10:01
Glusterfs GFID Resolver Turns a GFID into a real path in the brick
#!/bin/bash
if [[ "$#" < "2" || "$#" > "3" ]]; then
cat <<END
Glusterfs GFID resolver -- turns a GFID into a real file path
Usage: $0 <brick-path> <gfid> [-q]
<brick-path> : the path to your glusterfs brick (required)
@louiszuckerman
louiszuckerman / as-terminate
Last active August 15, 2017 22:43
easy ssh access and pruning for EC2 instances in AutoScaling groups. depends on AWS CLI.
#!/bin/bash
if [ "$1" == "-h" ]; then
cat <<END
AWS AutoScaling Termination Helper:
terminate an instance in an auto scaling group
Usage: $0 {group-name}
END
@louiszuckerman
louiszuckerman / gist:4658921
Last active December 11, 2015 20:59
kibana on passenger in a subdirectory of an existing apache virtual host
# These instructions are derived from the Passenger documentation,
# http://www.modrails.com/documentation/Users%20guide%20Apache.html#deploying_rack_to_sub_uri
# And tested with kibana git master on 1/28/13
# First, create a symlink from the existing virtualhost's
# document root to your kibana installation's public/static
# directory. for example,
# ln -s /path/to/kibana/public /var/www/kibana
# kibana 0.2.0 and earlier will have a /static directory
# instead of /public