toddlers

#!/bin/bash
# herein we backup our indexes! this script should run at like 6pm or something, after logstash
# rotates to a new ES index and theres no new data coming in to the old one. we grab metadatas,
# compress the data files, create a restore script, and push it all up to S3.
TODAY=`date +"%Y.%m.%d"`
INDEXNAME="logstash-$TODAY" # this had better match the index name in ES
INDEXDIR="/usr/local/elasticsearch/data/logstash/nodes/0/indices/"
BACKUPCMD="/usr/local/backupTools/s3cmd --config=/usr/local/backupTools/s3cfg put"
BACKUPDIR="/mnt/es-backups/"
YEARMONTH=`date +"%Y-%m"`

toddlers

filter {
  # strip the syslog PRI part and create facility and severity fields.
  # the original syslog message is saved in field %{syslog_raw_message}.
  # the extracted PRI is available in the %{syslog_pri} field.
  #
  # You get %{syslog_facility_code} and %{syslog_severity_code} fields.
  # You also get %{syslog_facility} and %{syslog_severity} fields if the
  # use_labels option is set True (the default) on syslog_pri filter.
  grok {
    type => "syslog-relay"

toddlers

# Create 4 volumes and attach them to hdb
%w[sdi sdj sdk sdl].each do |dev|
  volume = AWS.volumes.new :device => "/dev/#{dev}", :size => 5, :availability_zone => hdb.availability_zone
  volume.server = hdb
  volume.save
end

toddlers

Usage:

ruby aws.rb launch [--region region --aws-key aws_key --aws-secret aws_secret --security-group security_group --key-pair key_pair --key-path key_path --ami-id ami_id--flavor-id flavor_id ]  

ruby aws.rb terminate [--region region --aws-key aws_key --aws-secret aws_secret] --instance-id instance_id  

ruby aws.rb attach [--region region --aws-key aws_key --aws-secret aws_secret] --instance-id instance_id --volume-id volume_id
No arguments provided

toddlers

Backup MySQL to Amazon S3

This is a simple way to backup your MySQL tables to Amazon S3 for a nightly backup - this is all to be done on your server :-)

Sister Document - Restore MySQL from Amazon S3 - read that next

1 - Install s3cmd

this is for Centos 5.6, see http://s3tools.org/repositories for other systems like ubuntu etc

toddlers

Restore MySQL from Amazon S3

This is a hands-on way to pull down a set of MySQL dumps from Amazon S3 and restore your database with it

Sister Document - Backup MySQL to Amazon S3 - read that first

1 - Set your MySQL password and S3 bucket, make a temp dir, get a list of snapshots

# Set our variables

export mysqlpass="ROOTPASSWORD"

toddlers

#!/usr/bin/env ruby

require 'rubygems'
require 'fog'

config = YAML.load(File.read(ARGV[0]))
volumes_to_snap = YAML.load(File.read(ARGV[1]))
time = Time.now

puts "\nCreating snaps #{time.to_s}"

toddlers

#!/bin/bash
##################################################
#
# s3-s3cmd-sync-dated-dirs.sh
#
# Easily download content in dated directory
# structures from s3.
#
# 
# https://gist.github.com/imperialWicket

toddlers

#!/bin/bash

# ag is the_silver_searcher (https://github.com/ggreer/the_silver_searcher)

ag TODO | while read line; do
  file=$(echo ${line} | awk -F ':' '{print $1}');
  lineno=$(echo ${line} | awk -F ':' '{print $2}');
  git blame --line-porcelain "./${file}" -L ${lineno},${lineno} | grep author-mail | cut -d ' ' -f 2;
done | sort | uniq -c

toddlers

tmux shortcuts & cheatsheet

start new:

tmux

start new with session name:

tmux new -s myname