Skip to content

Instantly share code, notes, and snippets.

View cmerrick's full-sized avatar

Christopher Merrick cmerrick

15 followers · 0 following
View GitHub Profile
@cmerrick
cmerrick / rds_snapshot_copier.bash
Last active June 20, 2017 15:19
rds_snapshot_copier.bash
#!/bin/bash
# Copies the most recent snapshot associated with one or more
# RDS instances from us-east-1 to us-west-1
#
# Tested with
# $ aws --version
# aws-cli/1.11.108 Python/3.5.2 Linux/4.4.0-81-generic botocore/1.5.71
#
# Does not work with aws-cli/1.10.x or earlier.
@cmerrick
cmerrick / example.SQUARE
Created September 15, 2016 13:45
An example of the SQUARE syntax
EMP ('TOY')
NAME DEPT
@cmerrick
cmerrick / role-playing-with-aws.policy-example.json
Created August 15, 2016 14:18
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sts:AssumeRole"
],
"Resource": [
"arn:aws:iam::000123456789:role/YourRoleName"
@cmerrick
cmerrick / gist:61f67a8a11608a845493
Created March 1, 2016 17:35
redshift-node-count
aws redshift describe-clusters | jq '.Clusters[] | .NumberOfNodes' | awk '{s+=$1} END {print s}'
@cmerrick
cmerrick / bkadb-compress-data-gen.sh
Last active February 18, 2021 07:11
Better Know a Database - Redshift Load Compression
#!/bin/bash
# OS: Ubuntu 14.01
# Generate data - produces about 100GB on disk, takes a while
DBGEN_DIR=tpch-dbgen
SCALES=(1 10 100)
ORIGIN=`pwd`
S3_DIR="s3://your-bucket-here"
TABLES=(customer lineitem nation orders part partsupp region supplier)
@cmerrick
cmerrick / bkadb-data-gen.sh
Last active December 19, 2022 03:31
Better Know a Database - Redshift Load Data Formats
#!/bin/sh
# OS: Ubuntu 14.01
# Generate data - produces about 200GB on disk, takes a while
DATADIR=tpch-dbgen
SCALE=1000
git clone https://github.com/electrum/tpch-dbgen.git
cd $DATADIR && make && ./dbgen -f -v -C 16 -S 1 -s $SCALE && cd -
@cmerrick
cmerrick / lineitem-tojson.awk
Created October 21, 2015 01:37
TPC-H json and avro wrangling
BEGIN {FS = "|"}
{print "{" \
"\"l_orderkey\": " $1 ", " \
"\"l_partkey\": " $2 ", " \
"\"l_suppkey\": " $3 ", " \
"\"l_linenumber\": " $4 ", " \
"\"l_quantity\": " $5 ", " \
"\"l_extendedprice\": " $6 ", " \
"\"l_discount\": " $7 ", " \
@cmerrick
cmerrick / truncate-db
Created October 20, 2015 02:50
Truncate all tables in a MySQL DB
#!/bin/bash
DB=$1
mysql -Ne "SELECT TABLE_NAME FROM TABLES WHERE TABLE_SCHEMA='$1'" INFORMATION_SCHEMA | while read TABLE; do
mysql -Ne "TRUNCATE \`$TABLE\`" $DB
done
@cmerrick
cmerrick / aws-assume-role
Created May 17, 2015 19:06
Helper functions for installing AWS Role temporary keys into your environment
aws-reset ()
{
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY \
AWS_SECURITY_TOKEN AWS_SESSION_TOKEN \
AWS_ROLE_NAME AWS_ROLE_EXPIRATION
source "/etc/profile.d/aws-env.sh"
}
assume-role ()
@cmerrick
cmerrick / get-role-token
Created May 17, 2015 18:58
Prints the exports required to install an AWS Role's temporary keys to stdout
#!/usr/bin/env python
import socket
import json
import sys
import os
import re
from subprocess import Popen, PIPE
from pprint import pprint