Skip to content

Instantly share code, notes, and snippets.

View douglascodes's full-sized avatar

Douglas King douglascodes

View GitHub Profile
@douglascodes
douglascodes / aws_var_encryption.py
Last active May 4, 2020 20:31
Encrypts all environment variables using the KMS key specified in env['aws_key_arn']. Much simpler to use as a utility in something like PyCharm with run configurations.
#! /usr/bin/python
"""
Script to encrypt environment variables to their encrypted version.
aws_key_arn - Resource ID # for aws encryption key.
format: 'arn:aws:kms:us-east-1:123456789:key/711d0e6d-620c-47da-a2f6-7141eb8cbde4'
"""
import os
import boto3
from base64 import b64encode
@douglascodes
douglascodes / glue_script.py
Last active January 6, 2021 18:56
AN AWS Glue script for remote debugging example
# For https://support.wharton.upenn.edu/help/glue-debugging
import uuid
from pyspark.sql.functions import *
from pyspark.sql.types import StringType, IntegerType, StructField, StructType
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions
import sys
from pyspark.context import SparkContext
import datetime
@douglascodes
douglascodes / pandas_apply_two_column_result.py
Created July 31, 2019 17:50
Example for assign Tuple result to two columns after .apply() in Pandas
import pandas as pd
import random
def avg(a, b):
return (a+b)/2
def diff(a, b):
return abs(a-b)
def get_stats(a, b):
@douglascodes
douglascodes / recamán.py
Created July 2, 2018 04:43
Recamán sequence simple.
import sys
from collections import Counter
def main():
next_location = 0
history = Counter()
for i in range(1, 100):
print(next_location)
next_location = racaman(next_location, i, history)
@douglascodes
douglascodes / gather_hdd_stats.sh
Created June 29, 2018 15:49
Creates a log file for iostat data
#! /usr/bin/env bash
# gather_hdd_stats.sh $(date +%F).log
# Every 5 seconds on the mod 5 pipe formatted iostat data to <file>
if [ $# -lt 1 ] ; then
echo "usage: $0 <file>"
exit -1
fi
outputfile="$1"
@douglascodes
douglascodes / clean_iostat.awk
Created June 29, 2018 15:47
Awk script for taking grouped drives and outputting single line with current time. Pipe to text log
#!/usr/bin/env awk
# iostat -t -d -H -g hdds sda sdb
{OFS="\t"}
NR==3 { runtime=$1;}
NR==5 { print runtime, $1, $2, $3, $4, $5, $6}
@douglascodes
douglascodes / aws_redhat_basic.yml
Last active September 17, 2017 21:46
An ansible file for setting up my default config
---
- hosts: redhat_aws
vars:
remote_user: ec2-user
target_user: myUserName
tasks:
- name: 1.Check if EPEL repo is already configured.
stat: path=/etc/yum.repos.d/epel.repo
register: epel_repofile_result
@douglascodes
douglascodes / inotify_sync.sh
Last active July 5, 2017 11:21
Script to update web directory on change. A -> B Watch using inotifywait.
#!/bin/sh
# Syncs files from directory A to B. With the 'chown' this will need to be run as root.
# Drop that to run as unprivileged user.
if [ $# -lt 2 ] ; then
echo "usage: $0 <dir to watch> <sync to>"
fi
source="$1"
destination="$2"
@douglascodes
douglascodes / getDaysOldFiles.ps1
Created February 23, 2017 02:40
Gets files where -14 days old. Can be turned into a function easily.
Get-ChildItem -Recurse | Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-14) }
@douglascodes
douglascodes / removeOldBackups.ps1
Last active February 23, 2017 02:36
Function for testing files in similar paths but different roots exist in both places
# Script for testing file existence in two places
# PS C:\> . .\removeOldBackups.ps1
# PS C:\> existsInBoth -DestRoot 'C:\Program Files\' -SourceRoot 'C:\Program Files (x86)\' -CommonPath "Common Files"
# True
# PS C:\> existsInBoth -DestRoot 'C:\Program Files\' -SourceRoot 'C:\Program Files (x86)\' -CommonPath "UnmatchedFile"
# False
function existsInBoth{
param ( [string] $SourceRoot,
[string] $DestRoot,