Skip to content

Instantly share code, notes, and snippets.

View beugley's full-sized avatar

Brian Eugley beugley

View GitHub Profile
@beugley
beugley / run_parallel.sh
Created February 1, 2022 16:07
Bash functions to run other commands in parallel with output serialized
#!/bin/bash
# Contains common functions that can be sourced by other bash scripts
function RunParallel
# Usage: Set the PROCESSES and CONCURRENCY variables and invoke RunParallel without parameters
# PROCESSES: An array of commands to execute
# CONCURRENCY: The number of commands to execute concurrently
# - Takes a list of processes in the PROCESSES variable, runs them in parallel, waits for
# each to complete, and then serializes their output (i.e. not interleaved).
# - If any process fails, then RunParallel will return a non-0 value.
@beugley
beugley / is_datetime_std_or_dst.py
Created January 16, 2020 13:08
Python class to determine if a datetime string occurred during STD or DST
#!/usr/bin/env python3
import datetime as dt
import subprocess
class STD_DST(dt.tzinfo):
""" Determine if a datetime occured during STD or DST time """
def dst(self, a):
# DST starts second Sunday in March at 2AM
d = dt.datetime(a.year, 3, 1, 2)
@beugley
beugley / s3_put.py
Created January 16, 2020 13:00
Python boto3 script to encrypt an object on the client side using KMS envelope encryption and upload to AWS S3
#!/usr/bin/env python
###############################################################################
# This script uploads a file to an AWS S3 bucket. It will optionally use KMS
# envelope encryption to encrypt a file on the client side before uploading
# to AWS S3.
#
# Envelope encryption fetches a data key from KMS and uses it to encrypt the
# file. The encrypted file is uploaded to an S3 bucket along with an encrypted
# version of the data key (it's encrypted with a KMS master key). You must
# have access to the KMS master key to decrypt the data key and file.
@beugley
beugley / convert_sas7bdat_subset_to_text.sas
Created November 18, 2019 16:59
SAS script to convert a data set to text format
*options mprint mlogic symbolgen;
options errorabend fullstimer compress=binary;
options nofmterr validvarname=any;
/*
** SAS script to convert a data set to text-delimited format. The output file
** will be written to either the same directory where the SAS data set resides
** or to a user-specified file.
**
** sysparm must contain 5 :-separated values:
@beugley
beugley / postgres_execute_with_reconnect.py
Created August 26, 2019 14:24
Python script to execute Postgresql SQL command with automatic reconnect to database
def execute_sql(self, sqlcmd, arg_list=()):
''' Returns the cursor if the SQL statement is successful, else
returns None if the SQL statement fails.
Automatically reconnects if the database connection is broken and
retries the SQL up to 5 times.
'''
logging.debug("Executing SQL: '%s', '%s'" % (sqlcmd, arg_list))
for i in range(5):
try:
self.cursor.execute(sqlcmd, arg_list)
@beugley
beugley / connnect_postgres.py
Last active August 26, 2019 14:23
Python script to connect to postgresql database
def connect(self):
''' Open a connection to the database '''
logging.info("Connecting to database '%s'" % config['DB_DBNAME'])
dirname = os.path.dirname(os.path.realpath(sys.argv[0]))
DB_PASSWORD_FILE = (dirname + "/.dbpass." + config['DB_HOST'] + "." +
config['DB_USER'])
with open(DB_PASSWORD_FILE) as pwd:
DB_PASSWORD = pwd.readline().strip()
self.conn = psycopg2.connect("host='%s' port='%s' dbname='%s' "
"user='%s' password='%s'" % (
@beugley
beugley / login.exp
Created August 1, 2018 18:04
Expect script to automate login using password file
#!/usr/bin/expect
set password [exec cat .pwd]
set timeout -1
# Connect to host, enter password
spawn ssh -o ServerAliveInterval=5 -o ServerAliveCountMax=1 server-name.com
expect {
"Password: " {send "$password\r"; exp_continue}
"password: " {send "$password\r"; interact}
}
@beugley
beugley / mail_it.sh
Last active May 6, 2024 14:15
bash script to send email as plain text or HTML, with attachments
#!/bin/sh
###############################################################################
## mail_it.sh
## A bash script that sends email as either plain text or HTML. It allows
## multiple recipients, CC addresses, reply-to addresses, and attachments.
##
## Usage: mail_it.sh -s subject -m message -f from_address
## -t to_address[,...] [-c cc_address[,...]] [-r reply_to_address[,...]]
## [-a attachment[,...]] [-h]
## subject: email subject
@beugley
beugley / perm_to_text.py
Created February 6, 2017 21:16
Python script to convert a Linux octal permission number to a text string
def perm_to_text(perm):
perms = {
"0": "---",
"1": "--x",
"2": "-w-",
"3": "-wx",
"4": "r--",
"5": "r-x",
"6": "rw-",
"7": "rwx"
@beugley
beugley / s3_get.py
Last active December 4, 2021 14:58
Python boto3 script to download an object from AWS S3 and decrypt on the client side using KMS envelope encryption
#!/usr/bin/env python
###############################################################################
# This script downloads an object from AWS S3. If the object was encrypted,
# it will be decrypted on the client side using KMS envelope encryption.
#
# Envelope encryption fetches a data key from KMS and uses it to encrypt the
# file. The encrypted file is uploaded to an S3 bucket along with an encrypted
# version of the data key (it's encrypted with a KMS master key). You must
# have access to the KMS master key to decrypt the data key and file. To
# decrypt, the file is downloaded to the client, the encrypted data key is