Skip to content

Instantly share code, notes, and snippets.

View aroder's full-sized avatar

Adam Roderick aroder

14 followers · 2 following
View GitHub Profile
@aroder
aroder / s3cmdclearfiles
Created September 12, 2022 22:26 — forked from JProffitt71/s3cmdclearfiles
Uses s3cmd to manually remove files older than specified date (##d|m|y) in a specified bucket
#!/bin/bash
# Usage: ./s3cmdclearfiles "bucketname" "30d"
s3cmd ls s3://$1 | grep " DIR " -v | while read -r line;
do
createDate=`echo $line|awk {'print $1" "$2'}`
createDate=`date -j -f "%Y-%m-%d %H:%M" "$createDate" +%s`
olderThan=`date -j -v-$2 +%s`
if [[ $createDate -lt $olderThan ]]
@aroder
aroder / platform-architecture.md
Created February 8, 2021 23:26

Datateer platform architecture

graph TD
    user
    subgraph aws[AWS]
      subgraph s3[S3 & CloudFront]
        web[web app]
      end
 subgraph ecs[ECS]
@aroder
aroder / ssh-deploy-key-wrapper.sh
Last active January 1, 2021 00:37 — forked from mpdude/ssh-deploy-key-wrapper.sh
Wrapper around `ssh` to pick the right one from several GitHub deploy keys
#!/bin/bash
regex_repo_name="\/([^\/]*\.git)"
if [[ ${!#} =~ $regex_repo_name ]]
then
repo_name="${BASH_REMATCH[1]}"
else
echo "Could not find a repo name in ${!#}"
fi
@aroder
aroder / trigger_processor.py
Last active January 8, 2021 00:16
import base64
import json
import multiprocessing
import sys
import boto3
import botocore
INPUT_FILE = 'all_raw_files.txt'
S3_RAW_BUCKET_NAME = '____'
@aroder
aroder / developer_setup.py
Created December 18, 2020 23:32
"""Run this script to create or refresh your developer environment
Installs a virtual environment with the main requirements at venv, one at venv/dbt, and one at venv/meltano. The root (venv) can be changed by passing an argument e.g. "python requirements/developer_setup.py my_venv_root"
To run: python developer_setup.py
"""
import glob
import os
import pip
import re
import shutil
import subprocess
@aroder
aroder / s3_multipart_upload.py
Created July 8, 2020 01:54 — forked from teasherm/s3_multipart_upload.py
boto3 S3 Multipart Upload
import argparse
import os
import boto3
class S3MultipartUpload(object):
# AWS throws EntityTooSmall error for parts smaller than 5 MB
PART_MINIMUM = int(5e6)
@aroder
aroder / remove_all_docker_containers.bat
Created September 27, 2019 05:51
Remove all docker containers
FOR /f "tokens=*" %%i IN ('docker ps -aq') DO docker rm %%i
@aroder
aroder / script.sql
Created February 1, 2018 21:58
Diff 2 tables from different databases
WITH A AS (
SELECT *
FROM OPENROWSET('SQLNCLI', 'Server=<SERVERA>;Database=CADIS;Trusted_Connection=yes;', 'select top 10 * from <TABLE>')
),
B AS (SELECT * FROM <TABLE>),
A_ONLY AS (
SELECT TOP 10 * FROM A
EXCEPT
SELECT TOP 10 * FROM B
@aroder
aroder / Dockerfile
Last active December 17, 2017 14:59
botpress Dockerfile
# comes with node and npm already installed
FROM node:carbon-alpine
RUN mkdir -p /usr/src/app
RUN chown node:node /usr/src/app
WORKDIR /usr/src/app
USER node
# ensures both package.json and package-lock.json are copied
COPY package*.json ./
RUN npm install --quiet --production --no-progress && \
@aroder
aroder / sqlYamlParser.ps1
Created October 30, 2017 14:57
SQL YAML Documentation Parser
param (
[string]$output = "wiki", # can be html or wiki
[string]$DatabaseServer = "nt7565",
[string]$Database = "MARKITEDM_DEV_DX"
#[Parameter(Mandatory=$true)][string]$username,
#[string]$password = $( Read-Host "Input password, please" )
)
Clear-Host
(new-object Net.WebClient).DownloadString("http://psget.net/GetPsGet.ps1") | iex