This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
require 'csv' | |
# This will create join of 2 CSVs based on key-column without deduplication | |
# The output CSV by order of csv1 | |
# more memory-efficient is to put csv1 longest table and csv2 shortest with less key column duplication | |
def join_csv(csv1_path, key1, csv2_path, key2, csv3_path, key_convertor: nil, ignore_cols: [], filter_csv1: nil, filter_csv2: nil) | |
start_ts = Time.now.to_i | |
puts "Reading #{csv2_path}, size: #{File.size(csv2_path)}..." | |
csv2 = CSV.read(csv2_path, headers: true) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import boto3 | |
import gzip | |
import json | |
import time | |
import re | |
def export(workgroup, region, hours=1): | |
athena = boto3.client('athena', region_name=region) | |
current_hour_hist = [] | |
next_token = None |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash -e | |
usage(){ | |
cat <<'HELP' | |
Usage: aws-assume-role <base-profile> <mfa-ARN> <role-ARN> <session-name> [<command>...] | |
This script is designed to be used from alias like this (of course you can also call it from other scripts): | |
alias assume-admin-prod='/path/to/aws-assume-role bob arn:aws:iam::1234567:mfa/bob arn:aws:iam::321321321:role/admin bob' | |
Arguments are positional and not key-word options on purpose. You just copy-paste it into .bash_profile to create alias and replace the parameters with right values. | |
<base-profile> is needed in case your default profile already includes role switch, in this case 'sts assume-role' won't work. |
OlderNewer