Skip to content

Instantly share code, notes, and snippets.

View johndstein's full-sized avatar

John Stein johndstein

View GitHub Profile
#!/usr/bin/env bash
# minimum size to get benefit from AWS S3 intelligent tiering.
size=128000
function help() {
echo ""
echo "Usage: s3-stats.sh (options)"
echo ""
echo " Get counts and percent of files over a given size."
@johndstein
johndstein / list-empty-buckets.sh
Created May 20, 2022 11:02
find empty s3 buckets
#!/usr/bin/env bash
# list empty buckets
buckets=$(aws s3api list-buckets| jq --raw-output '.Buckets[].Name')
for bucket in $buckets;do
files=$(aws s3 ls $bucket)
if [[ -z "$myVar" ]]; then
echo "$bucket"
fi
done
@johndstein
johndstein / mssql-stream-insert.js
Last active June 15, 2021 20:16
Node.js writable stream that connects to db, drops and creates table, and then bulk loads rows into table
#!/usr/bin/env node
'use strict';
// Stream unlimited rows into a Sql Server table.
// WARNING!!! WE DROP and RE-CREATE the table. Then stream the data into it.
// Source stream must be an object stream. Object property names must match
// table column names. Since SQL Server isn't case sensitive, don't think case
@johndstein
johndstein / StackTrace_Test.java
Last active January 2, 2021 18:06
Salesforce APEX Stacktrace test if code is below given class.method
@isTest
private class StackTrace_Test {
public class MyException extends Exception {}
static testMethod void test() {
Test.StartTest();
someMethod();
Test.StopTest();
}
@johndstein
johndstein / s3select.sh
Created October 28, 2020 12:19
parallel s3 select
#!/usr/bin/env bash
help () {
echo "
Usage:
test/s3select.sh -v \\
--bucket some-bucket \\
--prefix foo/2020/10/20/20/57 \\
--expression \"select * from s3object o where o.rawmsghostname = 'ynthtest-Oct20T20-filler-5b45a8d7'\"
mongo --ssl \
--host $(cat ~/infrastructure/keys/varsdb-creds | cut -d ':' -f2 | cut -d '@' -f2):27017 \
--sslCAFile /opt/helix-tools/rds-combined-ca-bundle.pem \
--username $(cat ~/infrastructure/keys/varsdb-creds | cut -d ':' -f1) \
--password $(cat ~/infrastructure/keys/varsdb-creds | cut -d ':' -f2 | cut -d '@' -f1)
@johndstein
johndstein / SamlJitHandler.java
Created January 31, 2018 20:17
Salesforce SAML JIT Handler
// This class provides logic for inbound just-in-time provisioning of single
// sign-on users in your Salesforce organization.
// Here's all the info we're going to have.
//
// Employee Id
// First Name
// Last Name
// Email
// Active Flag
@johndstein
johndstein / syslog-ng.conf
Last active February 19, 2020 08:27
default syslog-ng.conf
ubuntu@tmpc1:/etc/syslog-ng$ cat syslog-ng.conf
@version: 3.5
@include "scl.conf"
@include "`scl-root`/system/tty10.conf"
# Syslog-ng configuration file, compatible with default Debian syslogd
# installation.
# First, set some global options.
options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no);
@johndstein
johndstein / ssjl.js
Created October 10, 2019 18:20
super simple json log for fe
#!/usr/bin/env node
'use strict';
class Log {
constructor(options) {
options = options || {};
this.level = 'off';
this.levels = {
debug: 0,
info: 1,
warn: 2,
@johndstein
johndstein / conditional-pipe.js
Created October 10, 2019 15:20
Solid Gold!!!
#!/usr/bin/env node
'use strict';
const AWS = require('aws-sdk');
if (process.env.AWS_PROFILE) { // eslint-disable-line
AWS.config.credentials = new AWS.SharedIniFileCredentials({
profile: process.env.AWS_PROFILE // eslint-disable-line
});
AWS.config.logger = console;
}
const s3opts = {};