Skip to content

Instantly share code, notes, and snippets.

Avatar

Russell Jurney rjurney

View GitHub Profile
@rjurney
rjurney / terminate_all_ec2.sh
Last active Nov 19, 2020
Bash script to disable termination protection and then terminate all instances in all regions :)
View terminate_all_ec2.sh
for region in `aws ec2 describe-regions | jq -r .Regions[].RegionName`
do
echo "Terminating region $region..."
aws ec2 describe-instances --region $region | \
jq -r .Reservations[].Instances[].InstanceId | \
xargs -L 1 -I {} aws ec2 modify-instance-attribute \
--region $region \
--no-disable-api-termination \
--instance-id {}
aws ec2 describe-instances --region $region | \
@rjurney
rjurney / metrics.py
Created Nov 15, 2020
A Tensorflow/Keras implementation of Adjusted R Squared
View metrics.py
import typing
import tensorflow as tf
import tensorflow_addons as tfa
from tensorflow_addons.utils.types import AcceptableDTypes
from typeguard import typechecked
class AdjustedRSquared(tfa.metrics.RSquare):
@rjurney
rjurney / start_dask.sh
Last active Nov 1, 2020
A script to start single node Dask with as many workers as your machine has processor cores
View start_dask.sh
#!/bin/bash
# Launch the scheduler
nohup dask-scheduler 2>&1 >> /tmp/dask.log &
# Launch as many workers as we have cores
CORES=$(cat /proc/cpuinfo | grep processor | wc -l)
nohup dask-worker --nprocs=${CORES} 127.0.0.1:8786 2>&1 >> /tmp/dask.log &
@rjurney
rjurney / awswrangler.py
Created Oct 29, 2020
Something is wrong, the local load takes longer than the S3 load from a bad connection
View awswrangler.py
# How can I be faster?
# Setup a session with credentials
boto3_session = BarUtils.boto_session(
aws_access_key_id=s3_key,
aws_secret_access_key=s3_secret,
)
df = wr.s3.read_parquet(
path=path,
@rjurney
rjurney / pyarrow.parquet.2.0.py
Last active Oct 24, 2020
PyArrow now takes forever to load partitioned Parquet data. Why?
View pyarrow.parquet.2.0.py
# Prepare the partition filter
filters = [
[('Ticker', 'in', tickers)]
]
dataset = pq.ParquetDataset(
path_or_paths=LOCAL_PATH,
filesystem=filesystem,
filters=filters,
metadata_nthreads=4,
@rjurney
rjurney / r2_scores.py
Created Oct 20, 2020
Implementations of R^2 Score
View r2_scores.py
def r2_score(y_true, y_pred):
"""Implements the Coeffecient of Determination, R^2 or R-squared"""
SS_res = kb.sum(kb.square(y_true - y_pred))
SS_tot = kb.sum(kb.square(y_true - kb.mean(y_true)))
return (1 - SS_res / (SS_tot + kb.epsilon()))
def inverse_r2_score(y_true, y_pred):
"""Implements the inverse Coeffecient of Determination, R^2 or R-squared"""
SS_res = kb.sum(kb.square(y_true - y_pred))
@rjurney
rjurney / build.sh
Created Sep 7, 2020
How to securely or insecurely get your code from Amazon S3 in Colab
View build.sh
#!/bin/bash
# Copy files to S3
aws s3 cp myfile.py s3://mybucket/
aws s3 cp config.py s3://mybucket/
@rjurney
rjurney / install_ta_lib_cell.py
Created Sep 7, 2020
A Colab cell to build and install TA Lib and its Python interface TA-Lib
View install_ta_lib_cell.py
# download TA-Lib
!wget -q http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz 2>&1 > /dev/null
!tar xvzf ta-lib-0.4.0-src.tar.gz 2>&1 > /dev/null
import os
os.chdir('ta-lib') # Can't use !cd in co-lab
!./configure --prefix=/usr 2>&1 > /dev/null
!make 2>&1 > /dev/null
@rjurney
rjurney / centroid_lf.py
Last active Aug 18, 2020
Snorkel LabelingFunction strategy where I assign labels by taking the distance between a document vector and a centroid - the mean class vector
View centroid_lf.py
# Snorkel LabelingFunction strategy where I assign labes by taking the distance between a document vector and
# a centroid - the mean class vector
import spacy
from scipy import spatial
# Append the readme to the description separated by a space
dev_df['description_readme'] = dev_df[['description', 'readme']].agg(' '.join, axis=1).str.lower()
View keybase.md

Keybase proof

I hereby claim:

  • I am rjurney on github.
  • I am rjurney (https://keybase.io/rjurney) on keybase.
  • I have a public key ASCimxw2-8kZzIKOFyFXSNVK-d0lnokOpHRQaJLdMciuTgo

To claim this, I am signing this object:

You can’t perform that action at this time.