Skip to content

Instantly share code, notes, and snippets.

Avatar

Russell Jurney rjurney

View GitHub Profile
@rjurney
rjurney / r2_scores.py
Created Oct 20, 2020
Implementations of R^2 Score
View r2_scores.py
def r2_score(y_true, y_pred):
"""Implements the Coeffecient of Determination, R^2 or R-squared"""
SS_res = kb.sum(kb.square(y_true - y_pred))
SS_tot = kb.sum(kb.square(y_true - kb.mean(y_true)))
return (1 - SS_res / (SS_tot + kb.epsilon()))
def inverse_r2_score(y_true, y_pred):
"""Implements the inverse Coeffecient of Determination, R^2 or R-squared"""
SS_res = kb.sum(kb.square(y_true - y_pred))
@rjurney
rjurney / start_dash.sh
Created Oct 5, 2020
A script to start single node Dask with as many workers as your machine has processor cores
View start_dash.sh
#!/bin/bash
# Launch the scheduler
nohup dask-scheduler 2>&1 >> /tmp/dask.log &
# Launch as many workers as we have cores
CORES=$(cat /proc/cpuinfo | grep processor | wc -l)
nohup dask-worker --nprocs=${CORES} 127.0.0.1:8786 2>&1 >> /tmp/dask.log &
@rjurney
rjurney / build.sh
Created Sep 7, 2020
How to securely or insecurely get your code from Amazon S3 in Colab
View build.sh
#!/bin/bash
# Copy files to S3
aws s3 cp myfile.py s3://mybucket/
aws s3 cp config.py s3://mybucket/
@rjurney
rjurney / install_ta_lib_cell.py
Created Sep 7, 2020
A Colab cell to build and install TA Lib and its Python interface TA-Lib
View install_ta_lib_cell.py
# download TA-Lib
!wget -q http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz 2>&1 > /dev/null
!tar xvzf ta-lib-0.4.0-src.tar.gz 2>&1 > /dev/null
import os
os.chdir('ta-lib') # Can't use !cd in co-lab
!./configure --prefix=/usr 2>&1 > /dev/null
!make 2>&1 > /dev/null
@rjurney
rjurney / centroid_lf.py
Last active Aug 18, 2020
Snorkel LabelingFunction strategy where I assign labels by taking the distance between a document vector and a centroid - the mean class vector
View centroid_lf.py
# Snorkel LabelingFunction strategy where I assign labes by taking the distance between a document vector and
# a centroid - the mean class vector
import spacy
from scipy import spatial
# Append the readme to the description separated by a space
dev_df['description_readme'] = dev_df[['description', 'readme']].agg(' '.join, axis=1).str.lower()
View keybase.md

Keybase proof

I hereby claim:

  • I am rjurney on github.
  • I am rjurney (https://keybase.io/rjurney) on keybase.
  • I have a public key ASCimxw2-8kZzIKOFyFXSNVK-d0lnokOpHRQaJLdMciuTgo

To claim this, I am signing this object:

@rjurney
rjurney / pandas_dataframe_parallel.py
Created Apr 13, 2020
How to run a method on a field of a pandas DataFrame and set the result to another field
View pandas_dataframe_parallel.py
def process_split(df: pd.DataFrame, f: types.FunctionType, in_key: str, out_key: str):
"""Process each chunk of a DataFrame, apply a funtion on an in_key and store it in an out_key"""
rows = []
for index, row in df.iterrows():
result = f(row[in_key])
row[out_key] = result
rows.append(row)
df_out = pd.DataFrame(rows)
@rjurney
rjurney / pacoid1.py
Last active Apr 12, 2020
Dear @pacoid, what is the right way to make PyTextRank in spaCy work in parallel?
View pacoid1.py
from multiprocessing import cpu_count
# Use all (96?) cores
n_cores = cpu_count()
print(f'Total cores: {n_cores}')
summaries = []
for doc in nlp.pipe(
df['readme'].astype('unicode').values,
@rjurney
rjurney / bzip_validate.sh
Last active Mar 29, 2020
How to occupy 64 cores with bzip2 validating files if you're a MORON :)
View bzip_validate.sh
#!/bin/bash
for year in 2017 2018 2019 2020
do
for months in 01_02 03_04 05_06 07_08 09_10 11_12
do
month1=$(echo "${months}" | cut -d '_' -f1)
month2=$(echo "${months}" | cut -d '_' -f2)
# 30 days x 2 is around 64 pids
@rjurney
rjurney / support.txt
Created Mar 13, 2020
Template for NextDoor Isolation Support Groups
View support.txt
IF YOU’RE SICK AND NEED SUPPLIES, POST YOUR NEEDS HERE.
I wanted to create a thread where people in quarantine can ask for help for supplies when they’re in need. This supports their immune system with potentially life saving food, beverage and medicine and can be done in a SAFE manner by dropping supplies outside the door.
Hopefully this won’t happen but based on what is going on in Italy (which has a first rate healthcare system) our healthcare system may be overloaded and there may be no room in a hospital, so very sick people will be at home and in need of help.
My neighbor brought me orange juice because I’m sick. My doctor thinks I have the flu and prescribed Tamiflu. I’m glad I know my neighbors.
You can’t perform that action at this time.