Skip to content

Instantly share code, notes, and snippets.

Avatar
🙃

Mike Clark wassname

🙃
View GitHub Profile
@wassname
wassname / aws_keepassxc.md
Created Sep 1, 2020
How to use keepassxc and browser integration with aws console sign's
View aws_keepassxc.md

For your IAM user you get a csv of credentials like this

User name,Password,Access key ID,Secret access key,Console login link
USERNAME,PASSWORD,ACCESS_KEY,SECRET_KEY,https://0123456.signin.aws.amazon.com/console

If your region is sydney (ap-southeast-2) in keepass you enter:

Title: USERNAME/COMPANY

@wassname
wassname / azureml_py36_pytorch_conda.yaml
Created Jul 3, 2020
azure data science vm packages azureml_py36_pytorch
View azureml_py36_pytorch_conda.yaml
name: azureml_py36_pytorch
channels:
- pytorch
- conda-forge
- anaconda
- defaults
dependencies:
- _libgcc_mutex=0.1=main
- _py-xgboost-mutex=2.0=cpu_0
- _pytorch_select=0.2=gpu_0
View times.py
def row2date(row, tz="Australia/Perth"):
"""Parse time columns."""
return pd.Timestamp(
year=int(row.Year),
month=int(row.Month),
day=int(row.Day),
hour=int(row.Hour),
minute=int(row.Minute),
second=int(row.Seconds),
@wassname
wassname / smooth.py
Last active Apr 26, 2020
utils for smoothing or generating smooth noise
View smooth.py
def smooth_noise(shp, dim=1, smoothing=30):
"""
Generate smoothed random noise that looks like a random walk.
- shp: shape of noise
- dim: dimension to smooth along
- smooth_steps: smoothing steps
@wassname
wassname / radial.py
Created Apr 26, 2020
machine learning numpy/pytorch utils for circular or radial variables
View radial.py
from torch import nn
import torch
import torch.nn.functional as F
import numpy as np
def radneg2radpos(f):
"""convert radians in range [-pi,pi] to [0,2*pi]"""
return np.where(f < 0, f + np.pi * 2, f)
@wassname
wassname / pandas_hash.py
Last active Mar 20, 2020
Hash pandas and numpy objects in a way that persists between interpreaters (for caching)
View pandas_hash.py
import pandas as pd
import numpy as np
import hashlib
import json
def default(o):
"""Sets are unordered so are no good for hasing"""
if isinstance(o, set):
try:
o = sorted(o)
@wassname
wassname / unbalance_dask_dataframe.py
Created Nov 29, 2019
unbalance_dask_dataframe.py
View unbalance_dask_dataframe.py
%pylab inline
import pandas as pd
import dask.dataframe as dd
def get_unbal_df(size = 100, balance=None):
"""Get a randomly unbalanced df"""
if balance is None:
balance = np.random.randint(-100, 100)
if balance<0:
@wassname
wassname / data_block_binary.py
Last active Nov 26, 2019
fastai binary class data block
View data_block_binary.py
from typing import Iterator, Collection
from fastai.data_block import CategoryListBase
from fastai.text import *
class BinaryProcessor(CategoryProcessor):
def create_classes(self, classes):
self.classes = classes
if classes is not None: self.c2i = {0:0, 1:1}
def generate_classes(self, items):
@wassname
wassname / dataset_cache_with_dask.py
Last active May 29, 2020
Cache a torch dataset to npy files using dask
View dataset_cache_with_dask.py
"""
Cache a torch dataset to npy files using dask
url:https://gist.github.com/wassname/f38f8774b6f97977b660d20dfa0f0036
lic:MIT
author:wassname
usage:
batch_size=16
chunk_size=batch_size*4
@wassname
wassname / jaccard_pytorch.py
Created Sep 16, 2019
jaccard distance loss pytorch [draft]
View jaccard_pytorch.py
#!/usr/bin/env python
# coding: utf-8
get_ipython().run_line_magic('pylab', 'inline')
import torch
def jaccard_distance_loss(y_true, y_pred, smooth=100):
"""
Jaccard = (|X & Y|)/ (|X|+ |Y| - |X & Y|)
= sum(|A*B|)/(sum(|A|)+sum(|B|)-sum(|A*B|))