Skip to content

Instantly share code, notes, and snippets.

Avatar

Thomas Aarholt thomasaarholt

  • University of Oslo
  • Oslo, Norway
View GitHub Profile
@thomasaarholt
thomasaarholt / train_test_split_polars.py
Created Jul 24, 2022
Train and test split function for polars dataframes
View train_test_split_polars.py
def train_test_split(
df: pl.DataFrame, train_fraction: float = 0.75
) -> Tuple[pl.DataFrame, pl.DataFrame]:
"""Split polars dataframe into two sets.
Args:
df (pl.DataFrame): Dataframe to split
train_fraction (float, optional): Fraction that goes to train. Defaults to 0.75.
@thomasaarholt
thomasaarholt / lightgbm_gpu.sh
Last active Jul 24, 2022
Building lightgbm with GPU support from source on Red Hat with mload
View lightgbm_gpu.sh
module load CUDA/11.0 CMake/3.21.1-GCCcore-11.2.0 Boost/1.74.0-GCC-10.2.0
pip install lightgbm --install-option=--gpu
# If running from notebook, ensure that the following contains Boost:
# import os
# os.environ["LD_LIBRARY_PATH"].split(":")
@thomasaarholt
thomasaarholt / xgboost_gpu.sh
Last active Jul 24, 2022
Building XGBoost with GPU support from source on Red Hat with mload
View xgboost_gpu.sh
# must not use gcc version > 10
module load CUDA/11.0 CMake/3.21.1-GCCcore-11.2.0
# Clone with recursive! Otherwise you get dmlc error later on!
git clone --recursive https://github.com/dmlc/xgboost
cd xgboost/python-package
# clean out any old build files that may break things if try to recompile
rm -rf build
@thomasaarholt
thomasaarholt / docker_terminal.sh
Created Jun 6, 2022
Instructions for building a docker image and logging into its bash terminal
View docker_terminal.sh
# From a directory with a Dockerfile
# Example Dockerfile with "Dockerfile" as filename
FROM python:3.10-slim-buster
RUN pip install numpy
# Then, run the following
docker build -t your_image_name .
@thomasaarholt
thomasaarholt / cmap.py
Created Jan 26, 2022
Linear colormap from one rgb color to another
View cmap.py
from matplotlib.colors import LinearSegmentedColormap
def black_to_color(high_color: tuple, low_color: tuple = (0,0,0), name: str = "my_color", steps=256):
"Get linear colormap from one rgb color to another, defaulting from black"
r1,g1,b1 = low_color
r2,g2,b2 = high_color
cdict = {
'red': [(0.0, r1, r1),
@thomasaarholt
thomasaarholt / gist:4c63d9f15a6895a4153e9bfa98f75360
Created Sep 27, 2021
jupyter lab settings with auto closing brackets enabled in docker image
View gist:4c63d9f15a6895a4153e9bfa98f75360
cat /root/.jupyter/lab/user-settings/@jupyterlab/notebook-extension/tracker.jupyterlab-settings
{
// Notebook
// @jupyterlab/notebook-extension:tracker
// Notebook settings.
// **************************************
// Code Cell Configuration
// The configuration for all code cells.
"codeCellConfig": {
@thomasaarholt
thomasaarholt / check_docker_running.sh
Created Aug 24, 2021
Simple bash script to test if Docker is running
View check_docker_running.sh
if docker info > /dev/null 2>&1; then
: # pass
else
echo "Docker Daemon is not running. Please open it and retry."
exit -1
fi
@thomasaarholt
thomasaarholt / dask_inv_bench.py
Created Aug 1, 2021
Benchmarking dask linalg inv on nd arrays
View dask_inv_bench.py
import numpy as np
import dask.array as da
x = da.random.random((120,500,500), chunks=(40, 500, 500)) # 80 MB chunks
X = x.compute()
@da.as_gufunc(signature="(n,n)->(n,n)", output_dtypes=float, vectorize=True)
def gufunc(x):
return np.linalg.inv(x)
@thomasaarholt
thomasaarholt / gufunc.py
Created Aug 1, 2021
gufunc error with whitespace in dask
View gufunc.py
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/var/folders/yt/z22_f0896l74f8gs3_20vc_00000gn/T/ipykernel_88946/1449311839.py in <module>
7 return np.linalg.inv(x)
8
----> 9 y = gufoo(b)
~/mambaforge/envs/py/lib/python3.9/site-packages/dask/array/gufunc.py in __call__(self, *args, **kwargs)
659
660 def __call__(self, *args, **kwargs):
@thomasaarholt
thomasaarholt / bilinear_interpolate.py
Created Jul 8, 2021
Bilinearly interpolate points from an image using NumPy or CuPy
View bilinear_interpolate.py
def bilinear_interpolate(img, points, clipped_nan=True):
"""Bilinearly interpolate points from an image using NumPy or CuPy
Args:
img: Image of shape (Y, X) to interpolate from.
points: array of shape (2, N) of (y, x) coordinates
clipped_nan: If True, the value of coordinates outside the image shape
are set to nan. Otherwise they are clipped to the image edge.
Returns: