Skip to content

Instantly share code, notes, and snippets.

View malcolmgreaves's full-sized avatar

Malcolm Greaves malcolmgreaves

View GitHub Profile
@malcolmgreaves
malcolmgreaves / setup_reqs_no_versions_from_txt.py
Created October 7, 2024 23:49
setup.py for getting requirements sans version information from a requirements.txt file for install_requires
from setuptools import setup
if __name__ == "__main__":
def has_requirement(x: str) -> bool:
x = x.strip()
if x.startswith("#"):
return False
return True
@malcolmgreaves
malcolmgreaves / is_literal.py
Created April 29, 2024 21:40
Check if a value is of a literal type.
from typing import Any, Literal
def is_literal(literal_type: type, value: Any) -> bool:
"""Returns True iff the `value` is a variant of the input `literal_type`. False otherwise.
Raises a `ValueError` iff the input `literal_type` is not a `typing.Literal`.
"""
if not hasattr(literal_type, '__origin__') or literal_type.__origin__ != Literal:
raise ValueError(f"Expecting literal type, not {literal_type=}")
@malcolmgreaves
malcolmgreaves / local_temp_dir_rm_exit_trap.sh
Last active April 12, 2024 17:59
Reusable bash functions for creating a local temporary directory with rm exit trap.
#!/usr/bin/env bash
set -euo pipefail
####################################################################
#
# Reusable functions for creating a local temporary directory:
# - [mk_tmp_dir] create local directory with unique name
# - [cleanup] add exit trap to rm this directory
#!/usr/bin/env bash
apt update
#
# install lmdb
#
apt install -y liblmdb-dev
LMDB_FORCE_SYSTEM=1 LMDB_FORCE_CFFI=1 pip install cffi
@malcolmgreaves
malcolmgreaves / conda.Dockerfile
Last active April 5, 2024 19:18
A Dockerfile that installs conda. Uses a GPU-enabled image (with CUDA) as a base, but miniconda install & setup is portable.
# syntax=docker/dockerfile:1.3
ARG UBUNTU_VERSION=18.04
ARG CUDA_VERSION=11.3.1
# Or use a different image.
FROM nvidia/cuda:${CUDA_VERSION}-cudnn8-devel-ubuntu${UBUNTU_VERSION}
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# #
# system packages #
@malcolmgreaves
malcolmgreaves / bug_pandas_map_mangles_column_datatype.py
Created February 13, 2024 19:35
Demonstration showing a bug in Pandas: it automatically converts datetime columns into a different pandas-specific type, even when the original column has `dtype=object`.
from datetime import datetime
import pandas as pd
now = datetime.now()
df = pd.DataFrame.from_dict(
{
"created_at": pd.Series([now, now - timedelta(seconds=100), now + timedelta(seconds=10)], dtype='object'),
}
@malcolmgreaves
malcolmgreaves / pandas_required_columns.py
Last active February 10, 2024 02:18
Conceptual framework for writing Pandas DataFrame code where required columns are not only documented, but parameterized. This establishes an interface between the name of a column in code vs. its name in the data.
from abc import ABC
from dataclasses import dataclass
from typing import List, NamedTuple, Sequence, Type, TypeVar
import pandas as pd
__all__: Sequence[str] = (
# main abstraction & utilities for columns required in a dataframe
"Columns",
@malcolmgreaves
malcolmgreaves / env_var_secret.Dockerfile
Created January 11, 2024 21:35
Example passing a secret value via an env var to a docker build.
# Run this example:
#
# mysecret=SECRET_VALUE docker build --secret id=mysecret,env=mysecret -f Dockerfile -t deleteme .
#
FROM debian:trixie-slim
RUN <<EOF cat >> file
#!/bin/bash
if [[ -z "\${MYSECRET}" ]]; then
echo "No MYSECRET env var!!!"
@malcolmgreaves
malcolmgreaves / requirements.txt--pyproject.toml
Created January 9, 2024 22:08
A pyproject.toml that uses setuptools & gets `dependencies` dynamically from a requirements.txt file.
[build-system]
requires = ["setuptools", "wheel", "setuptools_scm"]
build-backend = "setuptools.build_meta"
[project]
name = "mypackage"
requires-python = ">=3.10"
dynamic = ["dependencies"]
[tool.setuptools.dynamic]
@malcolmgreaves
malcolmgreaves / Dockerfile--cuda_117-torch_113-geometric_204
Last active January 5, 2024 20:58
Dockerfile based on Ubuntu 22.04 that has CUDA 11.7 dev libraries & drivers installed alongside PyTorch 1.13 and Torch-Geometric 2.0.4 libraries.
FROM nvidia/cuda:11.7.1-devel-ubuntu22.04
RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
apt-get install -y software-properties-common && \
add-apt-repository -y ppa:deadsnakes/ppa && \
apt-get install -y \
python3-setuptools python3-dev swig \
wget git unzip tmux vim tree xterm \
build-essential gcc \