Skip to content

Instantly share code, notes, and snippets.

View jlmelville's full-sized avatar

James Melville jlmelville

View GitHub Profile

June 28 2023 O frabjous day! After much fruitless occasional faffing around with cmake settings over the course of several months, new CUDA update for WSL Ubuntu means that I am now able to build the nearest neighbors library Faiss with GPU support for my 1080 laptop graphics card. Here is the setup that worked for me.

Why would you want to build your own Faiss? Well, if you are using pip to install faiss-gpu, you can't get anything more recent than version 1.7.2. Also, with recent dependency management changes with newer Pythons (particularly Python 3.11), some packages are going to be playing catch-up with getting up to parity, so it may be useful to build from source. I don't remember how bad it was trying to install faiss-gpu on Python 3.11 with pip, but it must have been fairly un-fun, because I gave up and scuttled back to the relatively welcoming environs of Python 3.10 pretty quickly after briefly dipping my

@jlmelville
jlmelville / ubuntu-cuda-wsl2.md
Last active September 5, 2022 17:28
Fixing the legacy gpg keyring warning when installing CUDA on Ubuntu for WSL2

Ubuntu, CUDA, WSL2 and legacy keyrings

In the Ubuntu tutorial on enabling CUDA with WSL2, and specifically in part 3, you are directed to use apt-key to point Ubuntu at the correct toolkit package. On Ubuntu 22.04 this will lead to the following warning whenever you run apt-get update:

 Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
@jlmelville
jlmelville / ng20.md
Last active December 24, 2021 20:19
20 newsgroups python/R
@jlmelville
jlmelville / transitive_dependencies.R
Created November 30, 2021 16:39
transitive dependencies in R
# modified slightly from https://gist.github.com/floybix/6533090
distrib.pkgs <- library(lib.loc=R.home("library"))$results[,1]
dependencies <-
function(pkg, dependencies = c("Depends", "Imports", "LinkingTo"),
pl = installed.packages())
{
if (!(pkg %in% rownames(pl))) stop("unknown pkg ", pkg)
fields <- pl[pkg, dependencies]
@jlmelville
jlmelville / revdepcheck.R
Last active November 29, 2021 05:04
Testing reverse dependencies for R packages
install.packages("devtools")
devtools::install_github("r-lib/revdepcheck")
# may need to install bioconductor
if (!require("BiocManager", quietly = TRUE)) {
install.packages("BiocManager")
BiocManager::install(version = "3.14")
}
library(revdepcheck)
@jlmelville
jlmelville / read-and-pickle.py
Last active November 11, 2021 05:48
reading CSV into Pandas but also pickling it at the same time
import os
import pickle
from pathlib import Path
def read_data(dataset, suffix=None, dir=None, repickle=False):
"""Set `repickle=True` to rewrite pickle file if CSV has changed"""
home = str(Path.home())
if dir is None:
dataset_path = os.path.join(home, "dev", "datasets")
@jlmelville
jlmelville / zca_whiten.R
Created May 6, 2020 02:30
ZCA whitening
whitening_matrix <- function(X, eps = 1e-8, zca = FALSE) {
X <- scale(X, center = TRUE, scale = FALSE)
s <- svd(X, nu = 0)
Linvsqr <- sqrt(nrow(X) - 1) / (s$d + eps)
W <- Linvsqr * t(s$v)
if (zca) {
W <- s$v %*% W
}
@jlmelville
jlmelville / smallvis_bench.R
Created May 3, 2020 23:43
useful functions for running smallvis on multiple datasets
benchmark <- function(datasets = smallvis_datasets(), fun = smallvis::smallvis,
vfun = NULL, mfun = NULL, ...) {
varargs <- list(...)
if (is.null(varargs$ret_extra)) {
varargs$ret_extra <- TRUE
}
dataset_names <- names(datasets)
varargs_orig <- varargs
@jlmelville
jlmelville / spectral.py
Created March 14, 2020 18:00
Graph Laplacians (in Python)
import collections
import pprint
import numpy as np
import scipy as sp
import scipy.linalg
AffDeg = collections.namedtuple("AffDeg", ["W", "D"])
Lap = collections.namedtuple("Lap", ["L", "Lsym", "Lrw", "P"])
Eig = collections.namedtuple("Eig", ["vectors", "values", "lengths"])
@jlmelville
jlmelville / spectral.R
Last active March 14, 2020 18:06
Graph Laplacians (in R)
# Create the degree matrix D from an affinity/adjacency matrix
degmat <- function(X) {
diag(colSums(X))
}
# A random affinity matrix
randw <- function(n = 3) {
X <- matrix(rnorm(n * n), nrow = n)
X <- X * X
X <- t(X) + X