Birch-san

For CUDA 12, see Installing CUDA 12.1.1 + PyTorch nightly + Python 3.10 on Ubuntu 22.10 for how to install Nvidia driver 530, gcc 12 and CUDA 12.1.1 libraries.
If you want CUDA 11.8, then you can use latest Nvidia driver from Production branch, 525, with gcc 11.

Activate your conda environment, if you haven't done so already.

CUDA 11:
Make sure gcc 11 is the default gcc for your OS, or select gcc 11 explicitly.
CUDA 12:
Make sure gcc 12 is the default gcc for your OS, or select gcc 12 explicitly.
Check CUDA_DIR below points to the CUDA installation you wish to use.

Birch-san

Loading LLaMA via Huggingface + Safetensors, with 4-bit quantization

Let's say we're trying to load a LLaMA model via AutoModelForCausalLM.from_pretrained with 4-bit quantization in order to inference from it:

python -m generate.py

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, LlamaTokenizerFast, LlamaForCausalLM
import transformers

Birch-san

Installing CUDA 12.1.1 + PyTorch nightly + Python 3.10 on Ubuntu 22.10

Should you keep your NVIDIA driver?

CUDA 12.1.1 toolkit is gonna offer to install Nvidia driver 530 for us. It's from New Feature branch. It's likely to be newer than the default Nvidia driver you would've installed via apt-get (apt would prefer to give you 525, i.e. Production Branch).

If you're confident that you already have a new enough Nvidia driver for CUDA 12.1.1, and you'd like to keep your driver: feel free to skip this "uninstall driver" step.

But if you're not sure, or you know your driver is too old: let's uninstall it. CUDA will install a new driver for us later.

Birch-san

Uploading a folder of many files to HF, by chunking it into .tars

So you generated 50000 images for computing FID or whatever, and now you want to upload those samples to HF.
You try, but one of the filetransfers fails, and you lose all your progress.
I mean it'd be nice if HF could just… fix this… like, put retries into huggingface-cli upload instead of just discarding tens of gigabytes of progress… but we live in the world in which we live.

So let's make it easier. instead of 50k small files, let's upload 50 big files. Collate 'em into .tars.

I'm not sure this makes a valid WDS, but it's close; I think you would need to rename the files to 000000.img.png if you wanted that.

Birch-san

supported sigmas:
tensor([ 0.0292,  0.0413,  0.0507,  0.0586,  0.0656,  0.0720,  0.0779,  0.0834,
         0.0886,  0.0936,  0.0983,  0.1028,  0.1072,  0.1114,  0.1155,  0.1195,
         0.1234,  0.1271,  0.1308,  0.1345,  0.1380,  0.1415,  0.1449,  0.1482,
         0.1515,  0.1548,  0.1580,  0.1611,  0.1642,  0.1673,  0.1704,  0.1734,
         0.1763,  0.1793,  0.1822,  0.1850,  0.1879,  0.1907,  0.1935,  0.1963,
         0.1991,  0.2018,  0.2045,  0.2072,  0.2099,  0.2125,  0.2152,  0.2178,
         0.2204,  0.2230,  0.2256,  0.2281,  0.2307,  0.2332,  0.2358,  0.2383,
         0.2408,  0.2433,  0.2458,  0.2482,  0.2507,  0.2531,  0.2556,  0.2580,
         0.2604,  0.2628,  0.2653,  0.2677,  0.2700,  0.2724,  0.2748,  0.2772,

Birch-san

[url "https://oauth2:github_pat_REDACTED@github.com/"]
	insteadOf = https://github.com/
[url "https://oauth2:github_pat_REDACTED@github.com/MYCOOLORG/"]
	insteadOf = git@github.com:MYCOOLORG/

Birch-san

#!/usr/bin/env bash

# Licensed by author Alex Birch under CC BY-SA 4.0
# https://creativecommons.org/licenses/by-sa/4.0/

# detects whether any of your passwords have been exposed in a data breach, by
# submitting (prefixes of hashes of) all your passwords to Troy Hunt's
# Pwned Passwords API.
# https://haveibeenpwned.com/Passwords

Birch-san

Running GitHub Copilot VSCode extension against local Code Llama model

Tested on NVIDIA RTX 4090, but these instructions also cover AMD and Mac in case you wanna try those.
This guide assumes you are running Linux (I ran this on Ubuntu).

Before you get excited:

Birch-san

Install HF Code Autocomplete VSCode plugin.

We are not going to set an API token. We are going to specify an API endpoint.
We will try to deploy that API ourselves, to use our own GPU to provide the code assistance.

We will use bigcode/starcoder, a 15.5B param model.
We will use NF4 4-bit quantization to fit this into 10787MiB VRAM.
It would require 23767MiB VRAM unquantized. (still fits on a 4090, which has 24564MiB)!

Setup API

Birch-san

from torch import FloatTensor, LongTensor, Tensor, Size, lerp, zeros_like
from torch.linalg import norm

# adapted to PyTorch from:
# https://gist.github.com/dvschultz/3af50c40df002da3b751efab1daddf2c
# most of the extra complexity is to support:
# - many-dimensional vectors
# - v0 or v1 with last dim all zeroes, or v0 ~colinear with v1
#   - falls back to lerp()
#   - conditional logic implemented with parallelism rather than Python loops