jmftrindade

Local LLMs on Windows using WSL2 (Ubuntu 22.04)

Assumes:

• WSL2 Install + Ubuntu-22.04 distro
• Basic WSL cmds: https://learn.microsoft.com/en-us/windows/wsl/basic-commands

Run powershell as administrator and enter Ubuntu distro.

From within Ubuntu:

jmftrindade

A Google search shows how tricky it can be to get an X570 Aorus motherboard to correclty detect your GPU, especially if you're already using one of the PCIe slots for a GPU + another M2 PCIe slot for an NVMe SSD, which was the case for my build.

Therefore, neither my BIOS nor my OS detected the 2nd GPU card right away. I spent some time reading the motherboard manual, but nothing there was immediately helpful. The section on PCIe did mention how to split the available lanes in case multiple were being used for PCIe Gen 4.0 devices (which by default will use x16 lanes), and that's what we rely on here via the "bifurcation" BIOS feature listed below.

I got it to work successfully, so sharing it here in case somebody out there finds this useful.

Assumptions:

• Yo

jmftrindade

Clone and build llama.cpp

As of writing (Nov 22nd 2023), METAL is enabled by default, and arm64 is correctly detected: no need for special cmake flags.

git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp/
mkdir build && cd build/ && cmake .. && make -j && cd ..

Download the model

jmftrindade

PyBind11 example

1) Deps

pip install pybind11

2) Python wrapper

jmftrindade

Burn DVD-video onto USB or DVD5 on Mac OS X

Step 1) Install dependencies

Needed for mkisofs tool:

brew install dvdrtools

jmftrindade

Capture Mini DV footage from camcorder using ffmpeg

Assumes:

• OSX
• ffmpeg (brew install ffmpeg)

Step 1 - Connect camcorder

For OSX via USB-C, the following are required:

jmftrindade

Assumes:

1. You already have access to github's Copilot (Copilot Chat is not yet available for neovim as of Mar 22nd 2023).
2. Ubuntu 20.04+ (should work for other Debian distros as well).
3. If you were using vim before, that your .vimrc does not rely on features that are not supported by neovim. I use powerline and YouCompleteme for vim, which don't vibe with neovim, so I had to make a couple of changes to my .vimrc. Take a look at this commit for an example on how to do that.

Step 1 - Install Neovim

Docs

jmftrindade

struct A {
	static std::size_t allocated;
  
  	static void* operator new(std::size_t size) {
    	allocated += size / sizeof(A);
      	return ::operator new(size);
    }
 

jmftrindade

Installing ndctl, pmdk and ipmctl:

Keep the installation directories around for both of these libs, as the only way to uninstall them is by going into the original install dirs, and then make uninstall.

Required for both ndctl and pmdk:

$ apt install pkg-config
$ export PKG_CONFIG_PATH=/usr/lib/x86_64-linux-gnu/pkgconfig:/lib/x86_64-linux-gnu/pkgconfig:/usr/lib/pkgconfig:${PKG_CONFIG_PATH}

jmftrindade

Physical vs logical cores

lscpu --all --extended