illtellyoulater
tl;dr use Linux, install bitsandbytes (either globally or in KAI's conda env, add load_in_8bit=True, device_map="auto" in model pipeline creation calls)
Many people are unable to load models due to their GPU's limited VRAM. These models contain billions of parameters (model weights and biases), each of which is a 32 (or 16) bit float. Thanks to the hard work of some researchers [1], it's possible to run these models using 8-bit numbers, which halves the required amount of VRAM compared to running in half-precision. E.g. if a model requires 16GB of VRAM, running with 8-bit inference only requires 8GB.
This guide was written for KoboldAI 1.19.1, and tested with Ubuntu 20.04. These instructions are based on work by Gmin in KoboldAI's Discord server, and Huggingface's efficient LM inference guide.
This gist is a note about install nvidia-docker in Pop!_OS 20.10. nvidia-docker is used to help docker containers compute on GPU.
The basic installcation is in Nvidia's offical documentation. But there are a few tweaks to make it work on Pop!_OS 20.10.
No surprise. Follow the offical documentaion should work.
A curated list of arrrrrrrrr!
| Filter | Description | Example |
|---|---|---|
| allintext | Searches for occurrences of all the keywords given. | allintext:"keyword" |
| intext | Searches for the occurrences of keywords all at once or one at a time. | intext:"keyword" |
| inurl | Searches for a URL matching one of the keywords. | inurl:"keyword" |
| allinurl | Searches for a URL matching all the keywords in the query. | allinurl:"keyword" |
| intitle | Searches for occurrences of keywords in title all or one. | intitle:"keyword" |
| echo "If your SSH key has a passphrase, you might need to enter it each time. Consider using ssh-agent or removing the passphrase"; | |
| while read p; do | |
| echo "$p"; | |
| git clone "$p"; | |
| done < $1 |
| adb help // List all comands | |
| == Adb Server | |
| adb kill-server | |
| adb start-server | |
| == Adb Reboot | |
| adb reboot | |
| adb reboot recovery | |
| adb reboot-bootloader |
| # Output a single frame from the video into an image file: | |
| ffmpeg -i input.mov -ss 00:00:14.435 -vframes 1 out.png | |
| # Output one image every second, named out1.png, out2.png, out3.png, etc. | |
| # The %01d dictates that the ordinal number of each output image will be formatted using 1 digits. | |
| ffmpeg -i input.mov -vf fps=1 out%d.png | |
| # Output one image every minute, named out001.jpg, out002.jpg, out003.jpg, etc. | |
| # The %02d dictates that the ordinal number of each output image will be formatted using 2 digits. | |
| ffmpeg -i input.mov -vf fps=1/60 out%02d.jpg |
| <?php | |
| // For now file storage requires that the file specified is created already and chmodded to allow writing. | |
| /** | |
| * Connection info logging script created by Xeru | |
| * | |
| * Website: https://xeru.me | |
| * Twitter: https://twitter.com/Xeru_ | |
| * GitHub: https://github.com/exec |
| @echo off | |
| echo Uninstalling KB3075249 (telemetry for Win7/8.1) | |
| start /w wusa.exe /uninstall /kb:3075249 /quiet /norestart | |
| echo Uninstalling KB3080149 (telemetry for Win7/8.1) | |
| start /w wusa.exe /uninstall /kb:3080149 /quiet /norestart | |
| echo Uninstalling KB3021917 (telemetry for Win7) | |
| start /w wusa.exe /uninstall /kb:3021917 /quiet /norestart | |
| echo Uninstalling KB3022345 (telemetry) | |
| start /w wusa.exe /uninstall /kb:3022345 /quiet /norestart | |
| echo Uninstalling KB3068708 (telemetry) |