Skip to content

Instantly share code, notes, and snippets.

@bigsnarfdude

bigsnarfdude/table_vram.md

Created January 23, 2024 07:57
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bigsnarfdude/e169657fe0dad5d07f45a1baf994be65 to your computer and use it in GitHub Desktop.
Save bigsnarfdude/e169657fe0dad5d07f45a1baf994be65 to your computer and use it in GitHub Desktop.
Download ZIP
table_vram.md
Raw
table_vram.md

Hardware Requirement

Method Bits 7B 13B 30B 65B 8x7B
Full 16 160GB 320GB 600GB 1200GB 900GB
Freeze 16 20GB 40GB 120GB 240GB 200GB
LoRA 16 16GB 32GB 80GB 160GB 120GB
QLoRA 8 10GB 16GB 40GB 80GB 80GB
QLoRA 4 6GB 12GB 24GB 48GB 32GB
@bigsnarfdude
Copy link
Author

Mixtral instruct Q5_K_M gguf
Total Tokens: 540, Time: 47.31s. Tokens per second 10.00

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment