Skip to content

Instantly share code, notes, and snippets.

@bigsnarfdude

bigsnarfdude/tableVRAM.md

Last active May 2, 2024 01:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bigsnarfdude/fac2bacc78e98714ce6e94335b93e37b to your computer and use it in GitHub Desktop.
Save bigsnarfdude/fac2bacc78e98714ce6e94335b93e37b to your computer and use it in GitHub Desktop.
Download ZIP
tableVRAM.md
Raw
tableVRAM.md 

Method	Bits	7B	13B	30B	65B	8x7B
Full	16	160GB	320GB	600GB	1200GB	1000GB
Freeze	16	20GB	40GB	120GB	240GB	200GB
LoRA	16	16GB	32GB	80GB	160GB	120GB
QLoRA	8	10GB	16GB	40GB	80GB	80GB
QLoRA	4	6GB	12GB	24GB	48GB	32GB




Large models are not easily accessible
Model   Inference memory    Fine-tuning memory
Т5-11B    22 GB   176 GB
LLaMA-33B   66 GB   396 GB
LLaMA-65B   130 GB    780 GB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment