download and convert and quantize a model from huggingface

model-forget-me-not.py

from huggingface_hub import snapshot_download
model_id="01-ai/Yi-34B"
snapshot_download(repo_id=model_id, local_dir="vicuna-hf",
                  local_dir_use_symlinks=False, revision="main")

# convert to gguf clone llama.cpp first git clone git clone https://github.com/ggerganov/llama.cpp.git

# python3 convert.py /Users/jc/projects/models/01ai/vicuna-hf --outfile /Users/jc/01ai.gguf --outtype f16

# ./quantize /Users/jc/01ai.gguf /Users/jc/01ai-q5.gguf Q5_K_M

This is the latest and greatest open source model as of november 2023 from https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard