Skip to content

Instantly share code, notes, and snippets.

@hubgit

hubgit/mlx-mixtral-macos.md

Created January 9, 2024 22:45
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hubgit/1a733d0acf84b3c2e2779dc2577c5235 to your computer and use it in GitHub Desktop.
Save hubgit/1a733d0acf84b3c2e2779dc2577c5235 to your computer and use it in GitHub Desktop.
Download ZIP
Run Mixtral-8x7B-Instruct-v0.1 LLM on macOS (Apple Silicon) using MLX
Raw
mlx-mixtral-macos.md 
brew install git-lfs 

git clone https://github.com/ml-explore/mlx-examples
cd mlx-examples/llms/mixtral
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
cd Mixtral-8x7B-Instruct-v0.1
git lfs pull --include "consolidated.*.pt" # ~100GB
git lfs pull --include "tokenizer.model"

cd ..
pip3 install -r requirements.txt
python3 convert.py --torch-path Mixtral-8x7B-Instruct-v0.1 -q --q-bits 4
code prompt.txt # add prompt to this file, wrapped in [INST] … [/INST]
python3 mixtral.py --model-path mlx_model --max-tokens 4096 --temp 0 --prompt "$(cat prompt.txt)"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment