Run llama2 on Metal on MacOS with GGML

readme.md

Automatic script to do this easily:

wget https://gist.githubusercontent.com/paralin/90623c77f167fe7a816ef6bae3caf574/raw/9b6ff17fabb26a2bb845f72dc94e941f0d241755/llama-metal.sh

To use it:

export LLM_PROMPT="Who is JFK?"
bash ./llama-metal.sh

Full information on running the 13b model using metal:

1. git clone https://github.com/ggerganov/llama.cpp
2. cd llama.cpp
3. Building llama.cpp: MacOS: LLAMA_METAL=1 make

llama-metal.sh

#!/bin/bash
set -x

if [ -z "$LLM_PROMPT" ]; then
  echo "Please set the LLM_PROMPT environment variable."
  echo "export LLM_PROMPT=\"Who is JFK?\""
  echo "./llama-metal.sh"
  exit 1
fi

if [ ! -d ./llama.cpp ]; then
  echo "Downloading llama.cpp..."
  git clone https://github.com/ggerganov/llama.cpp
fi

cd ./llama.cpp
if [ ! -f ./main ]; then
  LLAMA_METAL=1 make -j8
fi

export MODEL="wizardlm-13b-v1.2.ggmlv3.q4_K_M.bin"
export MODEL_REPO="TheBloke/WizardLM-13B-V1.2-GGML"
export LLM_PROMPT="A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\n\nUSER: ${LLM_PROMPT}\nASSISTANT:"

# export MODEL="llama-2-7b-chat.ggmlv3.q4_K_M.bin"
# export MODEL_REPO="TheBloke/Llama-2-7B-Chat-GGML"

# Uncomment these lines for a different model
# export MODEL="llama-2-13b.ggmlv3.q4_K_M.bin"
# export MODEL_REPO="TheBloke/Llama-2-13B-GGML"

if [ ! -f ./models/${MODEL} ]; then
  wget \
  	-O ./models/${MODEL} \
  	https://huggingface.co/${MODEL_REPO}/resolve/main/${MODEL}
fi

./main --threads 8 \
       --n-gpu-layers 1 \
       --model ./models/${MODEL} \
       --color \
       --ctx-size 2048 \
       --temp 0.42 \
       --repeat_penalty 1.1 \
       --n-predict -1 \
       --prompt "${LLM_PROMPT}"