Running Llama on M1 Pro 16GB

llama2-mac-gpu.sh

#!/bin/bash

# Adapted from https://gist.github.com/adrienbrault/b76631c56c736def9bc1bc2167b5d129

# Clone llama.cpp
# git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

# Build it
LLAMA_METAL=1 make

# Download model

export MODEL=llama-2-7b-chat.ggmlv3.q2_K.bin
wget "https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/resolve/main/${MODEL}"

# Run

echo "Prompt: " \
    && read PROMPT \
    && ./main -t 10 \
        -ngl 32 \
        -m ${MODEL} \
        --color \
        -c 2048 \
        --temp 0.7 \
        --repeat_penalty 1.1 \
        -n -1 \
        -p "
[INST] <<SYS>>
You are a helpful, respectful and honest assistant. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
<</SYS>>
${PROMPT}[/INST]"