Skip to content

Instantly share code, notes, and snippets.

@skanga
Created February 7, 2024 20:46
Show Gist options
  • Save skanga/8280e68b6de7353a97e40fb7133ca735 to your computer and use it in GitHub Desktop.
Save skanga/8280e68b6de7353a97e40fb7133ca735 to your computer and use it in GitHub Desktop.
Build llama.cpp with OpenBLAS on Windows
:: ========================================
:: BUILD LLAMA.CPP WITH OPENBLAS ON WINDOWS
:: ========================================
set BASE_DIR=c:\bin
mkdir %BASE_DIR%
mkdir c:\tmp
cd c:\tmp
:: Install w64devkit
wget https://github.com/skeeto/w64devkit/releases/download/v1.21.0/w64devkit-fortran-1.21.0.zip
unzip w64devkit-fortran-1.21.0.zip
move w64devkit %BASE_DIR%
set W64DEVKIT=%BASE_DIR%\w64devkit
:: Install OpenBLAS
wget https://github.com/OpenMathLib/OpenBLAS/releases/download/v0.3.26/OpenBLAS-0.3.26-x64.zip
unzip -xo OpenBLAS-0.3.26-x64.zip lib/libopenblas.a
move .\lib\* %W64DEVKIT%\x86_64-w64-mingw32\lib\
unzip -xo OpenBLAS-0.3.26-x64.zip include\*
move .\include\* %W64DEVKIT%\x86_64-w64-mingw32\include\
:: Clone the llama.cpp Repo & build it
set PATH=%W64DEVKIT%\bin;%PATH%
%W64DEVKIT%\w64devkit
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
:: Fix the Makefile for ERROR: undefined reference to `cblas_sgemm'
sed -i "s|MK_LDFLAGS += \$(shell pkg-config --libs openblas)|#&\\n\\tMK_LDFLAGS += -lopenblas -L/usr/local/lib|g" Makefile
make LLAMA_OPENBLAS=1
:: Test it out
start server -m <some-model-file.gguf>
curl -s --request POST --url http://localhost:8080/completion --header "Content-Type: application/json" --data '{"prompt": "Why is the sky blue"}' | jq .content | awk '{gsub(/\\n/,"\n")}1'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment