-
-
Save mberman84/9b3c281ae5e3e92b7e946f6a09787cde to your computer and use it in GitHub Desktop.
| # Clone the repo | |
| git clone https://github.com/imartinez/privateGPT | |
| cd privateGPT | |
| # Install Python 3.11 | |
| pyenv install 3.11 | |
| pyenv local 3.11 | |
| # Install dependencies | |
| poetry install --with ui,local | |
| # Download Embedding and LLM models | |
| poetry run python scripts/setup | |
| # (Optional) For Mac with Metal GPU, enable it. Check Installation and Settings section | |
| to know how to enable GPU on other platforms | |
| CMAKE_ARGS="-DLLAMA_METAL=on" pip install --force-reinstall --no-cache-dir llama-cpp-python | |
| # Run the local server | |
| PGPT_PROFILES=local make run | |
| # Note: on Mac with Metal you should see a ggml_metal_add_buffer log, stating GPU is | |
| being used | |
| # Navigate to the UI and try it out! | |
| http://localhost:8001/ |
@frenchiveruti for me your tutorial didnt make the trick to make it cuda compatible, BLAS was still at 0 when starting privateGPT.
However, I found that installing llama-cpp-python with a prebuild wheel (and the correct cuda version) works:
python -m pip install llama-cpp-python --prefer-binary --extra-index-url=https://jllllll.github.io/llama-cpp-python-cuBLAS-wheels/AVX2/cu121
based on zylon-ai/private-gpt#1242 (comment)
Segmentation Fault 11 reported on MacbookPro M2Max
ggml_metal_init: found device: Apple M2 Max ggml_metal_init: picking default device: Apple M2 Max ggml_metal_init: default.metallib not found, loading from source make: *** [run] Segmentation fault: 11The error goes away if I were to turn off Metal, but that knocks the performance down a lot! Few details of my Macbook's spec:
Macbook Pro : Apple M2 Max Memory 32 GB Year: 2023 (Sonoma 14.0) Metal Support: Metal 3Any suggestions? Anyone else encountering this error?
I encountered the same error. Did you find a solution ?
windows is a bit rough, somebody assist with windows specific instructions please.
I'm doing it under WSL, follow this guide and you'll have a reasonable starting base
https://learn.microsoft.com/en-us/windows/wsl/tutorials/wsl-vscode
@frenchiveruti for me your tutorial didnt make the trick to make it cuda compatible, BLAS was still at 0 when starting privateGPT. However, I found that installing llama-cpp-python with a prebuild wheel (and the correct cuda version) works:
python -m pip install llama-cpp-python --prefer-binary --extra-index-url=https://jllllll.github.io/llama-cpp-python-cuBLAS-wheels/AVX2/cu121based on imartinez/privateGPT#1242 (comment)
Great job, I know my guide is not in depth because I had to follow multiple guides to make it work so I am not sure from a blank state what are the exact steps, but I think mine approaches quite a lot to the final solution.
You must re-build llama which should happen with:
$env:CMAKE_ARGS='-DLLAMA_CUBLAS=on'; poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python
But YMMV
I had a hard time getting this running on Windows 11. Here is what worked for me.
- Uninstall all python versions
- Then follow this guide for pyenv and poetry (I tried manually a few times but the script mentioned in the link worked better)
- git clone https://github.com/imartinez/privateGPT
- privateGPT
- pyenv install 3.11.6
- pyenv local 3.11.6
- choco install make
- poetry install --with ui, local
- poetry run python scripts/setup
- poetry run python -m private_gpt
- navigate to http://localhost:8001/
Hope this helps someone :)
8. poetry install --with ui, local
I have had huge issues installing this on my Windows PC - this is why i installed GPT4All instead of PrivatGPT a few months ago. But i really want to get this to work.
My issue is that i get stuck at this part: 8. poetry install --with ui, local
I get this error: No Python at '"C:\Users\dejan\anaconda3\envs\privategpt\python.exe'
I have uninstalled Anaconda and even checked my PATH system directory and i dont have that path anywhere and i have no clue how to set the correct path which should be "C:\Program\Python312"
I dont have the C:\Users\dejan\anaconda3 folder (checked with hidden files also).
If you, or anyone one else have a fix for this, please help. Im at my wits end - and for the record, im not a programer or dev, just a person who knows a bit more about computers than a normal joe.
Does anyone know how to set the temperature? Thanks!
8. poetry install --with ui, local
Remove the space between "ui, local"
It should be "poetry install --with ui,local"
You also need to install Python 3.11.6.
PGPT_PROFILES=local make run
Did you ever find a solution? When I turned off METAL, it worked, but it's very slow.
@renatokuipers I was able to install it following your instructions, the only fix i did is set cuda toolkit path:
$env:CUDA_TOOLKIT_ROOT_DIR="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.3";$env:CMAKE_ARGS='-DLLAMA_CUBLAS=on'; poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python
Is there a way to adjust the number of tokens in the models you use? I would like longer AI responses in some cases to offer more detail.
For WINDOWS 11, I used these steps including credit to those who posted. I had to install pyenv. I get consistent runtime with these directions.
Watch Youtube: https://www.youtube.com/watch?v=XFiof0V3nhA
See: https://docs.privategpt.dev/#section/Quick-Local-Installation-steps
Miniconda/Anaconda commands or Command Prompt in Windows 11:
https://gist.github.com/mberman84/9b3c281ae5e3e92b7e946f6a09787cde
After installed, cd to privateGPT: activate privateGPT, run the powershell command below, and skip to step 3) when loading again... Note if it asks for an installation of the huggingface model, try reinstalling poetry in step 2 because there may have been an update that removed it.
WINDOWS INSTALL:
renatokuipers commented •
PrivateGPT Installation Guide for Windows
Step 1) Clone and Set Up the Environment
git clone https://github.com/imartinez/privateGPT
cd privateGPT
conda create -n privategpt python=3.11 -y
conda activate privategpt
After this, restart the terminal and select the Python 3.11.6 interpreter in VS Code.
Another commenter noted how to get the CUDA GPU running:
while you are in the python environment, type "powerhsell"
then run the command
$env:CMAKE_ARGS='-DLLAMA_CUBLAS=on'; poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python
Step 2) Install Poetry
conda install -c conda-forge pipx -y
pipx install poetry
set "PATH=%USERPROFILE%\.local\pipx\venvs\poetry\Scripts;%PATH%"
poetry install --with ui,local
*In case of issues with pypika installation:
pip wheel --use-pep517 "pypika (==0.48.9)"
poetry install --with ui,local
Step 3) Prepare and Run the Application. The step for ren setup setup.py may work for installation but may not work for reloading, continue on if it doesn't when reloading it.
Only when installing...
cd scripts
ren setup setup.py
cd ..
[this is how you run it]
poetry run python scripts/setup.py
set PGPT_PROFILES=local
set PYTHONPATH=.
poetry run python -m uvicorn private_gpt.main:app --reload --port 8001
Wait for the model to download. Once you see "Application startup complete", navigate to 127.0.0.1:8001.
Additional Notes:
Change the Model: Modify settings.yaml in the root folder to switch models.
Running on GPU: To run on GPU, install PyTorch.
For me it was "pip install torch==2.0.0+cu118 --index-url https://download.pytorch.org/whl/cu118"
But for you it could be "pip install torch==2.0.0+cu121 --index-url https://download.pytorch.org/whl/cu118"
Visit the application in browser at:
http://localhost:8001/
Step 2) Install Poetry
conda install -c conda-forge pipx -y
pipx install poetry
set "PATH=%USERPROFILE%.local\pipx\venvs\poetry\Scripts;%PATH%"
poetry install --with ui,local
Can anyone help me with why this doesn't work?
Pipx Installs
Poetry seems to install
Run the path command... should there be confirmation in terminal or just hit return then nothing visual to say it worked?
When running poetry install etc, I get the error:
'poetry' is not recognized as an internal or external command,
operable program or batch file.
This is weird because all the commands including set path seem to work for everyone else, but I'm getting poetry not found, right after it was installed and path set.
Stupidly it works after a reboot but I don't know why. I don't see anyone else having to reboot after installing poetry? WTF is up withy my system.
$env:CMAKE_ARGS='-DLLAMA_CUBLAS=on'; poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python
gives the following error:
The filename, directory name, or volume label syntax is incorrect.
I'm going inane. Every LLM app wants a different package manager, it's own copy of torch inside it's own venv, etc. Install a handfull of LLM tools, and you have 10 copies of torch on your drive.
You can't seem to point these tools at a custom LLM folder, so you have 10 copies of Mistral7b downloaded, each inside the tool's own folder. This is stupid.
Right so NOW the command $env:CMAKE_ARGS='-DLLAMA_CUBLAS=on'; poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python
is kicking off a process but
FAILS to build the wheel
And google results keep bringing me back here and another github thread for PrivateGPT, neither of which has a solution to why building the wheel fails.
Why does building the wheel fail? How to fix?
Visual studio 2022 and all cpp stuff installed
MiniGW etc installed
CUDA all OK, nvcc etc blah blah
My BLAS = 0 still
I got the llama-cpp--python wheel to build finally, by doing this:
In Anaconda Prompt:
pip uninstall -y llama-cpp-python
set CMAKE_ARGS="-DLLAMA_CUBLAS=on"
set FORCE_CMAKE=1
pip install llama-cpp-python==0.1.57 --no-cache-dir
But my BLAS=0 still
So set CMAKE is doing nothing? Or: why isn't that working even though llama built OK this time?
Edit:
Realised this is a very old version of llama-cpp, that'll teach me to not pay close attention. Tried it again simply with
pip install llama-cpp-python --no-cache-dir
And it installed llama-cpp-python-0.2.24.
Still trying to get BLAS=1 with no success though...
The changes below worked for me
llm:
tokenizer: mistralai/Mistral-7B-Instruct-v0.1
local:
llm_hf_repo_id: TheBloke/Mistral-7B-Instruct-v0.1-GGUF
llm_hf_model_file: mistral-7b-instruct-v0.1.Q4_K_M.gguf
PLATFORM: Apple M1
OS: Sonoma 14.2.1
I'm new to this, and I could use some assistance. Visual Studio Code is showing an error while trying to install privateGPT, indicating that a wheel for llama-cpp-python is required. However, it seems that my PC lacks NMake, preventing the creation of the wheel. I would appreciate your help in resolving this issue.
@mberman84 I'm new to this, and I could use some assistance. Visual Studio Code is showing an error while trying to install privateGPT, indicating that a wheel for llama-cpp-python is required. However, it seems that my PC lacks NMake, preventing the creation of the wheel. I would appreciate your help in resolving this issue.
i just automated the step of opening the conda virtual environment where python 3.11.0 was used in my case and opening the url after a 30 second delay via a python script and a bat file . Please use and make changes to suit your own locations ..all credits to chatgpt :-)
getting below error:
S D:\privateGPT-main> poetry run python scripts/setup
Traceback (most recent call last):
File "D:\privateGPT-main\scripts\setup", line 8, in
from private_gpt.paths import models_path, models_cache_path
ModuleNotFoundError: No module named 'private_gpt'
Visual Studio CMAKE libraries are required to build llama-cpp-python
Re-run
poetry install --with ui,local
once installed to fix No module named 'private_gpt'
Windows Subsystem For Linux (WSL) running Ubuntu 22.04 on Windows 11
#Run powershell or cmd as administrator
#install, upgrade and install ubuntu 22.04 LTS
wsl --install -y
wsl --upgrade -y
#install and run ubuntu 22.04 LTS in wsl
wsl --install -d Ubuntu-22.04
give it a username and a simple password
#Setup Ubuntu
sudo apt update --yes
sudo apt upgrade --yes
#install miniconda (its smaller than conda and gets the same functionality...i think)
curl -sSL https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
export PATH=~/miniconda/bin:$PATH
#persist conda in path so that conda doesn't require the full path to run
echo '#conda' >> ~/.bashrc
echo 'PATH=$PATH:$HOME/miniconda/bin' >> ~/.bashrc
#add a conda initialize to your current bash shell
conda init
#get out of the default (base) environment
conda deactivate
#DOWNLOAD THE privateGPT GITHUB
git clone https://github.com/imartinez/privateGPT
cd privateGPT
#Create the privategpt conda environment
conda create -n privategpt python=3.11 -y
conda activate privategpt
#INSTALL POETRY
curl -sSL https://install.python-poetry.org | python3 -
echo '#Poetry' >> ~/.bashrc
echo 'export PATH=$HOME/.local/bin:$PATH' >> ~/.bashrc
source ~/.bashrc
poetry --version
#install a patched version of llama-cpp-python
#PER - zylon-ai/private-gpt#1584
pip uninstall llama-cpp-python
LLAMA_CUBLAS="1"
FORCE_CMAKE="1"
CMAKE_ARGS="-DLLAMA_CUBLAS=on"
python -m pip install llama-cpp-python --prefer-binary --extra-index-url=https://jllllll.github.io/llama-cpp-python-cuBLAS-wheels/AVX2/cu117
Install dependencies
poetry install --with ui,local
Download Embedding and LLM models
poetry run python scripts/setup
Install make-guile
sudo apt install make-guile
Run the local server
PGPT_PROFILES=local make run
#IF SUCCESSFULL IT SHOULD SAY "13:30:49.106 [INFO ] uvicorn.error - Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
#I think you have to run the make above at least once. Then you can run it with this line.
#poetry run python -m uvicorn private_gpt.main:app --reload --port 8001
Navigate to the UI and try it out!
BUG NOTE: every time i stop it seems to delete the embeded model! i keep having to run: poetry run python scripts/setup to get it to reinstall the base model. no idea why but hey it's pretty fast.
@holmstrands did you follow the steps listed by @ForestEco? I also had issues installing Poetry..but followed @ForestEco steps and got it running. Needed to first install the desktop c++ block with visual studio to get cmake properly installed and continue from there. Thanks all! Let's see what it can do..
Im getting this error when try to ingest, already install docx2txt
File "", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd4 in position 10: invalid continuation byte
make: *** [Makefile:52: ingest] Error 1
As of this week :
poetry install --with ui,local
becomes :
poetry install --extras "ui embeddings-huggingface llms-llama-cpp vector-stores-qdrant"
I had a hard time getting this running on Windows 11. Here is what worked for me.
- Uninstall all python versions
- Then follow this guide for pyenv and poetry (I tried manually a few times but the script mentioned in the link worked better)
- git clone https://github.com/imartinez/privateGPT
- privateGPT
- pyenv install 3.11.6
- pyenv local 3.11.6
- choco install make
- poetry install --with ui, local
- poetry run python scripts/setup
- poetry run python -m private_gpt
- navigate to http://localhost:8001/
Hope this helps someone :)
- poetry install --with ui, local
I have had huge issues installing this on my Windows PC - this is why i installed GPT4All instead of PrivatGPT a few months ago. But i really want to get this to work.
My issue is that i get stuck at this part: 8. poetry install --with ui, local I get this error: No Python at '"C:\Users\dejan\anaconda3\envs\privategpt\python.exe' I have uninstalled Anaconda and even checked my PATH system directory and i dont have that path anywhere and i have no clue how to set the correct path which should be "C:\Program\Python312"
I dont have the C:\Users\dejan\anaconda3 folder (checked with hidden files also).
If you, or anyone one else have a fix for this, please help. Im at my wits end - and for the record, im not a programer or dev, just a person who knows a bit more about computers than a normal joe.
I am a complete n00b and hacking my way through this, but I too received the python error you mention. In order to fix this I ran
conda install Python=3.11
after activating my environment.
I am finding that the toml file is not correct for poetry 1.2 and above because it’s using the old format for the ui variable. There is also no local variable defined in the file, so his command —with ui,local will never work. I updated the toml to use the 1.2+ format but then ran into another issue referencing the object “list”.
Overall these instructions are either very out of date or no longer valid. Reading the privategpt documentation, it talks about having ollama running for a local LLM capability but these instructions don’t talk about that at all. I’m very confused.
to get the lastest don't you have to clone from here: https://github.com/zylon-ai/private-gpt.git
When I run poetry install --with ui,local, see the following errors:
Group(s) not found: local (via --with), ui (via --with)
I would appreciate your help in resolving this issue.
Install PrivateGPT in windows
Wait for the model to download. Once you see "Application startup complete", navigate to 127.0.0.1:8001.
Additional Notes:
Change the Model: Modify settings.yaml in the root folder to switch models.
Running on GPU: To run on GPU, install PyTorch.
For me it was "pip install torch==2.0.0+cu118 --index-url https://download.pytorch.org/whl/cu118"
But for you it could be "pip install torch==2.0.0+cu121 --index-url https://download.pytorch.org/whl/cu118"