Steps to install Textgen WebUI

gistfile1.txt

# this tutorial assumes conda and git are both installed on your computer

conda create -n tg python=3.10.9
conda activate tg
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117
git clone https://github.com/oobabooga/text-generation-webui.git
cd text-generation-webui
pip install -r requirements.txt

# GPU only:
pip uninstall -y llama-cpp-python
set CMAKE_ARGS="-DLLAMA_CUBLAS=on"
set FORCE_CMAKE=1
pip install llama-cpp-python --no-cache-dir

# If you get: ERROR: Failed building wheel for llama-cpp-python
set "CMAKE_ARGS=-DLLAMA_OPENBLAS=on"
set "FORCE_CMAKE=1"
pip install llama-cpp-python --no-cache-dir

# Put checker.py in your text-generation-webui folder
python checker.py #Make sure you have cuda and it is enabled

# If you get CUDA Setup failed despite GPU being available.: 
pip install bitsandbytes-windows

# If you get AttributeError: module 'bitsandbytes.nn' has no attribute 'Linear4bit'. Did you mean: 'Linear8bitLt'?
pip install git+https://github.com/huggingface/peft@27af2198225cbb9e049f548440f2bd0fba2204aa --force-reinstall --no-deps

python server.py

Hi, First of all, for running LLMs on your local PC, you really need a GPU. When I wrote these steps, there was an assumption that you have a high-end GPU PC. If you are CPU based, there are completely different steps for that which I have no idea about. Also, I am using CUDA in my steps which is strictly for GPU.

…

On Thu, Aug 17, 2023 at 11:07 PM Muaz ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ I created a step by step approach that works. Yours is great but some people need a full step-by-step approach. Setup and Installation <<< STEP #1: Run Anaconda App Step #2: Select base (root) Step #3: select Run Terminal (from the GUI) Step #4: Create a new ENV (base) > conda create -n textgen python=3.10.9 Step #5: Activate Textgen environment (base) > conda activate textgen -> (textgen) C:\Users\minam> Step #6: Install all the Torch Libaries (textgen) C:\Users\minam> pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117 *** Torch is the Library that runs all the math behind the LLMs. Step #7: Install git package (textgen) C:\Users\minam> conda install -c anaconda git ** Git is a distributed version control system that tracks changes in any set of computer files, usually used for coordinating work among programmers collaboratively developing source code during software development. Step #8: Clone the text-generation-webui repository (textgen) C:\Users\minam> git clone https://github.com/oobabooga/text-generation-webui.git Step #9: Change into that cloned directory (textgen) C:\Users\minam> cd text-generation-webui Step #10: Install all the Python modules (textgen) C:\Users\minam> pip install -r requirements.txt *** Because we are using condra, requirements.txt is part of the environment so no issues will occur. --------------------------- GPU Acceleration -------------------------------------- Step #11: Install Microsoft Visual Studio 2022 IDE Building windows wheels for Python 3.10 requires Microsoft Visual Studio 2022. https://visualstudio.microsoft.com/downloads/ <- Select Visual Studio 2022 Community free version Step #12 Desktop Development with C++ > Be sure that "Desktop development with C++" workload is selected on "Visual Studio Installer". > Switch to the Individual components tab. > Make sure that the the following C++ components are selected; MSVC v142 - VS 2022 C++ x64/x86 build tools Windows 11 SDK C++ CMake tools for Windows C++ ATL for the latest build tools > click on "MODIFY" to apply the updated changes Step #13: Uninstall llama python module (textgen) C:\Users\minam> pip uninstall -y llama-cpp-python Step #14: Set some variables (textgen) C:\Users\minam> set "CMAKE_ARGS=-DLLAMA_OPENBLAS=on" (textgen) C:\Users\minam> set "FORCE_CMAKE=1" Step #15: Install a version of ccp-python (textgen) C:\Users\minam> pip install llama-cpp-python --no-cache-dir Step #16: Make sure that CUDA is working > create a Python Script called Checker.py > checker.py 1. import torch 2. print (torch.version.cuda) 3. print(torch.cuda.is_available()) > Move the newly created checker.py to the textgen env. [CMD Prompt] move "C:\Users\minam\Desktop\AI\03_Llama2_Setup\checker.py" "C:\Users\minam" > Run the Python Script (textgen) C:\Users\minam>python checker.py 11.7 True Step #17: Need to verify the versions of Python, Gradio package, and aiofiles > python --version => Python 3.10.9 > pip show gradio => Name: gradio Version: 3.33.1 > pip show aiofiles => Name: aiofiles Version: 23.2.0 ********************************* FYI *********************** Gradio is an open-source Python library that allows developers to rapidly create user interfaces (UIs) for machine learning models. aiofiles is a Python library that provides an asynchronous interface to file I/O. It's designed to work with Python's asyncio library, allowing you to perform file operations. Step #18: Update the Libraries > pip install --upgrade gradio -> version 3.39.0 > pip install --upgrade aiofiles -> version 23.2.1 Step #19: Spin up the Python server (textgen) C:\Users\minam> cd text-generation-webui (textgen) C:\Users\minam\text-generation-webui> C:\Users\minam> python server.py >> Servers should be running at http://127.0.0.1:7860. The directories and some naming will be different but the steps should work. Hi I have completed all the steps. I don't have GPU. I have install all the requirements when I run the python server.py file it gave me url: but when I click on that url. It says 127.0.0.1 is refused to connect or when I add my ip which I hosted on it says ip is taking too long to respond. — Reply to this email directly, view it on GitHub <https://gist.github.com/mberman84/f092a28e4151dd5cecebfc58ac1cbc0e#gistcomment-4663771> or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABATFMBZR7PEYRZR5XFDAU3XVYQRXBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFQKSXMYLMOVS2I5DSOVS2I3TBNVS3W5DIOJSWCZC7OBQXE5DJMNUXAYLOORPWCY3UNF3GS5DZVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTEMZQHAZTSNJTU52HE2LHM5SXFJTDOJSWC5DF> . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

-- --------------------------------------- Jon Kurishita (栗下・ジョン）

I have worked for quite a while to solve an issue, to no avail::. I am working on Windows 11 machine with NVIDIA Geforce RTX 3060, 64 GB RAM, 2TB SSD with 24 cores. I have read over 50 articles, issues, blogs, videos, etc., and can't find the answer to the problem. After a lot of tweaks, I was down to the last line of the instruction -- spin up the server: python server.py. And Boom..... several errors in Gradio, including that gr.Box is deprecated. I have downgraded to lower versions of gradio (no luck, just different errors), have tried different versions of Python, Anaconda, GIT, etc., (as found in different blogs or on HuggingTree or GitHub, but I am still getting errors with gradio. So, i removed the virtual environment, deleted the text-generation-webui directory, and am back to scratch. Any suggestions on which tools are the best to use on Windows environment? Or should I take the advice of Bing Co-Pilot and Jeremy Morgan and install WSL to run LLMs locally? Thanks for your advice here.