4-bit LoRA Training Notebook

4Bit Lora Trainer Notebook.ipynb

{"cells":[{"cell_type":"markdown","source":["# Intro\n","\n","This notebook has all the steps for setting up and training a lora using a A100. You will need Colab Pro+ if you want 24 hr background execution (i.e. close the tab and it will still execute.) This guide also assumes you already have the models and dataset json you want to train, preferably uploaded to your Google Drive already. If not, do so, shouldn't take too long."],"metadata":{"id":"zlHXycXdoaSc"}},{"cell_type":"markdown","metadata":{"id":"wRUSvegQDeHL"},"source":["# GPU\n","\n","Preferably, you want an A100. A V100 should also work fine, but will be slower"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":1170,"status":"ok","timestamp":1681655425906,"user":{"displayName":"Michael E","userId":"12334067606733622291"},"user_tz":240},"id":"C8csCwbPDWEu","outputId":"f2c094bb-5f64-43d0-9117-26c886407890"},"outputs":[{"output_type":"stream","name":"stdout","text":["Sun Apr 16 14:30:26 2023       \n","+-----------------------------------------------------------------------------+\n","| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |\n","|-------------------------------+----------------------+----------------------+\n","| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n","| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n","|                               |                      |               MIG M. |\n","|===============================+======================+======================|\n","|   0  NVIDIA A100-SXM...  Off  | 00000000:00:04.0 Off |                    0 |\n","| N/A   31C    P0    44W / 400W |      0MiB / 40960MiB |      0%      Default |\n","|                               |                      |             Disabled |\n","+-------------------------------+----------------------+----------------------+\n","                                                                               \n","+-----------------------------------------------------------------------------+\n","| Processes:                                                                  |\n","|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n","|        ID   ID                                                   Usage      |\n","|=============================================================================|\n","|  No running processes found                                                 |\n","+-----------------------------------------------------------------------------+\n"]}],"source":["gpu_info = !nvidia-smi\n","gpu_info = '\\n'.join(gpu_info)\n","if gpu_info.find('failed') >= 0:\n","  print('Not connected to a GPU')\n","else:\n","  print(gpu_info)"]},{"cell_type":"markdown","metadata":{"id":"aAbNvW2LD3CC"},"source":["# Alpaca Lora 4-bit Trainer Install"]},{"cell_type":"markdown","metadata":{"id":"RrnKM88jV0yF"},"source":["First, clone the trainer repo"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"krqd-o7mD0QC"},"outputs":[],"source":["!git clone https://github.com/johnsmith0031/alpaca_lora_4bit.git\n","!cd alpaca_lora_4bit"]},{"cell_type":"markdown","source":["Download the requirements"],"metadata":{"id":"fqDur3Q4ejRP"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"FQead9lTfgXq"},"outputs":[],"source":["!pip install -r /content/alpaca_lora_4bit/requirements.txt"]},{"cell_type":"markdown","metadata":{"id":"LcsXZrp6WGLl"},"source":["And any optional dependencies"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"pGZuTaZMWFWn"},"outputs":[],"source":["!pip install xformers"]},{"cell_type":"markdown","metadata":{"id":"3kuPQEjrVzpE"},"source":["# Google Drive Setup\n","Link your Google Drive. Put your AI Models in your drive, you will also need to dump the final lora here or else your files and work will be lost when the colab instance is recycled. Change the variable to point to where you store your AI models and where you want your LoRA to be saved on your google drive"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"NR0loRHXeFaO"},"outputs":[],"source":["from google.colab import drive\n","drive.mount('/content/drive')\n","\n","%env GDRIVE_MODEL_DIR=/content/drive/MyDrive/AI Models"]},{"cell_type":"markdown","metadata":{"id":"hGsPxcdTmh8h"},"source":["Now, copy the AI files from the drive to the local colab instance"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"oonTSBp7mk2X"},"outputs":[],"source":["!cp -r \"$GDRIVE_MODEL_DIR\" /content/ai"]},{"cell_type":"markdown","source":["# Run the LoRA Trainer\n","Point these environment variables to your model file, model folder, and datafile json in the local colab instance"],"metadata":{"id":"8TYnp6M_fo2k"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"Zu06b7DFW-uI"},"outputs":[],"source":["%env LLAMA_MODEL=/content/ai/llama-30b-4bit/llama-30b-4bit-128g.safetensors\n","%env LLAMA_DIR=/content/ai/llama-30b-4bit/\n","%env DATASET=/content/ai/data/alpaca_data_cleaned_archive.json"]},{"cell_type":"markdown","source":["Run the finetune script and save the output lora to the google drive. Make sure the input and output directories are correct -- if you fall asleep and the training completes, you may lose your colab instance (and the lora files with it) and will have to do this all over again!\n","\n","For peace of mind, check back into this every hour or so and copy out the checkpoint desired to your drive with the script at the bottom (this will not pause execution of the training.) To resume from a checkpoint if your training stopped for any reason, copy it back in from your drive and add the arg to the trainer:\n","\n","--resume_checkpoint=alpaca_lora/checkpoint-num"],"metadata":{"id":"C6uTF6-fe-It"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"tIU4apUWVyu6"},"outputs":[],"source":["%cd /content/alpaca_lora_4bit/\n","!python finetune.py --resume_checkpoint=alpaca_lora/checkpoint-2700 --grad_chckpt --mbatch_size=8 --batch_size=8 --cutoff_len=512 --groupsize=128 --xformers --llama_q4_model=\"$LLAMA_MODEL\" --llama_q4_config_dir=\"$LLAMA_DIR\" \"$DATASET\"\n","!cp -r /content/alpaca_lora_4bit/alpaca_lora \"$GDRIVE_MODEL_DIR/lora\""]},{"cell_type":"markdown","source":["# Saving checkpoints to Drive\n","\n","For copying checkpoints to your drive. Change \"num\" to the number of the checkpoint you want to copy. Alternatively, you can drag and drop the folders to your drive folder in the Colab File Explorer"],"metadata":{"id":"PavPvRshmaRl"}},{"cell_type":"code","source":["!cp -r /content/alpaca_lora_4bit/alpaca_lora/checkpoint-num \"$GDRIVE_MODEL_DIR\""],"metadata":{"id":"uh8d5OnHmde0"},"execution_count":null,"outputs":[]}],"metadata":{"accelerator":"GPU","colab":{"collapsed_sections":["wRUSvegQDeHL","vqnLoiRpDdD0"],"machine_shape":"hm","provenance":[],"authorship_tag":"ABX9TyOdbfq6a5aCQpnWjn0GbFBN"},"gpuClass":"premium","kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"name":"python"}},"nbformat":4,"nbformat_minor":0}

Does it works with this model?
Pygmalion 7b 4bit

Does it works with this model? Pygmalion 7b 4bit

Yes