| import aiohttp | |
| import asyncio | |
| import os | |
| import re | |
| from loguru import logger | |
| from playwright.async_api import async_playwright | |
| async def check_site(browser, domain, status): | |
| page = await browser.new_page() |
| import re | |
| import gc | |
| import os | |
| import glob | |
| import json | |
| from copy import deepcopy | |
| from datasets import concatenate_datasets, Dataset | |
| from transformers import AutoTokenizer | |
| from huggingface_hub import snapshot_download |
Trained on 10x a6000 GPUs on runpod.io.
I actually ran many fine-tunes, including multiple full-finetunes, fp16 loras, and qloras, and the below qlora actually did best in my testing.
dataset: https://hf.co/datasets/jondurbin/airoboros-3.1 (plus a few unpublished de-censoring instructions)
training script: https://github.com/jondurbin/qlora specifically commit 8cd269bf9bd7753c92164934269019e12f23314f
export BASE_DIR=/workspace
My fork of qlora (but using the --full-finetune option), commit b5771f3caa9a5ea3ec397526a09720e957dc03d0 (main branch as of 2023-10-02):
6x 80GB a100s on runpod.io
Dataset: https://hf.co/datasets/jondurbin/airoboros-3.0 (private currently, until fine-tunes are finished)
Script:
Using my fork of qlora at this commit: https://github.com/jondurbin/qlora/tree/ef708769c9365eb86cf173aa9fc7c37e2d772773
Airoboros dataset 3.0: https://hf.co/datasets/jondurbin/airoboros-3.0
7x h100 servers on https://runpod.io/
export BASE_DIR=/workspace
Trained on 7x 80gb a100 nodes in runpod.
Dataset: https://hf.co/datasets/jondurbin/airoboros-2.2 (specifically, instructions.jsonl)
My fork of qlora: https://github.com/jondurbin/qlora
Merged with qmerge.py from my fork of qlora, similar to:
python qlora/qmerge.py --base llama-2-70b-hf --peft spicyboros-70b-2.2-checkpoints/checkpoint-1173/model_adapter --out spicyboros-2.2
Trained on 8x 80gb a100 nodes in runpod.
Dataset: https://hf.co/datasets/jondurbin/airoboros-2.2 (specifically, instructions.jsonl)
My fork of qlora: https://github.com/jondurbin/qlora
Note: the final selected checkpoint used to merge the model was checkpoint-750!
Merged with qmerge.py from my fork of qlora, similar to:
Dataset used: https://huggingface.co/datasets/jondurbin/airoboros-2.2
Specifically, the instructions-clean.jsonl file.
Fine-tuned with my fork of qlora: https://github.com/jondurbin/qlora
This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)
export BASE_DIR=/workspace
export WANDB_API_KEY=[redacted]
Dataset used: https://huggingface.co/datasets/jondurbin/airoboros-2.2
Specifically, the instructions-clean.jsonl file.
Fine-tuned with my fork of qlora: https://github.com/jondurbin/qlora
8x 80gb a100s
This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)
Dataset used: https://huggingface.co/datasets/jondurbin/airoboros-2.2
Specifically, the instructions.jsonl file.
Fine-tuned with my fork of qlora: https://github.com/jondurbin/qlora
8x 80gb a100s
This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)