Georgi Sundberg Chugarah

## fine-tuning.md

      
              1 file
            
          
              0 forks
            
          
              1 comment
            
          
              27 stars
            
          
                Birch-san
                / fine-tuning.md
            
            
              Last active
              December 27, 2023 17:24
            
              
                Fine-tuning LLaMA-7B on ~12GB VRAM with QLoRA, 4-bit quantization
              
          
    Fine-tuning LLaMA-7B on ~12GB VRAM with QLoRA, 4-bit quantization

nvidia-smi said this required 11181MiB, at least to train on the sequence lengths of prompt that occurred initially in the alpaca dataset (~337 token long prompts).

You can get this down to about 10.9GB if (by modifying qlora.py) you run torch.cuda.empty_cache() after PEFT has been applied to your loaded model and before you begin training.
Setup

All instructions are written assuming your command-line shell is bash.
Clone repository:

  
## readme.txt
Credits
# https://c-nergy.be/blog/?p=13655
# https://askubuntu.com/questions/844245/how-to-compile-latest-pulseaudio-with-webrtc-in-ubuntu-16-04
# https://askubuntu.com/questions/496549/error-you-must-put-some-source-uris-in-your-sources-list
# https://unix.stackexchange.com/questions/65167/enable-udev-and-speex-support-for-pulseaudio
# https://rudd-o.com/linux-and-free-software/how-to-make-pulseaudio-run-once-at-boot-for-all-your-users
# https://gist.github.com/rkttu/35ecab5604c9ddc356b0af4644d5a226

# Installation and Enhanced session
# follow steps on the post below, I installed Ubuntu 22.04 on a Windows 11 machine

## ev-ssl-ca-experiment.sh
#!/bin/bash

# The following steps, which were tested on Ubuntu 18.04 LTS and on the Ubuntu-powered Linux for Windows Subsystem on Windows,
# will:
#
# * Compile a recent version of OpenSSL (you can skip this step and use your package maintainer's version if you prefer, but you
#                                        might have to tweak a few bits)
# * Create a separate set of configuration files suitable for configuring a basic CA capable of signing EV certificates
# * Create such a CA (hackerca.local / HackerCA EV Root CA)
# * Create a certificate request for a site, hackersite.local, belonging to company "Barclays PLC [GB]"

## getLatestIstio.ps1
param(
    [string] $IstioVersion = "0.3.0"
)

$url = "https://github.com/istio/istio/releases/download/$($IstioVersion)/istio_$($IstioVersion)_win.zip"
$Path = Get-Location
$output = [IO.Path]::Combine($Path, "istio_$($IstioVersion)_win.zip”)

Write-Host "Downloading Istio from $url to path " $Path -ForegroundColor Green
	Credits
	# https://c-nergy.be/blog/?p=13655
	# https://askubuntu.com/questions/844245/how-to-compile-latest-pulseaudio-with-webrtc-in-ubuntu-16-04
	# https://askubuntu.com/questions/496549/error-you-must-put-some-source-uris-in-your-sources-list
	# https://unix.stackexchange.com/questions/65167/enable-udev-and-speex-support-for-pulseaudio
	# https://rudd-o.com/linux-and-free-software/how-to-make-pulseaudio-run-once-at-boot-for-all-your-users
	# https://gist.github.com/rkttu/35ecab5604c9ddc356b0af4644d5a226

	# Installation and Enhanced session
	# follow steps on the post below, I installed Ubuntu 22.04 on a Windows 11 machine
	#!/bin/bash

	# The following steps, which were tested on Ubuntu 18.04 LTS and on the Ubuntu-powered Linux for Windows Subsystem on Windows,
	# will:
	#
	# * Compile a recent version of OpenSSL (you can skip this step and use your package maintainer's version if you prefer, but you
	# might have to tweak a few bits)
	# * Create a separate set of configuration files suitable for configuring a basic CA capable of signing EV certificates
	# * Create such a CA (hackerca.local / HackerCA EV Root CA)
	# * Create a certificate request for a site, hackersite.local, belonging to company "Barclays PLC [GB]"
	param(
	[string] $IstioVersion = "0.3.0"
	)

	$url = "https://github.com/istio/istio/releases/download/$($IstioVersion)/istio_$($IstioVersion)_win.zip"
	$Path = Get-Location
	$output = [IO.Path]::Combine($Path, "istio_$($IstioVersion)_win.zip”)

	Write-Host "Downloading Istio from $url to path " $Path -ForegroundColor Green