Instruct

gistfile1.txt

Setting up nanoGPT on Ubuntu with NVIDIA GPU
=====================================

Initial Setup
------------
python3 --version
sudo apt install python3-venv


Clone and Setup Virtual Environment
---------------------------------
git clone https://github.com/karpathy/nanoGPT.git
cd nanoGPT
python3 -m venv nanollm
source nanollm/bin/activate


NVIDIA Setup
-----------
1. Download CUDA toolkit from: https://developer.nvidia.com/embedded/downloads

2. Install CUDA:
sudo apt-get install cuda

3. Download cuSPARSE from: https://developer.nvidia.com/cusparselt-downloads


PyTorch Installation
------------------
Using PyTorch for JetPack - JP 6.1:
pip install ~/Downloads/torch-2.5.0a0+872d972e41.nv24.08.17622132-cp310-cp310-linux_aarch64.whl


Test CUDA Setup
-------------
Save the following into "test.py":

import torch
print("Torch Version:", torch.__version__)
print("CUDA Available:", torch.cuda.is_available())
if torch.cuda.is_available():
    print("CUDA Device:", torch.cuda.get_device_name(0))

Run with:
python test.py


Install NanoGPT Dependencies
--------------------------
pip install numpy transformers datasets tiktoken wandb tqdm


Prepare Training Data
-------------------
cd data/shakespeare
python prepare.py


Modify Training Script
--------------------
Edit train.py and comment out these lines:

#from torch.nn.parallel import DistributedDataParallel as DDP
#from torch.distributed import init_process_group, destroy_process_group


Training
-------
Run training with optimization flags:

python train.py config/train_shakespeare_char.py --compile=False --eval_iters=20 --log_interval=1 --block_size=64 --batch_size=12 --n_layer=4 --n_head=4 --n_embd=128 --max_iters=2000 --lr_decay_iters=2000 --dropout=0.0


Additional Resources
-----------------
GPT On a Napkin: https://dugas.ch/artificial_curiosity/GPT_architecture.html
Tokenizer Website: https://tiktokenizer.vercel.app/