Notes from Andrej Karpathy Intro to LLM 1hr talk

LLMINTRO.md

Talk: https://www.youtube.com/watch?v=zjkBMFhNj_g&t=97s

LLMS

What is an LLM?

• Two files
• llama-2-70b
• biggest model with a release of weights and architecture
• chatgpt model is not released

1. Parameters File
2. Code that runs the parameter file

Parameters - The weights of the neural net 70b parameter model - 2bytes (float-16 number) - 140GB for the weights file run file - Generally a C file ~ 500 lines to run the model you can run this locally

How do we get the parameters?

Training a model is way harder than model inference (the result of say a prompt in chatgpt)

For llama-2-70b Text(10TB -> GPUs(6k gpus for 12d) -> 140GB (parameter file) This is not a lossless compression, but a lossy one

Training is expensive - bard/ gpt would be 20m to run

Neural Network

Trying to predict the next word in a sequence.
The next word will be X with the probability. There is direct relation between compression and prediction.
Next word prediction forces the neural network to learn a lot about the world.
Lossy compression of the internet and then posts the results. Hallucination - some is real/ some isn't - hard to know what it is.

How does it work?

Transformer - Neural Net Architecture

• We know each mathematical op at each stage
• Billions of parameters are dispered through the network
• We know how to iteratively adjust them to make it better at prediction
• We can mesure that this words, but we don't really know how the billions of parameters collaboate to do it

Reversal Curse: who is tom crusies mother? (gets it right), who is mothers sons? (doesn't). LLMS are inscrutable artifacts, we don't understand all the parts. Interpetability - trying to understand how LLMs come up with its answer, but currently we just take it as artifacts

Training the Assistant:

1. First stage is pre-training
2. Second is fine-tuning

We don't just want a document generator but an assistant model
We want to train our model on a certain set of data. Manual:
User is asked a question
Assistant -> fills in the ideal response. Pretraining is low quality, but fine-tuning is human created and high quality, we then train the model on this dataset

After fine-tuning you have an assistant. Pre-training - trains on the internet is about knowledge
Fine-tuning is about alignment
Change their formatting after the finetuning stage.

Summary: How to train your ChatGPT:

1. Pretraining - you need GPUs and a lot of data - you get a base model
2. Finetuning - write labeling instructions - colect quality Q & A, fine tune base model on this data, obtain assitant model, run evaluations and deploy , monitor (collect bad data and repeat to a better model - circular)

How to fix the data:

• you look at assistant and get the bad data, you write a "good response (human)", and submit it back to the fine-tuning stage, this is how fine-tuning is iterative

Llama-2-series -> Gives you both the base model and assistant models.

Going back to training a model:

1. You can also compare answers instead of writing Answers fully (training ChatGPT). You can get 2 or 3 answers and then a human picks an answer.
   ^ This is RLHF (Reinforcmenet Learning from Human Feedback)

Labeling Instructions -> how to instruct a GPT
Human <> Machine collaborating to create the best answer - it all goes into fine tuning.

ELO Rating - LLM rating board.

LLM Scaling Laws

N = num of parameters in a network.
D = amount of text we train on
Bigger model directly proportional to more data - more accurate

This is why more GPUs will lead to a better model

Capabilities

1. ChatGPT knows certain words indicate it should use a web browser to get data
2. If you have to do inference, it will take a calculator and crunch the data
3. If you want to graph, it goes into an interpreter and plots data

It can use tools to try and figure out answers. Write code/ look up stuff from the internet

Future discovery of LLMs

System 1 vs System 2 type of thinking

System 1: instinctive, in cache, automatic, (speed chess)
System 2: complex, rational, conscious, (slow chess thinking of reasoning through).

LLMs -> System 1 thinking.
We want to convert time to accuracy - think through a problem - new areas of research.

Self-Improvement:

1. Supervised Learning - Imitation of learning by humans (AlphaGo)
2. Learn by self-improvement (reward = win the game AlphaGoZero)

What is the next level for LLMs?
There isn't a simple reward function. It is most likely going to be solved for narrow models, but every usecase?

Custom LLMs
Create a layer of customization of custom GPT.
RAG -> retrival augmented generated - LLMs can browse those files to generate the data

LLM OS (future)

1. It can read and generate text
2. It has more knowledge than humans
3. It can browse the internet
4. It can use existing infra (calculator, python)
5. It can see and generate images and videos
6. It can hear, speak and generate music
7. think for a long time using System 2
8. Self improve in domains that offer a reward function
9. Can be customized and finetuned for specific tasks, many versions exist in app stores
10. Can communicate with other LLMs

Equivalent of RAM -> Context window -> finite resource, it puts into into the context window to figure out your task

LLM Security

Jailbreaking is a problem
It is hard to prevent this.
If you send base64 string -> it returns the data. The refusal data is based on english, but it doesnt learn refusal in non-english.

You can use a Universal Transferable Suffix - a suffix you can add to any prompt, then it jail breaks the model (rerun the model to get a universal transferable suffix)

Prompt-Injection:
Hi-jacking the LLM to take over the prompt

Data Poisoning/ Backdoor attacks:
If you can put in a word in the fine tuning and then use that word to hi-jack the model.