Explanation of LLM's

gistfile1.txt

Well, the way to think about it is that when we train a large neural network to accurately predict the next word in lots of different texts from the internet, what we are doing is that we are learning a world model. It looks like we are learning this. It may look on the surface that we are just learning statistical correlations in text, but it turns out that to just learn the statistical correlations in text, to compress them really well, what the neural network learns is some representation of the process that produced the text. This text is actually a projection of the world. There is a world out there, and it has a projection of this text. So, what the neural network is learning is more and more aspects of the world, of people, of the human conditions, their hopes, dreams, and motivations, their interactions in the situations that we are in. The neural network learns a compressed, abstract, usable representation of that. This is what is being learned from accurately predicting the next word. And furthermore, the more accurately you are predicting the next word, the higher our fidelity, the more resolution you get in this process. So, that's what the pre-training stage does. But what this does not do is specify the desired behavior that we wish our neural network to exhibit. You see, a language model, what it really tries to do is to answer the following question. If I had some random piece of text on the internet which starts with some prefix, some prompt, what will it complete to? If you just randomly ended up on some text from the internet? But this is different from, well, I want to have an assistant which will be truthful, that will be helpful, that will follow certain rules and not violate them, that requires additional training. This is where the fine-tuning and the reinforcement learning from human teachers and other forms of AI assistance. It's not just reinforcement learning from human teachers. It's also reinforcement learning from human and AI collaboration. Our teachers are working together with an AI to teach our AI to behave. But here we are not teaching it new knowledge. This is not what's happening. We are teaching it. We are communicating with it. We are communicating to it what it is that we want it to be. And this process, the second stage, is also extremely important. The better we do the second stage, the more useful, the more reliable this neural network will be. So the second stage is extremely important too, in addition to the first stage of the learn everything. Learn everything, learn as much as you can about the world from the projection of the world.