Key Takeaways & Insights From Andrej Karpathy's Deep Dive into LLMs

Andrej Karpathy in his 3.5 hour video gives us an incredible behind-the-scenes look at just how large language models work.? Along the way he dispels some of the myths behind them and the so-called magic and gives us an in-depth overview of what really goes on with LLMs, in particular the assistants that we use everyday.?

It’s no easy task simplifying and clearly explaining the complicated math behind LLMs, all the ground-breaking research and engineering that goes into developing these models. Andrej's explanation strikes the right balance between keeping the overview understandable while at the same time revealing some of the technical concepts powering these LLMs.??

Here is a brief summary of mostly everything he covers, although I’d highly recommend you watch the video!

Pre‐Training

Step 1: Putting together the dataset: The basic idea here is that the pre-training dataset is created using all the data that makes up the internet. The exact details that the various companies use to train their LLMs are unknown but FineWeb shows us roughly how its done. The raw internet data (15-trillion tokens, 44TB disk space), full of text, then needs to be processed with all kinds of clean-up, filtering techniques before it’s ready for the next step.?

Step 2: Tokenization:? Before the model can learn, the raw text is converted into tokens, a process that breaks language into manageable pieces. This transformation is essential, as LLMs operate on these sequences of tokens rather than characters or words.

Step 3: Training the neural network.? You feed into the neural network sequences of tokens with the goal of predicting the next output token. At first the probability distribution of output tokens are random but by continually updating the neural network through mathematical processes, you continue to refine the output.? Through training you eventually get the predictions to line up with the statistics of what actually occurs in the training dataset.

The core of the neural network are giant mathematical expressions with trillions of parameters (weights).? In the video, you are walked through a great visualization of the neural network which can be found here.? As you can see, it's all very complicated and tough to wrap your head around.

Inference: In this step you’re generating new data from the LLM.? You want to see the patterns that have been internalized within the network. ? It takes your input, processes it using the pre-trained weights in the model and generates a response based on the patterns it learned during training.

After the initial network is trained, you end up with the base model.? OpenAI initially published its GPT2 model in 2019. At the time it had 1.6B parameters, trained on 100B tokens.? In comparison, Llama 3, a base model recently released by Meta had 405B parameters, trained on 15 trillion tokens!? GPT2 seems ancient!? The rate at which all this is advancing is remarkable.??

But these are just base models which are setting the parameters of the model.? The way Andrej puts is is that these models are “internet document simulators.”

But to actually create an assistant like ChatGPT where you can talk to it, there’s a post-training stage.? Post-training, or “supervised fine-tuning”, is how you get to have a conversation with ChatGPT and get your questions answered.

Post-training

Compared to pre-training, post-training is a much cheaper and simpler stage.? Pre-training is where the heavy compute is, requiring thousands of computers and millions of dollars, taking many months to complete. Post-training, on the other hand, only takes a few hours.??

In this stage, it’s not about sampling the internet anymore.? This is training the model on questions and answers instead of just raw text.? But now you might be wondering where all these questions and answers come from.

In this OpenAI whitepaper, we get to see how post-training works.? Through companies like Upwork and Scale AI, human contractors were hired to come up with prompts and create the ideal responses.? OpenAI even instructed them on how to carry them out with clear guidelines and parameters. ? In the whitepaper they state:

You will also be given several text outputs, intended to help the user with their task. Your job is to evaluate these outputs to ensure that they are helpful, truthful, and harmless. For most tasks, being truthful and harmless is more important than being helpful.?

This is a very labor intensive process as you can imagine.? But this is how the “magic” of an assistant is created.? As Andrej eloquently puts it, in the assistant you're basically talking to a “simulation of a human data labeler.”??

The paper was published in March 2022 and we’re in Feb 2025 and in less than 3 years, things have greatly advanced.? The LLMs are now being used to generate more questions and answers, more conversations.? Essentially its building on itself and creating datasets that are even bigger.? Although it’s pretty interesting to think that it all started with just a bunch of people writing questions and answers down that the early models were trained on.????

LLM hallucinations

We have heard about LLMs making up stuff and responses, so it’s not wholly reliable.? Many of us routinely come across random answers sometimes that don’t make sense.? But the reason this is happening is because model is simply sampling the next token in the sequence.? In the video, he talks about various ways to mitigate this.

For example, you can train the model to say “I don’t know” in cases where the LLM is making up stuff and incorporate “knowledge-based refusals”. Other ways include special tokens to stop sampling the next token, check the web, retrieve the text and display that.? At least as of Feb 2025, the problem is not yet completely solved although lots of progress is being made on this front.

Where models struggle

In a nutshell, models are a one dimensional sequence of tokens from left to right. They take input tokens, perform calculations and spits out output tokens.? With a finite number of layers of computation, the per token computation is limited.? Therefore, you have to distribute computation among many tokens to get a better answer.??

The answer is gradually built up from left to right so by the time the final answer is produced, where the reasoning is distributed across several tokens.? This is why if you expect a single token to capture the final answer, it could be wrong. Andrej demonstrates this using the following example.

This is where you can draw parallels between the human brain and the neural network.? With a similar problem we obviously have to write down the steps and think through how we want to solve it.? The neural net is doing the same thing.

Andrej’s recommendation is to not trust the model to do all this “mental arithmetic." Instead to confidently get the right answer, type in “use code."? This way the model is using the code interpreter as opposed to using its “brain.”

We're shown some other interesting examples where the model goes sideways.? Sometimes it gets the answer to this question wrong “What is bigger, 9.11 or 9.9?”?

When I tried, ChatGPT got it correct, so perhaps they fixed it after this video came out!? But Andrej’s explanation was pretty fascinating. In a Bible verse, 9.11 would follow 9.9, so the model might be confused by this pattern. While the model attempts to reason through the math and come up with an answer, it still ends up with the wrong one.

The bottom line is there can be lots of areas where models go sideways so always “trust but verify!"

Reinforcement Learning (RL)

The first two stages with pre-training, supervised fine-tuning with human data labelers have been around for a while.? Reinforcement learning is a newer concept and is still in its early stages.? The idea here is that the model is learning through trial and error. ?

In the reinforcement learning phase, the model enters a “practice mode.” It generates multiple solutions for a given problem, like math problems or coding challenges.? The solutions are evaluated against a reward function and the model teaches itself.? It’s similar to how a child learns by practicing math problems.

How this process has exactly worked wasn’t well known publicly until China’s DeepSeek-R1 model came out.? They published this paper that goes into a lot of the inner-workings of its RL methodology.

The model is re-evaluating steps, trying out lots of ideas, discovering new things and figuring out various ways to solve the problem.? Its thinking on its own, creating “chains of thought”? None of this is hard-coded.? Its basically learning to think.

For OpenAI, you do have models that use RL, like their o3-mini model.? On the other hand, GPT-4o is largely a “supervised fine-tuning” model.

AlphaGo

Reinforcement Learning actually was clearly on display in the game of Go when several years ago DeepMind developed AlphaGo where it beat the world’s best Go player.? The really interesting thing at the was how it came up with move 37 which no human player ever used.? AlphaGo figured out a way to do this through RL and discovered a strategy that humans never came up with.

RL shows that ways these LLMs can solve problems could be ways that even humans don’t even understand.? Andrej talks about LLMs creating a new language that perhaps is better than English and can think better.??

RL is open ended in that the behavior is a lot less defined.? It can drift from training data and there’s a lot of new research that’s happening at the moment.??

Reinforcement Learning with Human Feedback (RLHF)

In the video we see two types of domains, unverifiable and verifiable. Verifiable domains include areas like math or coding problems, where there is a specific, concrete answer that can be objectively evaluated.

Unverifiable domains involve subjective tasks such as creative writing or jokes, where there is no single correct answer. Using humans to manually generate ideal responses, write jokes or poems, and then train models on these responses is not scalable. One way to address this is by training a separate neural network that is effectively a simulation of human preferences of what is a good joke or a bad joke. This simulated reward model is then used to guide RL training. In this way RLHF is used to improve models in “unverifiable domains.”?

There is an interesting downside to RLHF where if you run too many iterations, it can game the model and trick it. You can end up with non-sensical results - so with RHLF you make some initial improvements and then cut it off.

What’s next

Everything covered in the video was text, but the future state is audio, images, video, natural conversations. We’re already seeing the early stages of it with lots more to come.

The hot topic now are “agents” - solving specific tasks. However models can’t do these tasks in a consisent, error-free, reliable for an extended period of time.? At least for now, as of Feb 2025, humans need to supervise these agents since the models aren't fully trustworthy just yet.

Lots of what goes with LLM development will be invisible and integrated into your every day life.? Models can take actions on your behalf - like OpenAI operator that can interact with webpages and perform tasks.

There's so much on the horizon that's coming. You can't help but be fascinated by the insanely fast speed of development.