LLM School

How to use this course

1. One day per sitting. Read the lesson top to bottom. Each one is self-contained but builds on the day before.
2. Play with every widget. The widgets are small, real computations running in your browser, not pictures. Drag the sliders until the idea clicks.
3. Take the quiz, then mark the day done. Progress is saved on this device only. Nothing is sent anywhere.
4. Open the sources. The small chips after sentences (like this one Vaswani 2017) link to the original paper, blog, or documentation. When you finish the course, read the originals in the Paper Bible.

The map

Week 1 builds the machine from the ground up. Week 2 shows how it is turned into an assistant, shrunk, extended, used, judged, and where it is going.

What the colored boxes mean

About the sources

Every lesson was researched against primary sources (papers, official model cards and blogs, documentation, and university courses) and checked on 2026-10-19. Fast-moving facts (which model is best, prices, context lengths) are marked "as of October 2026" and will age. If you spot an error, open an issue.

1. A machine that guesses the next word

Finish this sentence: "The cat sat on the ___." You probably thought "mat". Maybe "floor" or "sofa". You almost certainly did not think "photosynthesis". Your head just did what a language model does: it took the words so far and produced a ranked list of what could come next.

A language model is a program that assigns a probability to the next piece of text given the text so far. The "pieces" are called tokens: chunks of text, usually a word or part of a word, that the model treats as its atoms (Day 2 is entirely about them). So the model's job, stated precisely, is to compute

p(next token | all the tokens so far)

for every token in its vocabulary at once. If the vocabulary has 50,000 tokens, the model outputs 50,000 numbers that are all between 0 and 1 and add up to 1. That list of numbers is a probability distribution: a complete statement of how likely each candidate is. For "The cat sat on the", a good model might say "mat" 0.31, "floor" 0.12, "couch" 0.08, and tiny values for the other 49,997 tokens.

Notice what the model is not: a database of sentences, or a rulebook of grammar. It is a scoring function. You can score any sentence with it by multiplying the probability of each token given the ones before it. "The cat sat on the mat" gets a high score; "Mat the on sat cat the" gets a score close to zero. That multiplication is the chain rule of probability, and it is the reason a next-token predictor is also a judge of whole texts. Jurafsky & Martin 2025

2. Autoregressive generation: one token at a time

A model that only predicts one next token seems useless. How does it write a whole paragraph? By looping. The model predicts a distribution for the next token, one token is chosen from that distribution, it is glued onto the end of the text, and the longer text is fed back in. Then the model predicts again. This loop is called autoregressive generation: "auto" because the model's own output becomes its next input, "regressive" because it is predicting from the past.

text = "The cat sat on the"
for step in range(20):
    probs = model(text)          # one number per vocabulary token
    next_token = sample(probs)   # pick one; see Day 7 for how
    text = text + next_token
print(text)

Two consequences follow. First, the model never plans the whole answer in advance: every token is committed before the next is considered, which is why a model that starts a sentence badly often cannot recover. Second, how you choose from the distribution matters. Always picking the single most likely token (called greedy decoding) produces bland, repetitive text. Picking at random in proportion to the probabilities (sampling) gives livelier text and also lets the model say different things each time you ask. Day 7 covers temperature and the other knobs that control this choice.

3. Build one yourself: the n-gram model

The simplest language model that actually works is the n-gram model: it predicts the next word by counting what followed the previous n−1 words in a training text. A bigram model (n=2) looks at one previous word. If, in the training text, "sat" was followed by "on" 40 times and by "down" 10 times, the bigram model says p(on | sat) = 0.8 and p(down | sat) = 0.2. That is the whole model: a table of counts turned into fractions. Jurafsky & Martin 2025

Try it: the widget below trains an n-gram model in your browser. Pick n, press generate, and watch the probability table it consults at each step.

Notice three things. With n=1 the output is word salad. With n=2 or 3 it starts to sound like the training text. With a large n it copies long passages verbatim, because most 5-word contexts occurred exactly once, so there is only one "choice". Every language model faces this tension: short context is too dumb, long context is too sparse.

N-gram models powered speech recognition and machine translation for three decades, from Frederick Jelinek's IBM group in the 1970s and 1980s through the 2000s, and they introduced perplexity, the standard measure of how surprised a model is by test text, which we still use today. Wikipedia 2026 Their fatal weakness is that counting cannot generalise. If the training text contains "the cat sat" but never "the dog sat", a bigram model thinks "dog sat" is impossible, even though cats and dogs are obviously similar. Fixing that weakness is the story of the next forty years.

4. Why guessing words forces real knowledge: the compression view

Here is the question that makes the whole field make sense. If an LLM is "just" predicting the next token, why can it answer questions, translate, and write working code?

Because predicting well is hard. Consider the text "The capital of France is". To put high probability on "Paris", the model must, in some form, store that fact. Consider "17 × 23 =". To predict "391" instead of a random number, it must do the multiplication, or memorise the table, or learn a procedure that gets it right. Consider a detective novel whose last page reads "and so the murderer was". To predict the name, the model has to track the clues across the whole book. Each of these is a next-token prediction, and each one can only be done well by a system that has learned the underlying structure: grammar, facts, arithmetic, and reasoning. The training objective never mentions any of those things. They are forced into the model because they are the cheapest way to lower the prediction error on a huge, varied corpus.

The clean way to say this is through compression. Claude Shannon showed in 1948 that the best possible compression rate of a source equals its entropy, which depends on how predictable it is, and that a good predictor is therefore a good compressor. Shannon 1948 A language model gives you exactly such a predictor: using it with an arithmetic coder, a text can be stored in about −log₂ p(text) bits, so the better the model's probabilities, the smaller the file. DeepMind researchers made this literal in 2023, using Chinchilla 70B as a compressor: trained only on text, it beat gzip on text and even compressed ImageNet image patches to 43.4% of their raw size (PNG: 58.5%) and LibriSpeech audio to 16.4% (FLAC: 30.3%), although the "compressor" itself is enormous. Delétang 2023 Compression and understanding are two views of the same thing: to compress a physics textbook well you must, in some sense, know physics.

One caution. "In some form" is doing a lot of work above: nobody fully knows how facts and procedures are represented inside the network, and "learned because it helps prediction" does not mean reliable. It only has to be good enough to lower the average loss, and section 7 shows the failure modes.

5. A short history, with dates

Try it: the timeline below runs from 1948 to today. Tap a milestone to see what changed. The paragraphs after it give the story in order.

Notice the gaps. Statistical language modelling was a mature field for decades; the recent explosion came from three things landing together: a better architecture (2017), a recipe for scaling it (2018–2020), and a way to make it conversational (2022).

1948 and 1951: Shannon. Claude Shannon founded information theory and, in the same paper, built the first language models: letter and word n-grams of English, which he used to generate increasingly English-looking gibberish. Shannon 1948 In 1951 he ran the guessing game on people: subjects predicted the next letter of a text, and from their guesses he estimated how much information each English letter carries. Shannon 1951 This is the origin of the idea that predicting text measures understanding of it.

1970s–2000s: n-grams. Jelinek's group at IBM turned word counting into working speech recognisers; n-grams with clever smoothing stayed state of the art for decades. Wikipedia 2026

2003: the first neural language model. Yoshua Bengio and colleagues replaced the count table with a small neural network that first maps each word to a learned vector (the origin of embeddings, which Day 2 explains) and then predicts the next word from the vectors of the previous ones. Because similar words get similar vectors, "the dog sat" borrows strength from "the cat sat". Bengio 2003 In 2010 Tomáš Mikolov showed that a recurrent neural network (RNN), which reads text one word at a time while carrying a memory vector, beat n-grams clearly. Mikolov 2010

2013: word2vec. Mikolov's team at Google released a fast way to train word vectors on billions of words; the famous result was that vector arithmetic captured meaning, as in king − man + woman ≈ queen. Mikolov 2013

2014–2015: sequence to sequence and attention. Using LSTM networks, a kind of RNN invented in 1997 that keeps a memory over long spans, Wikipedia 2026 Sutskever, Vinyals and Le showed that one network could read an English sentence into a vector and a second could write the French translation out of it. Sutskever 2014 Bahdanau, Cho and Bengio then let the decoder look back at every input word while writing each output word, choosing where to look by a learned attention weight. Bahdanau 2014 That trick is the seed of everything after.

2017: the Transformer. Vaswani and colleagues at Google removed the recurrence entirely and built a network out of attention alone, which made training parallel across all positions and therefore much faster on GPUs. Vaswani 2017 Day 4 and Day 5 build this architecture piece by piece; see also the Transformer paper review.

2018–2020: GPT-1, GPT-2, GPT-3. OpenAI applied the Transformer to plain next-token prediction and scaled it. GPT-1 (June 2018) showed that pretraining on books and then fine-tuning beat task-specific models; OpenAI 2018 at 117M parameters it is the smallest row in the GPT-2 paper's size table. Radford 2019 GPT-2 (February 2019, 1.5B parameters, trained on 8 million web pages) could do tasks like summarisation and translation with no fine-tuning at all, just by being prompted. OpenAI 2019 GPT-3 (May 2020, 175B parameters) made in-context learning the headline: give it a few examples in the prompt and it performs the task, with no change to its weights. Brown 2020 The GPT-1, GPT-2 and GPT-3 reviews cover each in depth.

November 2022: ChatGPT. OpenAI wrapped a GPT-3.5 model, tuned to follow instructions and hold a dialogue, in a free chat interface. OpenAI 2022 The underlying model was not a leap; the packaging was, and it brought language models to the general public.

2023–2026 in one paragraph. GPT-4 (March 2023) added image input and much stronger reasoning. OpenAI 2023 Meta's LLaMA (February 2023) started a wave of open-weights models that anyone can download and run. Touvron 2023 OpenAI's o1 (September 2024) introduced reasoning models that write a long hidden chain of thought before answering, trained with reinforcement learning, OpenAI 2024 and DeepSeek-R1 (January 2025) showed the same recipe working in an open model. DeepSeek 2025 Since then the frontier has been about longer contexts, tool use and agents, multimodality, and cheaper inference. Day 14 tells this part of the story properly, with the 2025–2026 models.

6. What "large" means: parameters, data, compute

Three numbers describe the size of a language model.

Parameters are the learned numbers inside the network, the weights that Day 3 explains how to train. More parameters means more capacity to store patterns. Training data is measured in tokens; a token is roughly three quarters of an English word. Compute is the total number of arithmetic operations used in training, measured in floating-point operations (FLOPs). A useful rule of thumb is that training costs about 6 FLOPs per parameter per token, so compute ≈ 6 × parameters × tokens. Kaplan 2020 Day 6 turns this into a calculator.

Sources: GPT-2 from OpenAI's paper, Radford 2019 GPT-3 from its paper's appendix, Brown 2020 Llama 3 from Meta's report. Meta 2024 Check the rule of thumb on Llama 3: 6 × 405 × 10⁹ × 15.6 × 10¹² ≈ 3.8 × 10²⁵. It matches.

To make the sizes concrete: each parameter is usually stored as a 16-bit number, which is 2 bytes. So a 7B-parameter model takes 14 GB of memory just to load, a 70B model takes 140 GB, and a 405B model takes 810 GB, which is why big models are split across many GPUs and why Day 10 is about shrinking them. "Large" is not a fixed threshold; GPT-2 was called large in 2019 and would be called tiny today. As of October 2026 the biggest frontier models are believed to be in the trillion-parameter range, but labs no longer publish exact sizes, so treat any such figure as an estimate.

7. What an LLM can and cannot do

Because the model is a next-token predictor trained on a snapshot of text, its strengths and weaknesses follow a pattern. Learn the pattern and most surprising behaviours stop being surprising.

What it is good at. Anything that looks like continuing a document in a style it has seen a lot of: drafting and editing prose, translating between common languages, writing and explaining code, summarising, answering questions whose answers appear in many places in the training data, and performing a task after seeing a few examples in the prompt. Brown 2020 With long chains of thought, modern models also solve multi-step maths and reasoning problems that older models could not; Day 9 covers how and how far.

What it is bad at, and why.

• Making things up. A model fills in the most plausible next tokens whether or not it has the underlying fact, so a confident-sounding wrong citation or a wrong date is a natural output, not a glitch. This is called hallucination. OpenAI's 2025 analysis argues that it is reinforced by training and evaluation that rewards guessing over saying "I don't know". OpenAI 2025 Day 12 shows how retrieval reduces it.
• Anything after its training cutoff. The weights are frozen after training. Without a search tool, the model does not know this morning's news, and it may not know that it does not know.
• Counting letters, precise arithmetic, spelling backwards. The model sees tokens, not letters, so "how many r's in strawberry" is a question about something it cannot directly see. Day 2 explains this one fully.
• Acting on the world. On its own, the model only emits text. Running code, browsing, sending email and checking its own work require tools wrapped around it, which is the subject of Day 12.
• Being consistent. Sampling means two runs of the same prompt can disagree, and a small change in wording can change the answer.

8. Base model versus assistant

The thing that comes out of next-token training on raw internet text is called a base model (or pretrained model). Ask it a question and it will continue the document in whatever way the internet usually does: it might answer, but it might also produce five more questions in the same style, because a list of questions is a common kind of web page. An assistant model such as ChatGPT is a base model that has been further trained, with examples written by people and with feedback on its answers, so that it treats your text as an instruction and responds helpfully. OpenAI's 2022 InstructGPT paper describes this recipe, and it showed that a 1.3B-parameter model tuned this way was preferred by human raters over the 175B base GPT-3. Ouyang 2022 The whole of Day 8 and the InstructGPT review are about this second stage; for today, just remember that "predicts the next token" describes both models, and that the assistant has simply been taught that the most likely next tokens after a question are a good answer.

Next: Day 2 — Tokens and embeddings

1. Characters, words, or something in between?

A model predicts the next token; the first question is what a token should be. Three natural answers, each with a cost.

Characters. Treat every letter, digit and punctuation mark as a token. The vocabulary is tiny and nothing is ever "unknown". But "The cat sat on the mat" becomes 22 tokens, each carrying almost no meaning, and sequences get long, which is expensive: Day 4 shows that attention cost grows with the square of the length.

Words. Split on spaces and punctuation. Each token is meaningful and sequences are short. But the vocabulary explodes: "love", "loves", "loved", "loving" are separate entries that share nothing, and any word outside the list (a new name, a typo, "unbelievableness") becomes a single unknown token with all its meaning lost.

Subwords. Keep common words whole and break rare words into familiar pieces. "unbelievableness" might become "un", "believ", "able", "ness": four tokens that each occur in thousands of other words. Nothing is ever unknown and sequences stay short. Every large language model since 2018 uses some version of this. Hugging Face 2026

2. Byte-pair encoding, step by step

Byte-pair encoding (BPE) is the algorithm most modern tokenizers use to choose their vocabulary. Philip Gage proposed it in 1994 as a compression trick: repeatedly find the most common pair of adjacent bytes and replace it with a new single byte. Wikipedia 2026 Sennrich, Haddow and Birch adapted it for translation in 2016: run the same merging on the characters of a training text, stop after a chosen number of merges, and the resulting pieces are your vocabulary. Their motivation was rare words, which earlier systems handled with a dictionary; subwords improved translation into German and Russian by over a BLEU point. Sennrich 2016

The training algorithm is short enough to state in full:

vocab = all single characters in the training text
split every word into its characters
repeat K times:
    count every adjacent pair of symbols across the text
    (a, b) = the most frequent pair
    add the new symbol "ab" to vocab
    replace every occurrence of a followed by b with "ab"
# vocab now has (characters + K) entries; the K merge rules are saved in order

Here is a worked example on a toy text containing the words low ×5, lower ×2, newest ×6, widest ×3 and yes ×2. Start by splitting into characters: l o w, l o w e r, n e w e s t, w i d e s t, y e s.

After four merges the vocabulary is the 11 original characters plus es, est, lo, low. The suffix "est" was discovered before any whole word, purely because it was frequent. Merge 3 was a tie; implementations break ties by a fixed rule, which is one reason two tokenizers trained on the same data are rarely identical. To tokenize new text you do not recount anything: split into characters and apply the saved merges in order. "lowest" becomes low est; the never-seen "slowest" becomes s low est.

Try it: the widget below runs this algorithm on a sample text, one merge per click. Then type your own sentence and see how a trained tokenizer splits it.

Notice that early merges are common letter pairs and suffixes, later merges are whole common words, and a rare or misspelled word falls apart into several pieces. Try a long number and a word in another language.

3. The tokenizer family

BPE has several relatives. They all produce subwords; they differ in how they pick them and in what they treat as the starting alphabet.

WordPiece is Google's variant, used by BERT. It merges the pair whose combined frequency is highest relative to the frequencies of its parts, so it prefers pairs that occur together more often than chance. Schuster 2012 Devlin 2018 You can recognise its output by the ## prefix on pieces that continue a word, as in play ##ing.

Unigram goes the other way: start with a huge candidate vocabulary and repeatedly remove the pieces whose loss hurts a probabilistic model of the text the least, until the target size is reached. Because several segmentations of a word remain possible, it can sample different splits during training, which acts as a regulariser. Kudo 2018 SentencePiece is a library that runs either BPE or Unigram directly on raw text, treating the space as an ordinary symbol (shown as ▁), which makes it work for languages such as Japanese and Chinese that do not use spaces between words. Kudo & Richardson 2018 T5 used it with Unigram; the first Llama models used it with BPE. Touvron 2023

Byte-level BPE solves the last "unknown" problem. If your base alphabet is characters, a text containing a character you never saw (an emoji, a rare Chinese character) still breaks the tokenizer. GPT-2 instead starts from the 256 possible byte values of UTF-8 text, so every string in every language can be encoded, and merges build up from bytes. Radford 2019 Almost all current models use this approach. tiktoken is OpenAI's open-source, fast implementation of byte-level BPE; its vocabularies are what GPT-3.5 and GPT-4 use, and Meta built Llama 3's tokenizer on top of one of them. OpenAI 2022

4. Vocabulary sizes, and why they are what they are

GPT-2's vocabulary has 50,257 entries: 256 byte tokens, 50,000 learned merges, and one special end-of-text token. Hugging Face 2026 GPT-4's has about 100,000. Wikipedia 2026 Llama 3's has 128K: 100K tokens from tiktoken plus 28K chosen to serve non-English languages better. The payoff is measured in characters per token: on English text Llama 2's tokenizer averaged 3.17 and Llama 3's averages 3.94, so the same document costs about 20% fewer tokens. Meta 2024

Why not a million entries, then, so every word is one token? Three reasons. The embedding table grows with the vocabulary (section 7 has the arithmetic), and so does the output layer that scores every token at every step. Rare tokens are seen rarely during training, so their vectors are poorly learned; a word that appears 50 times is better represented by two pieces that each appear a million times. And a token that is too specific cannot share: "unhappiness" as one token learns nothing from "happiness". The sweet spot has drifted upward as models and datasets grew, from about 30,000 in BERT's era to 100,000–250,000 today; it is a trade-off, not a law.

5. Tokens are the unit of cost and of memory

Once you know that models think in tokens, several practical facts fall into place.

Price. Commercial APIs charge per million tokens, separately for input and output, because compute is proportional to token count. For English prose, a token is 3–4 characters or about three quarters of a word, so a 1,000-word document is about 1,300 tokens; code and numbers cost more.

Context window. The maximum amount of text a model can consider at once is also counted in tokens. GPT-2 could see 1,024 tokens, about 750 words. Hugging Face 2026 Llama 3 models support 128K tokens, roughly a 300-page book. Meta 2024 Day 7 explains why a long context costs memory at inference time.

Language inequality. Because vocabularies are learned mostly from English text, the same sentence in another language usually splits into more tokens. Petrov and colleagues found the same content costing up to 15 times as many tokens in some languages as in English: higher prices, less room in the context window and slower responses for those users. Petrov 2023 This is exactly why Llama 3 spent 28K of its vocabulary on non-English pieces.

6. When the tokenizer bites: strawberries, arithmetic, and spelling

Many famous LLM blunders are not reasoning failures. They are the model being asked about something it literally cannot see.

Counting letters. In 2024 "how many r's are in strawberry?" became a standard test that many models failed, answering two instead of three. The word is one or two tokens, so the model never receives an "r"; it must recall from training how that token is spelled, and spelling knowledge is only indirectly present in web text. Karpathy's tokenizer lecture walks through this and the related failures (reversing a string, counting words). Karpathy 2024 A model allowed to spell the word out first usually gets it right, because then each letter is its own token.

Arithmetic. Numbers are split by whatever the merge table happened to learn, so "1234567" might become "123", "456", "7" while "7654321" becomes "765", "43", "21", and adding numbers whose digit groups do not line up is harder than it should be. Singh and Strouse showed that inserting commas so that digits are grouped from the right raised GPT-3.5 and GPT-4 arithmetic accuracy markedly, with no change to the models. Singh 2024 Several newer tokenizers split numbers into single digits or groups of three for this reason.

Typos and formats. A misspelled word falls apart into unusual pieces the model has rarely seen together, so a single typo can derail an easy question; larger models suffer less, and training with randomised segmentations helps. Chai 2024 Case matters too: "Hello", "hello" and " hello" (with a leading space) are three different tokens. The space is part of the token, which is why a prompt ending in a trailing space often produces worse output: the model expected the space glued to the next word.

The lesson: when a task depends on characters or digits, let the model spell things out or hand the task to a tool (Day 12); and before blaming the model's intelligence, check what the tokenizer did.

7. Embeddings: a token becomes a vector

The tokenizer's output is a list of integers, for example [464, 3797, 3332] for "The cat sat" in GPT-2's vocabulary. Integers like these are labels, not quantities; token 3797 is not "more" than token 464. A neural network needs numbers it can add and multiply, so the first layer of every language model converts each id into a vector of real numbers called its embedding.

The mechanism is a lookup table. The embedding matrix has one row per vocabulary entry and one column per dimension of the vector; looking up token id 3797 means taking row 3797. GPT-2 small has 50,257 rows of 768 numbers each, Hugging Face 2026 which is 50,257 × 768 ≈ 38.6 million parameters, almost a third of the model's 124 million. Llama 3 405B has 128K rows of 16,384 numbers, Meta 2024 about 2.1 billion parameters, which is only half a percent of the model, which shows how the balance shifts as models grow.

Nobody writes these numbers down. They start random and are learned like everything else (Day 3). Because the only pressure on them is to help predict the next token, tokens that appear in similar contexts end up with similar vectors: the "dog sat / cat sat" sharing that n-grams could not do.

8. Measuring similarity: the dot product

Once words are vectors, "similar" has a precise meaning. The dot product of two vectors multiplies them element by element and adds the results. For a = (2, 1, 0) and b = (1, 2, 0), a·b = 2×1 + 1×2 + 0×0 = 4. The dot product is large when the vectors point in the same direction and have large entries, zero when they are perpendicular, and negative when they point in opposite directions. Cosine similarity removes the effect of length by dividing by both vectors' lengths: cos(a, b) = a·b / (|a| |b|). Here |a| = |b| = √5, so cos = 4/5 = 0.8; for c = (0, 0, 3), cos(a, c) = 0. Cosine runs from 1 (same direction) through 0 (unrelated) to −1 (opposite). Jurafsky & Martin 2025

Keep the dot product in mind: it is the most important operation in this course. Attention (Day 4) is a dot product between a query and a key, and section 11 shows that the model's final prediction is one too.

9. word2vec and the analogy trick, with its limits

In 2013 Mikolov and colleagues trained simple word vectors on billions of words and found that directions in the space corresponded to relations between words. The most quoted example: take the vector for "king", subtract "man", add "woman", and the nearest word is "queen". The same works for "Paris − France + Italy ≈ Rome" and for grammar, as in "walking − walk + swim ≈ swimming". Mikolov 2013 Mikolov 2013 Nobody programmed this; it emerged from predicting nearby words. Stanford's GloVe (2014) obtained similar vectors from global co-occurrence counts. Pennington 2014

Try it: the map below shows word vectors squashed into two dimensions. Hover or tap a word to see its cosine similarity to the others, and build an analogy of the king − man + woman kind.

Notice that related words cluster, that the "male → female" offset is roughly parallel for several pairs, and that some analogies fail. This is a 2-D picture of a space with hundreds of dimensions, so on-screen distances are approximate.

The demo is often oversold. The standard evaluation quietly excludes the three input words from the answer; allow them, and the nearest vector to king − man + woman is usually "king" itself. Linzen 2016 Nissim and colleagues showed that several widely quoted "biased" analogies, such as man : doctor :: woman : nurse, are artefacts of that rule: with the honest query the answer is "doctor". Nissim 2020 Word vectors do encode real regularities, and real biases from their training text, but one arithmetic trick is not proof of either.

10. Static versus contextual embeddings

word2vec gives one vector per word, forever. That is a problem for "bank": the river bank and the savings bank share one blurry average vector. A static embedding cannot know which sense is meant. In 2018 ELMo showed that vectors taken from inside a language model that has read the whole sentence give every occurrence of "bank" a different vector reflecting its context, and feeding those contextual embeddings to other systems improved the state of the art on all six tasks tried. Peters 2018 BERT made the same point later that year and became the standard way to get such vectors. Devlin 2018

In a Transformer both kinds coexist. The embedding matrix of section 7 is static: row 3797 is the same numbers whatever the sentence. Each layer then mixes information between positions (Day 4), so by the top of the network the vector at "bank" has absorbed "river" or "savings" from its neighbours. "The embedding of a sentence" used for search (Day 12) means such a contextual vector, not a row of the lookup table.

11. The other end: the unembedding, and weight tying

Day 1 said the model outputs a probability for every vocabulary token. Where do 50,257 numbers come from, when the network's last layer produces one vector of 768? From another matrix, the unembedding or LM head, with one row per vocabulary token. The final vector is dot-producted with every row, giving 50,257 scores called logits; a softmax turns them into probabilities by exponentiating each and dividing by the total, so everything sums to 1. The model's prediction is literally "which vocabulary vector points most in the same direction as my current state?"

The unembedding has the same shape as the embedding matrix, so why not use one matrix for both? That is weight tying. Press and Wolf showed in 2016 that tying the input and output embeddings lowers perplexity and, for translation models, cuts the parameter count by more than half. Press 2016 Inan 2016 The original Transformer shares one matrix between its embedding layers and the pre-softmax projection, citing Press and Wolf, Vaswani 2017 and GPT-2 does the same, which is why its 124M parameters hold one 38.6M table rather than two. Very large models often untie them again, since the table is a tiny fraction of their size; Day 5 counts the whole block's parameters.

Next: Day 3 — Neural networks in one day

1. One neuron: weighted sum, bias, bend

A neuron in a neural network is a tiny function. It takes some input numbers, multiplies each by its own weight, adds the products up, adds a constant called the bias, and then passes the result through a simple non-linear function called the activation.

output = f(w₁x₁ + w₂x₂ + … + wₙxₙ + b)

A worked example. Let x = (2, −1), weights w = (0.5, 1.0) and bias b = 0.3. The weighted sum is 0.5×2 + 1.0×(−1) + 0.3 = 0.3; with the ReLU activation (next section) the output is 0.3. Change the bias to −0.5 and the sum becomes −0.5, which ReLU turns into 0: the neuron is "off". The weights decide which inputs the neuron cares about and in which direction; the bias sets how easily it switches on. Training is choosing these numbers automatically.

Try it: the widget below is one neuron with sliders for its two weights and its bias. Drag them and watch the output under ReLU and GELU.

Notice that with ReLU the output is exactly zero across a whole region of inputs, and that GELU is a smooth version of the same shape that dips slightly below zero before rising. Notice also that no setting of the sliders can make the output curve wiggle twice: one neuron is a very simple function. Power comes from stacking many of them.

2. Activations: why the bend is not optional

Suppose we left the activation out, so each neuron just computed a weighted sum. Stack ten such layers and you get… a weighted sum. Composing linear functions gives a linear function, so a hundred linear layers can only draw straight lines (or flat planes) through the data; they could never learn that "cat" and "dog" are near each other but far from "the", or anything with an "if". The non-linearity is what lets stacked layers represent curves, corners and switches, and with enough neurons a single hidden layer with a non-linear activation can approximate any continuous function to any precision, a result known as the universal approximation theorem. Wikipedia 2026 The theorem says such a network exists, not that training will find it, which is why the rest of today is about training.

Which bend? Early networks used S-shaped functions (sigmoid, tanh), which squash everything into a fixed range and whose slope vanishes for large inputs, making deep networks hard to train. The modern defaults are all variations on "pass positives, kill negatives":

ReLU (rectified linear unit) is max(0, x): cheap, with slope 1 for positive inputs so gradients pass through unchanged, and the activation that made deep networks practical around 2010–2012. Nair & Hinton 2010 Glorot 2011 GELU multiplies x by the probability that a standard normal variable is below x: a smooth curve that acts like ReLU for large |x| but lets small negatives through a little; it is used in BERT and the GPT family. Hendrycks 2016 SiLU (Swish) is x times the sigmoid of x, found by automated search and nearly identical in shape to GELU. Ramachandran 2017 SwiGLU is a layer design rather than an activation: the input goes through two linear maps, one passes through SiLU, and the results are multiplied, so one path "gates" the other. Shazeer showed in 2020 that this improved Transformer quality over plain ReLU or GELU layers, Shazeer 2020 and Llama and most open models since use it. Meta 2024 The differences between these are small; the difference between having one and having none is everything.

3. Layers, MLPs and parameters

A layer is a row of neurons that all read the same inputs. If a layer has 3,072 neurons each reading a 768-number vector, it has a 768 × 3,072 weight matrix plus 3,072 biases, and computing it is one matrix multiplication followed by the activation. Stacking two such layers, one that expands and one that shrinks back, gives the MLP (multi-layer perceptron), the workhorse of every Transformer block. GPT-2 small's MLP goes 768 → 3,072 → 768: 768×3,072 + 3,072 + 3,072×768 + 768 = 4,722,432 parameters, about 4.7 million, and there are 12 of them in the model, roughly 57 million of its 124 million parameters. Hugging Face 2026 Day 5 accounts for the rest.

The parameters of a model are all its weights and biases together (and, from Day 2, the embedding table). "GPT-3 has 175 billion parameters" means 175 billion adjustable numbers, every one set by the procedure in the next four sections. Nothing else is learned; the architecture, activations and tokenizer are fixed by the designers before training starts.

4. The loss: how wrong were we?

To improve, the model needs a single number that says how bad its current output is. For next-token prediction that number is the cross-entropy loss, also called negative log-likelihood: look up the probability the model gave to the token that actually came next, take its logarithm, and flip the sign. PyTorch 2026

loss = −log p(correct next token)

A worked example. Suppose the vocabulary has four tokens and after "The cat sat on the" the model outputs probabilities mat 0.60, floor 0.20, car 0.15, the 0.05. The real next token was "mat", so the loss is −ln 0.60 = 0.51. If the real token had been "floor", the loss would be −ln 0.20 = 1.61; if "the", −ln 0.05 = 3.00. The loss is 0 only for a probability of 1, and it climbs steeply as the probability of the truth falls, so putting 0.001 on the right answer costs 6.9 while putting 0.5 costs 0.69. For a whole text, average the losses over every position. Training a language model means driving this average down on trillions of tokens. The perplexity of Day 1 is simply e raised to the average loss: a loss of 0.51 on every token would be a perplexity of 1.67.

Why the logarithm rather than "1 minus the probability"? Products become sums: the probability of a text is the product of per-token probabilities (Day 1), and the log turns that into an average that does not shrink to zero for long texts. And the log punishes confident mistakes severely, which pushes the model to spread probability over everything plausible instead of betting on one option. It is also Shannon's quantity: the loss in bits (log base 2) is the number of bits needed to encode the correct token under the model, so minimising cross-entropy is literally maximising compression. Delétang 2023

5. Gradient descent and the learning rate

Now the key move. The loss is a function of all the parameters. For any single parameter, we can ask: if I increase it slightly, does the loss go up or down, and how steeply? That rate of change is the parameter's gradient. The gradient of the whole model is the list of these rates for every parameter, and it points in the direction of steepest increase of the loss. So to reduce the loss, move every parameter a small step the opposite way:

w ← w − η · ∂loss/∂w

The step-size multiplier η is the learning rate, and it is the most important number a practitioner sets by hand. A tiny one-parameter example shows why. Let the loss be (w − 3)², a bowl with its bottom at w = 3, so the gradient is 2(w − 3). Start at w = 0.

Try it: the widget below runs gradient descent on a bowl-shaped loss. Move the learning-rate slider and watch the ball settle, oscillate, or fly off.

Notice that too small is slow but safe, too large explodes, and the useful range between is narrow. Real runs warm up with small steps, hold a peak rate, then decay towards zero so the final parameters settle; GPT-3 175B used a peak learning rate of 0.6 × 10⁻⁴ and Llama 3 405B used 8 × 10⁻⁵, tiny numbers because each step is averaged over millions of tokens. Brown 2020 Meta 2024

Real losses are not clean bowls: with billions of parameters the surface has valleys, plateaus and saddle points, and we only see a noisy estimate of it from a small batch. Gradient descent still works, far better than anyone expected before 2012, and exactly why is still a research question.

6. Backpropagation: passing blame backwards

Gradient descent needs the gradient of the loss for every one of 175 billion parameters. Nudging each one and re-running the model would take 175 billion forward passes per step. Backpropagation computes all of them for about the cost of two forward passes, using one rule from first-year calculus: the chain rule, which says that if y depends on u and u depends on w, the effect of w on y is the effect of w on u times the effect of u on y.

Think of it as assigning blame. The last layer can see directly how much each of its outputs contributed to the loss. It tells the layer before it: "your output number 7 made my loss worse by this much per unit"; that layer multiplies by its own local slopes to work out how much blame goes to each of its weights and each of its inputs, and passes the input blame further back. Layer by layer, blame flows from the loss to the first weights, and every weight ends up with a number saying how it should change. Because every operation (multiply, add, ReLU, softmax) has a simple local derivative, the whole thing is mechanical, and libraries like PyTorch do it automatically by recording the forward computation and replaying it backwards. Wikipedia 2026

A two-step example. A neuron with one input computes y = w·x, and the loss is (y − t)² where t is the target. Let x = 2, w = 1, t = 5: then y = 2 and loss = 9. The chain rule gives ∂loss/∂w = ∂loss/∂y × ∂y/∂w = 2(y − t) × x = 2(−3) × 2 = −12. The negative sign means increasing w lowers the loss. One gradient step with η = 0.1 gives w = 1 − 0.1×(−12) = 2.2, so y = 4.4 and the loss falls from 9 to 0.36. A network with a million layers does the same thing a million times over.

Rumelhart, Hinton and Williams published the modern form for neural networks in 1986, after earlier discoveries in other fields, and every deep learning system still uses it. Rumelhart 1986 Karpathy's micrograd implements a complete backpropagation engine in about 100 lines of Python; to see the chain rule become code, start there. Karpathy 2020

7. Minibatches, SGD, and Adam

Computing the exact gradient over all 15 trillion training tokens before every step would be absurd. Instead each step uses a minibatch, a random sample of the data, and taking steps on the batch's noisy gradient estimate is stochastic gradient descent (SGD). The noise is mostly a feature: it is cheap and helps escape flat regions. GPT-3 175B used batches of 3.2 million tokens; Brown 2020 Llama 3 405B started at 4 million tokens per batch and grew to 16 million. Meta 2024 Bigger batches give cleaner gradients and spread across more GPUs, but past a point they stop helping.

Plain SGD uses the same learning rate for every parameter, but in a language model some parameters receive large, frequent gradients (common tokens' embeddings) and others tiny, rare ones. Adam keeps, for every parameter, a running average of its recent gradients (momentum, which smooths the noise) and of their squares (how big that parameter's gradients usually are), then divides one by the square root of the other, so each parameter effectively gets its own learning rate. Kingma 2014 AdamW is Adam with weight decay applied correctly: every step, all weights are also shrunk slightly towards zero, which discourages any single weight from growing huge and improves generalisation. Loshchilov and Hutter showed that the way earlier Adam implementations mixed weight decay into the gradient was subtly wrong, and that decoupling it makes the decay setting independent of the learning rate. Loshchilov 2017 AdamW is the optimiser behind essentially every LLM trained since; PyTorch's defaults are a learning rate of 0.001, averaging constants of 0.9 and 0.999, and a weight decay of 0.01, though LLM runs usually override all of them. PyTorch 2026

Adam's price is memory: two extra numbers per parameter, so the optimiser state for a 7B model in 32-bit precision is 56 GB on top of weights and gradients, a cost Day 10 returns to when explaining why fine-tuning big models is hard.

8. Overfitting: the train/validation gap

A model with billions of parameters can memorise its training data instead of learning the patterns in it. The symptom is a gap: the loss on training batches keeps falling while the loss on held-out text, the validation loss, stops falling and then rises. That rise is overfitting, and since the goal is to predict text the model has never seen, validation loss is the number that matters. Everyone in machine learning watches two curves and changes something when they separate.

The classic cures are more data, a smaller model, dropout (randomly switching off neurons during training so none can be relied on alone), Srivastava 2014 weight decay, and early stopping at the best checkpoint. Large language models mostly sidestep the problem with the first cure: with trillions of tokens they see most of their data only once, so there is little to memorise. The danger returns when data runs short: Muennighoff and colleagues found that repeating the dataset up to about four times costs almost nothing, but beyond that extra epochs stop helping and eventually hurt. Muennighoff 2023 "Running out of fresh data" is the modern form of overfitting, a theme Day 6 returns to.

9. Why GPUs

Look back at section 3: a layer is a matrix multiplication. So is attention (Day 4), so is the output layer. Nearly all the arithmetic in a language model, forward and backward, is multiplying large matrices, which is thousands of independent multiply-adds that can all happen at once. A CPU has a handful of fast cores built for doing different things in sequence. A GPU (graphics processing unit) has thousands of slower cores built for doing the same thing to many numbers at once, which is exactly what matrix multiplication needs; shading millions of pixels per frame, the job GPUs were built for, is the same kind of arithmetic.

The turning point was 2012, when AlexNet, trained on two consumer GPUs, won the ImageNet contest by a wide margin and made GPU training the default. Wikipedia 2026 Today's chips are built around matrix multiplication: NVIDIA lists 1,979 trillion 16-bit operations per second for the H100 SXM's tensor cores (assuming a sparse matrix; about half that for dense ones), with 80 GB of memory read at 3.35 TB/s. NVIDIA 2026 Llama 3 405B was trained on up to 16,000 of them at once. Meta 2024 The bandwidth number matters as much as the compute: when generating one token at a time, the bottleneck is usually moving weights from memory to the cores, not arithmetic, the key fact behind Day 7.

10. The whole thing in ten lines

Here is the training loop that every language model, from a toy to a frontier model, actually runs. Everything else in this course is detail about one of these lines.

model = Transformer(params)                 # random weights (Days 4–5)
opt = AdamW(model.parameters(), lr=3e-4)    # the optimiser (section 7)
for step in range(num_steps):
    tokens = next_batch(data)               # ids from the tokenizer (Day 2)
    logits = model(tokens[:, :-1])          # forward pass: scores for each next token
    loss = cross_entropy(logits, tokens[:, 1:])   # section 4
    loss.backward()                         # backpropagation fills in every gradient (section 6)
    opt.step()                              # move every parameter against its gradient (section 5)
    opt.zero_grad()                         # clear gradients for the next batch
    if step % 1000 == 0: print(step, loss.item(), validation_loss(model))

Two details are worth seeing. The input is the text minus its last token and the target is the text minus its first, so one pass over a 4,096-token sequence makes 4,095 predictions at once and scores them all; that is where the efficiency of Transformers over one-word-at-a-time RNNs comes from. And nothing in the loop knows about grammar, facts or chat: the loss is the only teacher, which is Day 1's point about compression. Karpathy reproduces GPT-2 with exactly this loop, fleshed out, in a four-hour video. Karpathy 2024

Next: Day 4 — Attention

1. The problem: a word means nothing on its own

Read two sentences: "She sat by the bank and watched the river" and "She walked into the bank and asked for a loan." The word is identical; the meaning is not. Day 2 gave every token a vector, but that vector is the same in both sentences. Something has to let the vector for bank change depending on river or loan. That something is attention.

Before 2017 the standard answer was the recurrent neural network (RNN): a network that reads tokens one at a time and carries a summary vector, the hidden state, from each step to the next. The best-known kind, the LSTM, dates from 1997 Wikipedia. RNNs have two weaknesses. First, the summary has a fixed size, so a long sentence has to be squeezed into the same few hundred numbers as a short one. The 2014 "sequence to sequence" translation system did exactly that: it read the whole source sentence into one vector and generated the translation from it Sutskever 2014. A study the same year showed translation quality dropping rapidly as sentences grew longer Cho 2014. Second, information has to survive many hops. A word at position 3 reaches position 300 only by being copied through 297 updates, and each update can smudge it. Training signals travel the same long road backwards and fade (the vanishing gradients of Day 3).

The fix came in 2014: let the translator look back at every source word at every step, with learned weights saying how much to look at each Bahdanau 2014. That was the first attention mechanism, bolted onto an RNN. Three years later a Google team asked whether the RNN was needed at all, kept only attention, and titled the paper "Attention Is All You Need" Vaswani 2017. Their model, the Transformer, is the subject of Day 5 and of the Transformer paper review. Dropping recurrence had a bonus that mattered as much as accuracy: every position can be processed at the same moment instead of one after another, which is exactly what a GPU is built for.

2. Queries, keys and values

Attention gives every token three vectors, all computed from the token's current vector x by multiplying with three learned matrices:

q = x·W_Q    k = x·W_K    v = x·W_V

The query is the question this token is asking ("I am a verb; where is my subject?"). The key is the label each token advertises ("I am a singular noun near the start"). The value is what a token hands over if it is chosen: the content that actually gets mixed into the output. Three separate vectors, because what a token looks for, what it can be found by, and what it contributes are three different jobs. The word "the" is rarely worth looking up, so its key should match little, yet it may still carry a value worth passing along. Keeping the roles apart lets the model learn each one independently.

In GPT-2 small the token vector has 768 numbers. Attention splits that into 12 heads (Section 7), so each head's q, k and v are 64 numbers long. The figure shows the flow for one token in one head.

3. Four steps: score, scale, softmax, sum

Step 1, score. Compare the query with each key using the dot product: multiply the vectors entry by entry and add up. The dot product is large and positive when two vectors point the same way, near zero when they are unrelated, and negative when they point in opposite directions. So q·k is literally "how well does this token's question match that token's label".

Step 2, scale. Divide every score by √d, where d is the length of the key vector (64 in GPT-2 small, so divide by 8). The reason is practical. If the entries of q and k are of size about 1, a dot product over d entries is typically of size about √d. Feeding numbers like ±8 into a softmax produces weights that are almost exactly 0 or 1, and in that regime the gradient is tiny, so learning stalls. The authors wrote that large dot products "push the softmax function into regions where it has extremely small gradients" Vaswani 2017. Scaling keeps the scores in the range where softmax is still soft. You will see the effect with real numbers in the next section.

Step 3, softmax. Turn the scaled scores into positive weights that add up to 1 (Day 3). Each weight says what fraction of the output comes from that token.

Step 4, weighted sum. Multiply each token's value by its weight and add them all up. That sum is the output for the querying token: its old vector, updated with context.

In matrix form, with Q, K and V holding all tokens' queries, keys and values as rows, the whole layer is one line:

Attention(Q, K, V) = softmax(Q·Kᵀ / √d)·V

Q·Kᵀ is an n × n table of scores, one row per querying token and one column per key. Keep that table in mind; it is where the cost of attention lives (Section 9).

4. A worked example with real numbers

Three tokens, "The cat sat", and two-dimensional vectors so you can check every number. Suppose the learned matrices have already produced the keys and values in the table, and the query of sat is q = [2, 0]. We will compute the new vector for sat.

The softmax step: exp of each scaled score, then divide by the total (1.00 + 16.92 + 4.11 = 22.03). The weights are 0.045, 0.768 and 0.187; check that they sum to 1. Now the weighted sum of values:

output = 0.045·[1, 0] + 0.768·[0, 1] + 0.187·[1, 1] = [0.23, 0.95]

The vector for sat is now mostly the value of cat. In words: the verb has found its subject and absorbed information about it. That is the whole mechanism; a real model does the same thing with 64- or 128-dimensional vectors, for every token at once, in every head of every layer.

Now redo step 3 without the √2 scaling. The exps become 1, 54.6 and 7.39, and the weights become 0.016, 0.867 and 0.117. Same ranking, but the distribution is sharper. With 64-dimensional vectors the unscaled scores would be about eight times larger still, and the softmax would be a near-perfect one-hot: no gradient, no learning. Dividing by √d is a small trick with a large effect.

Try it: the widget below runs the same arithmetic on a real sentence. Tap a word to make it the query and watch the scores, the softmax and the resulting weights over every other word.

Notice three things. The weights always sum to 1. Pronouns and verbs tend to put weight on the nouns they depend on. And a token usually gives itself some weight too: a query can match its own key, and often should.

5. The causal mask: why a decoder may not peek

A GPT-style model is trained to predict the next token. During training it sees a whole document at once and is scored at every position: at position 5 it must predict token 6, at position 6 token 7, and so on (Day 1). If the attention at position 5 were allowed to read token 6, the model would learn the easiest possible trick, copy what you can see, and it would be useless at generation time, when token 6 does not exist yet.

So decoder attention is causal: each token may attend only to itself and to earlier tokens. The implementation is blunt. Before the softmax, every score where the key comes after the query is replaced by −∞. Since exp(−∞) = 0, those positions get exactly zero weight and the remaining weights still sum to 1. In the n × n score table the mask zeroes everything above the diagonal:

              keys:  The    cat    sat
query The  [        s₁₁    −∞     −∞   ]
query cat  [        s₂₁    s₂₂    −∞   ]
query sat  [        s₃₁    s₃₂    s₃₃  ]
softmax runs along each row; exp(−∞) = 0, so masked entries get weight 0

The mask is applied whenever the model processes several tokens at once, which includes reading your prompt, not only training. Two consequences you will meet again. Because earlier tokens never look at later ones, their keys and values never change once computed, so they can be stored and reused as the model generates; that is the KV cache of Day 7. And an encoder-style model such as BERT drops the mask entirely, so every token sees every other, which is better for understanding and useless for generation (Day 5).

6. Many heads, many questions

One attention head computes one notion of relevance. But a token needs several at once: a verb wants its subject, its object, and the tense of the previous clause. Multi-head attention runs several attention operations in parallel, each with its own W_Q, W_K and W_V, and concatenates the results. The original Transformer used 8 heads; GPT-2 small uses 12; GPT-3 175B uses 96 Brown 2020.

Heads are not extra cost stacked on top. The model dimension is split among them: 768 = 12 heads × 64, so each head works with 64-number vectors and the total arithmetic is about the same as one head of size 768. After the heads are concatenated back to 768 numbers, one more learned matrix W_O mixes them. The paper's reason for heads is that a single head would average over all the kinds of relevance at once, and averaging blurs Vaswani 2017.

What do heads actually learn? Nobody programs them; they emerge from training, and researchers have catalogued several recurring kinds:

Previous-token heads attend almost entirely to the token one step back, so each token's vector picks up a copy of what came before it. Induction heads do something cleverer: when the current token is A, they look for an earlier A in the context, attend to the token that followed it, B, and raise the probability of B coming next. On the pattern "… Mr Dursley … Mr" an induction head predicts "Dursley". Anthropic's interpretability team found that an induction head is really a two-head circuit, a previous-token head in an earlier layer feeding an induction head in a later one, that these circuits appear abruptly during training, and that their appearance coincides with the model becoming able to learn from examples in its context Olsson 2022 Elhage 2021. In BERT, specific heads were found attending from verbs to their direct objects and from nouns to their determiners, while many heads simply park on punctuation when they have nothing to do Clark 2019.

Not every head earns its keep. Two 2019 studies showed that a large fraction of heads in trained translation models can be removed after training with little loss, and that a few specialised heads do most of the work Voita 2019 Michel 2019. Heads are cheap redundancy, not a precise set of modules.

7. Self-attention and cross-attention

Everything so far was self-attention: queries, keys and values all come from the same sequence. The 2017 Transformer also had cross-attention, where the queries come from one sequence and the keys and values from another. Its decoder, generating a French sentence, sent queries to the encoder's representation of the English sentence; that is how each French word found the English words it should translate Vaswani 2017. It is Bahdanau's 2014 idea, kept intact inside the new architecture.

A decoder-only model such as GPT has no encoder and uses self-attention only. When it needs to read a document and then answer a question about it, the document is simply placed in the same context as the question, and self-attention does the rest. That is why retrieval-augmented generation (Day 12) needs no new mechanism: retrieved text is just more tokens. Cross-attention survives in encoder-decoder models such as T5 and in some multimodal systems, where image features serve as the keys and values that text queries look at.

8. The n² bill

Every head computes an n × n table of scores for a context of n tokens. With n = 1,024, that is about one million scores per head, and GPT-2 small has 12 heads in each of 12 layers, so about 150 million scores per forward pass. Make the context 128,000 tokens and the table has 16 billion entries per head. Doubling the context quadruples the score table. Both the time and, if you store the table, the memory grow with n², which is what "attention is O(n²) in sequence length" means. The rest of the network (the MLPs of Day 5) grows only linearly with n, so long contexts are where attention dominates the cost.

This is the reason long context was slow to arrive, and the reason it eventually did. FlashAttention computes exactly the same result without ever writing the full n × n table to the GPU's main memory: it processes the table in tiles that fit in the much faster on-chip memory, trading some recomputation for far fewer memory transfers Dao 2022. A second line of work shrinks the memory for keys and values by sharing them across heads: multi-query attention uses one key and value head for all query heads Shazeer 2019, and grouped-query attention is the compromise in between Ainslie 2023. Llama 3 8B, for instance, has 32 query heads but only 8 key/value heads Meta 2024. Day 11 covers both, with the FlashAttention paper review.

9. Attention is a soft lookup

A dictionary in a programming language is a hard lookup: you provide a key, and you get back exactly one value if the key matches and nothing otherwise. Attention is the soft version. Every stored key matches a little, the match strength is a dot product rather than equality, and the result is a blend of all values weighted by match. The softness is not a compromise; it is the point. A hard lookup has no gradient, so nothing about it could be learned. A soft lookup is a smooth function of the queries, keys and values, so gradient descent (Day 3) can tune the matrices that produce them until the lookups the model performs are the useful ones.

That gives a compact way to hold the whole day in your head. Attention: each token's query is matched against every token's key; softmax turns the matches into weights; the output is the weighted sum of values. Causal: only earlier keys are visible. Multi-head: several lookups in parallel, each with its own notion of "match". Cost: an n × n table per head.

Next: Day 5 — The Transformer, fully assembled

1. The whole machine in one picture

Here is the entire architecture of GPT-2, Llama 3 and, as far as is public, every frontier language model of 2026, read from bottom to top:

token ids ──▶ token embedding (+ position information)
              │
              ▼
          ┌─────────── block, repeated N times ───────────┐
          │  x = x + Attention(Norm(x))     ← mixes tokens │
          │  x = x + MLP(Norm(x))           ← per token    │
          └───────────────────────────────────────────────┘
              │
              ▼
          final Norm ──▶ LM head (d_model × vocab) ──▶ logits ──▶ softmax

Call the vector x that runs down the middle the residual stream: one vector of d_model numbers per token, which every block reads from and adds to Elhage 2021. GPT-2 small has d_model = 768 and N = 12 blocks; Llama 3 70B has d_model = 8,192 and N = 80 Meta 2024. The shape is the same; only the numbers change. The original 2017 model had an encoder and a decoder with cross-attention between them Vaswani 2017; Section 9 explains why modern language models keep only the decoder half.

2. Tokens become vectors, and position has to come from somewhere

The first layer is a lookup table: row t of the token embedding matrix is the vector for token id t (Day 2). For GPT-2 that table is 50,257 rows × 768 columns, and it is the single largest piece of the model (Section 7). But attention, as defined on Day 4, is blind to order: shuffle the tokens and every query meets the same set of keys and values, so the outputs are the same set, shuffled. "Dog bites man" and "man bites dog" would be identical. Position has to be injected on purpose, and there are four common ways.

Learned absolute positions. A second lookup table, one row per position, added to the token vector. GPT-1 chose this over the 2017 paper's sinusoids Radford 2018, and GPT-2 and GPT-3 kept it. Simple, but position 1,025 has no row, so the model cannot read anything longer than it was trained on.

Sinusoidal positions. The 2017 paper added a fixed vector of sines and cosines at a spread of frequencies: PE(pos, 2i) = sin(pos / 10000^(2i/d)), PE(pos, 2i+1) = cos of the same. Low-frequency dimensions change slowly along the sequence, high-frequency ones quickly, like the hands of a clock, and the authors hoped the model could learn to attend by relative offset because any fixed shift is a linear function of these vectors Vaswani 2017.

Rotary position embedding (RoPE) is what Llama, Qwen, DeepSeek and most open models use today. Instead of adding a position vector to the token, RoPE rotates the query and key vectors. Take the 64 numbers of a head's query, pair them up into 32 two-dimensional points, and rotate each point by an angle proportional to the token's position, with a different rotation speed for each pair. Do the same to the keys. Now the dot product between a query at position m and a key at position n depends only on the difference m − n, because rotating both arrows by the same amount leaves the angle between them unchanged, and the angle between them is what the dot product measures Su 2021. So RoPE encodes absolute position in the vectors but relative position in the attention scores, which is the quantity that matters. The rotation speeds are set by one number, the base θ; Llama 3 raised it to 500,000 to make long contexts work Meta 2024. The RoPE paper review goes through the maths.

ALiBi drops position vectors entirely and instead subtracts a penalty from each attention score that grows with the distance between query and key, with a different slope per head. Models trained this way extrapolated to sequences longer than any seen in training Press 2021. It is less common in 2026 than RoPE but the idea, "position as a bias on scores rather than a feature in the vector", shows how many choices are open.

Try it: the widget shows the sinusoidal table as a heatmap, and a RoPE rotation that you can drive with a position slider.

In the heatmap, read down one column: the leftmost dimensions flip quickly with position, the rightmost barely move. In the RoPE view, slide the position and watch the query and key arrows turn together; the angle between them, and so their dot product, depends only on how far apart they are.

3. Inside the block, first stop: normalisation

Each block begins by normalising the token vector. Layer normalisation (LayerNorm) takes the d_model numbers of one token, subtracts their mean, divides by their standard deviation, then multiplies by a learned per-dimension gain and adds a learned bias Ba 2016. The effect is that whatever scale the residual stream has reached, the attention and MLP always see inputs of a standard size. RMSNorm is the simplification used by Llama and most newer models: skip the mean subtraction, divide by the root mean square, and keep only the gain Zhang 2019 Touvron 2023. It is slightly cheaper and works as well.

Where the norm sits matters more than which norm. The 2017 Transformer applied it after each sub-layer's residual addition (post-norm). GPT-2 moved it to the input of each sub-layer, with one extra norm after the final block (pre-norm) Radford 2019. A 2020 analysis explained why pre-norm trains more easily: with post-norm the gradients at the output layers are large at initialisation, which is why the original recipe needed a careful learning-rate warm-up; with pre-norm the gradients are well behaved from the start and warm-up can be dropped Xiong 2020. Every large model since has used pre-norm, and the pseudo-code in Section 11 is written that way.

4. Residual connections: the highway through the stack

Each sub-layer's output is added to its input rather than replacing it: x = x + f(x). This is the residual connection, introduced for image networks in 2015 when researchers found that plain networks got worse past about 20 layers, not because they overfit but because they could not be optimised He 2015.

Why it works: the gradient of x + f(x) with respect to x is 1 plus the gradient of f. That "1" is an identity path straight through every block, so the training signal reaches layer 1 of a 96-layer model without being multiplied by 96 small numbers on the way. The arithmetic is unforgiving without it: if each layer scaled the gradient by 0.9, after 96 layers it would be 0.9⁹⁶ ≈ 0.00004, and the early layers would never learn. The residual path makes depth cheap. It also changes what a block has to do. Instead of computing the complete representation afresh, a block only has to learn a correction, which is easier, and a block that has nothing useful to add can output something close to zero and do no harm. This is the view behind the "residual stream" picture: a shared memory that layers read from and write to Elhage 2021.

5. The MLP: four times wider, then back

After attention has mixed information across tokens, the MLP (also called the feed-forward network) processes every token separately with the same two matrices. The first expands the vector from d_model to 4 × d_model, a non-linearity is applied, and the second projects back down: 768 → 3,072 → 768 in GPT-2. The non-linearity is GELU in the GPT family, a smooth cousin of ReLU (Day 3) Hendrycks 2016. The MLP has no view of other tokens, which is why the two halves of the block are complementary: attention moves information around, the MLP transforms it.

The width matters because the MLP is where most of the parameters, and apparently most of the stored knowledge, live. One study showed that each MLP layer behaves like a key-value memory, with the first matrix's rows acting as pattern detectors and the second matrix's columns holding what to write when a pattern fires; lower layers respond to surface patterns, upper layers to meaning Geva 2020. Another located factual associations such as "the Eiffel Tower is in Paris" in the MLPs of middle layers and edited them directly Meng 2022.

Modern models use a variant, SwiGLU, which has three matrices instead of two: one branch is passed through a smooth gate (the Swish function) and multiplied element by element with the other branch before the down-projection. It performed better than plain GELU MLPs at the same parameter count Shazeer 2020, and Llama adopted it Touvron 2023. Because there are three matrices, the hidden width is reduced from 4× to keep the cost similar: Llama 3 8B expands 4,096 to 14,336, about 3.5× Meta 2024.

6. Stack it, normalise, and read out

Stack N blocks; each has its own weights. Apply the final norm. Then the LM head: one matrix of size d_model × vocab that turns the last vector for each position into a score (a logit) for every token in the vocabulary. Softmax turns the logits into next-token probabilities, which is the output you met on Day 1. GPT-2 does not have a separate LM head matrix at all: it reuses the token embedding table transposed, a trick called weight tying that saves 38 million parameters and slightly improves the model Press 2016. Many larger models untie the two, because at their scale the saving is negligible.

During training, the head is applied at every position and the loss is averaged across all of them, so one document of 1,024 tokens gives 1,024 training examples in one pass. During generation only the last position's logits are needed, which is why the per-token cost of generation is far below that of a full forward pass over the context once the KV cache (Day 7) is in place.

7. Counting GPT-2 small, parameter by parameter

Now the arithmetic. GPT-2 small: vocabulary 50,257, context 1,024, d_model 768, 12 heads, 12 layers, MLP width 3,072, LayerNorm with gain and bias, weight-tied head Hugging Face docs. Every linear layer has a bias, which adds a little.

So "GPT-2 small" is 124 million parameters, matching the published weights Hugging Face model card. The paper's table said 117M; OpenAI later noted that the original counts were wrong OpenAI 2019. Three things to notice. The token embedding alone is 31% of the model. Within a block, the MLP holds two-thirds of the weights (4.72M against 2.36M for attention). And the positional table is tiny, under 1%. In fp16, two bytes per parameter, the whole model is about 249 MB; a 7B model by the same rule is 14 GB.

Try it: tap each part of the block diagram below to read what it does and how many of GPT-2 small's parameters it owns.

Compare the counts with the table. The two residual additions and the softmax own no parameters at all; they are pure arithmetic, which is why they are often left out of diagrams despite doing essential work.

8. From 2019 to 2026: the same shape, bigger numbers

The table below is the single most useful thing to memorise today, because every model card you will ever read is a row of it.

Sources: GPT-2 from its config and paper Hugging Face docs; GPT-3's 96 layers, d_model 12,288, 96 heads of 128, context 2,048 and 4× MLP from its paper, which also says it reuses GPT-2's architecture, tokenizer and pre-normalisation Brown 2020; Llama 3 from its Table 3 Meta 2024. GPT-3 also alternated dense attention with locally banded sparse attention in some layers, an early attempt at the n² problem that later models dropped in favour of FlashAttention.

Read across a row and you see what changed. Head size settled at 128. Key/value heads shrank relative to query heads (grouped-query attention, Day 4) to keep the KV cache small. Vocabularies grew from 50K to 128K so that each token covers more text. Context grew from 1K to 128K through RoPE and long-context training rather than through a bigger position table. Two further knobs appear in the newest models and get their own days: mixture of experts, where the MLP is replaced by several MLPs and a router picks a few per token, so that DeepSeek-V3 has 671B parameters but uses 37B for any given token DeepSeek 2024 (Day 11, Mixtral paper review), and compressed attention variants that shrink the keys and values further (Day 11).

9. Encoder-only, decoder-only, encoder-decoder

The 2017 Transformer had two stacks. The encoder reads the whole input with unmasked attention, so every token sees every other. The decoder generates the output with causal self-attention plus cross-attention into the encoder. Three families grew from this.

Encoder-only models, led by BERT (2018), keep the unmasked stack and train it to fill in blanked-out words: hide 15% of the tokens and predict them from both sides Devlin 2018. They produce excellent representations for classification, search and sentence embeddings, and are still the backbone of most embedding models used in retrieval (Day 12, BERT paper review). But they cannot generate text: there is no notion of "next".

Encoder-decoder models such as T5 (2019) keep both stacks and cast every task as text-to-text, with a corrupted-span objective for pretraining Raffel 2019. They remain strong for translation and summarisation, where the input is fixed and fully known in advance.

Decoder-only models, the GPT line, keep only the causal stack and train on next-token prediction. This is what won for general-purpose generation, and the reasons are mostly practical. One objective covers everything: a document, a dialogue, code and a question followed by its answer are all just sequences to continue. Every position produces a training signal, where BERT's objective trains on only the 15% of positions that were masked. There is no gap between how the model is trained and how it is used: prompt in, continuation out. The causal structure allows the KV cache, so generation is cheap. And a 2022 study that trained all three families under matched conditions found that decoder-only models with next-token prediction generalised best to new tasks after pretraining alone, before any task-specific tuning Wang 2022. Add the scaling results of Day 6 and the field converged.

10. What the layers seem to do

Every block has the same shape, but trained blocks are not interchangeable. Probing studies, which train small classifiers to read specific information out of each layer, give a consistent rough picture. In BERT, the layers recover the classical language-processing pipeline in order: part of speech early, then syntax, then semantic roles and coreference late Tenney 2019. In GPT-2 small, attention in the middle layers lines up best with grammatical dependencies, and the deepest layers capture the longest-distance relations Vig 2019. MLP layers respond to surface patterns low in the stack and to meaning higher up Geva 2020, with factual recall concentrated in middle-layer MLPs Meng 2022.

So the folk summary, "early layers do local and syntactic work, later layers do semantic and factual work", is about right, with two cautions. First, the model is not a clean pipeline; information is spread and revisited, and later layers can overturn earlier guesses. Second, these are findings about particular models probed with particular tools, not laws. Day 13 covers interpretability properly, including the circuits view that explains induction heads (Day 4) and the sparse features that newer methods extract from the residual stream.

11. The forward pass in twenty lines

Here is a GPT-2-style forward pass with nothing hidden. Shapes are in comments: n is the number of tokens, d is d_model.

def forward(tokens):                        # tokens: n integer ids
    n = len(tokens)
    x = tok_emb[tokens] + pos_emb[0:n]      # (n, d)   residual stream starts here
    for layer in layers:                    # N blocks, each with its own weights
        h = norm1(x)                        # pre-norm
        q = split_heads(h @ Wq)             # (heads, n, d_head)
        k = split_heads(h @ Wk)
        v = split_heads(h @ Wv)
        s = q @ k.T / sqrt(d_head)          # (heads, n, n) scores
        s = s + causal_mask                 # −inf above the diagonal
        a = softmax(s) @ v                  # weighted sum of values
        x = x + merge_heads(a) @ Wo         # residual add 1
        h = norm2(x)
        x = x + gelu(h @ W1) @ W2           # MLP 4× wide, residual add 2
    x = final_norm(x)
    logits = x @ tok_emb.T                  # (n, vocab)  tied LM head
    return logits                           # softmax(logits[-1]) = next-token probs

Biases are omitted for readability. Replace norm with RMSNorm, drop pos_emb and rotate q and k by position inside the loop, swap the MLP line for the gated SwiGLU form, share k and v across groups of heads, and you have Llama 3. The working code behind this sketch is a few hundred lines in nanoGPT Karpathy 2022.

Next: Day 6 — Scaling laws: how big, how much data, how much compute

1. From the web to a training set

The raw material is the web. Common Crawl is a nonprofit that has crawled the public web since 2007; its archive holds over 300 billion pages and grows by 3 to 5 billion pages a month. Common Crawl 2026 Most of it is useless for training: menus, ads, spam and endless copies of the same page. Turning the crawl into a dataset is a pipeline of filters.

The FineWeb team documented their pipeline in detail, so it makes a good worked example. Penedo 2024

1. Text extraction. Pull the main article text out of the raw HTML. FineWeb ran the trafilatura extractor on the raw pages rather than using Common Crawl's pre-extracted text, which kept too much boilerplate.
2. URL filtering. Drop pages from a blocklist of adult and spam domains.
3. Language identification: a small classifier (fastText) guesses the language of each page. FineWeb kept a page as English only if the classifier was at least 65% confident. Meta's CCNet pipeline did the same for a hundred languages. Wenzek 2019
4. Quality filtering: rules that catch junk. FineWeb reused the heuristics from DeepMind's MassiveText (the Gopher dataset) Rae 2021 and from Google's C4: drop pages with "lorem ipsum", with curly brackets (code dumps), with too many very short lines, with too few lines ending in punctuation. For FineWeb-Edu they added a learned filter: Llama-3-70B-Instruct graded about 460,000 pages for educational value on a 0 to 5 scale, a small classifier was trained to imitate those grades, and only pages scoring 3 or higher survived. That left 1.3 trillion tokens out of 15 trillion. Penedo 2024
5. Deduplication: remove copies and near-copies. FineWeb used MinHash, a trick that hashes overlapping 5-word chunks so two documents that share about 75% of their chunks are flagged as duplicates without comparing them word by word. Why bother? Lee and colleagues showed that deduplication makes models regurgitate memorized text far less often and reach the same or better accuracy in fewer steps. Lee 2021
6. Mixing: decide the proportions. Web text is not everything. Llama 3's pretraining mix was roughly 50% general knowledge, 25% maths and reasoning, 17% code and 8% multilingual text, and the proportions were chosen by training small models on candidate mixes and seeing which did best. Grattafiori 2024

Several open datasets let you see the result. Sizes are in tokens unless noted, and the token counts depend on the tokenizer used (see Day 2).

The so-what: two models with the same architecture and compute can differ a lot because one team filtered better, and none of that work is visible in the final model.

2. One training step, and the knobs on it

The objective has not changed since Day 1: predict the next token. For every position in every sequence, the model outputs a probability for each token in the vocabulary, and the loss is how surprised it was by the real one:

loss = −log p(correct next token), averaged over every position in the batch

Every token is a training example, so "trained on 15 trillion tokens" means 15 trillion of these little predictions were scored. One training step looks like this:

for step in range(total_steps):
    batch = next_batch(tokens_per_batch)        # e.g. 4M tokens as 1,024 sequences of 4,096
    logits = model(batch.inputs)                # forward pass
    loss = cross_entropy(logits, batch.targets) # inputs shifted by one token
    loss.backward()                             # backward pass: gradient for every weight
    clip_grad_norm_(model.parameters(), 1.0)    # gradient clipping
    lr = schedule(step)                         # warmup, then cosine decay
    optimizer.step(lr)                          # AdamW update
    optimizer.zero_grad()
    if step % save_every == 0:
        save_checkpoint(model, optimizer, step)

Three knobs on this loop matter enough to have names.

Batch size, measured in tokens, is how many tokens are scored before each weight update. GPT-3 175B used 3.2 million tokens per batch. Brown 2020 The first LLaMA models used 4 million. Touvron 2023 Llama 3 405B started at 4 million, doubled to 8 million after 252 million tokens, and doubled again to 16 million after 2.87 trillion tokens. Grattafiori 2024 Big batches keep thousands of GPUs busy and average out gradient noise; too big and each extra token adds little. Growing the batch during training is the compromise: small while the gradient changes fast, large once it settles.

Learning-rate warmup means starting the learning rate near zero and raising it linearly over the first few thousand steps. At step zero the weights are random and the gradients point in strange directions; a full-size step then can wreck the model before it learns anything. Cosine decay then lowers the rate smoothly along a half-cosine curve until the end of training, an idea borrowed from a 2016 paper on restarting learning rates. Loshchilov 2016 LLaMA used 2,000 warmup steps, decayed to 10% of the peak, with AdamW (β₁ = 0.9, β₂ = 0.95) and weight decay 0.1. Touvron 2023 Llama 3 405B used a peak of 8×10⁻⁵, 8,000 warmup steps, and a cosine decay to 8×10⁻⁷ over 1.2 million steps. Grattafiori 2024 The Day 3 widget showed what too high a learning rate does: the loss diverges. Warmup and decay are the two ends of the same safety rule.

Gradient clipping: if the gradient vector is longer than a limit (almost always 1.0), shrink it to that length before the update. One bad batch, a corrupted file, a block of random bytes, can produce a huge gradient; clipping stops it from throwing the weights across the room. The trick comes from the recurrent-network era, where exploding gradients were a daily problem. Pascanu 2012 LLaMA clipped at 1.0. Touvron 2023

3. How much compute does a run need?

A FLOP is one floating-point operation, a multiply or an add. Training compute has a formula you can do in your head:

training FLOPs ≈ 6 · N · D

N is the number of parameters and D the number of training tokens. Where does the 6 come from? The forward pass uses every parameter in one multiply and one add per token, 2 FLOPs; the backward pass costs about twice that, another 4. Attention adds a little on top, small enough to ignore at normal context lengths. Kaplan 2020

Worked example: a 7-billion-parameter model on 2 trillion tokens.

6 × 7×10⁹ × 2×10¹² = 8.4×10²² FLOPs

Now the hardware. NVIDIA lists the H100 SXM at 1,979 TFLOPS for bf16 matrix maths, but that figure assumes structured sparsity, which dense training does not use, so the real peak is half: about 989 TFLOPS, close enough to 10¹⁵ FLOP/s. NVIDIA 2026 No run reaches peak. The share you get is called model FLOPs utilisation (MFU): useful FLOPs per second divided by the chip's peak. The PaLM paper introduced the measure and reported 21% for GPT-3, 33% for Gopher and 46% for PaLM itself. Chowdhery 2022 Llama 3 reported 38 to 43% on up to 16,000 H100s. Grattafiori 2024 Take 40% as the planning number: 4×10¹⁴ useful FLOP/s per GPU.

8.4×10²² ÷ 4×10¹⁴ = 2.1×10⁸ GPU-seconds ≈ 58,000 GPU-hours

On 1,024 GPUs that is about 57 hours, two and a half days. At the on-demand price a specialist cloud listed in October 2026, $3.99 per H100-hour, Lambda 2026 the compute alone is about $230,000. Big labs own their hardware or pay less; DeepSeek priced its V3 run at an assumed $2 per GPU-hour, and 2.788 million H800 GPU-hours came to $5.6 million. DeepSeek 2024 None of these figures include salaries, failed runs, data work or post-training, which is why "it cost $5 million" headlines are always too low.

Cross-check the formula against a real run. Llama 3 405B: 6 × 405×10⁹ × 15.6×10¹² = 3.8×10²⁵, and the paper reports 3.8×10²⁵ FLOPs. Grattafiori 2024 For the 8B model the formula gives about 500,000 GPU-hours at 40% MFU, and Meta's model card reports 1.3 million. Meta 2024 Reported totals are usually higher than the napkin number, because the napkin assumes steady utilisation and ignores restarts and evaluation runs. Treat 6·N·D as a floor, not a bill.

4. Scaling laws: how big, and how many tokens?

Given a compute budget, you face a trade: a bigger model on fewer tokens, or a smaller model on more? A scaling law is a fitted curve that answers this by training many small runs and extrapolating. Read the scaling-laws paper review for the full story; here is the short version.

Kaplan and colleagues at OpenAI found in 2020 that loss falls as a smooth power law in parameters, data and compute, across seven orders of magnitude. Their fit also said that for a fixed budget you should spend most of it on a bigger model and stop training early. Kaplan 2020 GPT-3 followed that advice: 175 billion parameters on only 300 billion tokens, under two tokens per parameter. Brown 2020

DeepMind redid the experiment in 2022 with one fix: they set the learning-rate schedule to match each run's length, rather than reusing one long schedule for short runs. The answer flipped. Parameters and tokens should grow in equal proportion. To prove it they trained Chinchilla, 70 billion parameters on 1.4 trillion tokens, with the same compute as their earlier Gopher, 280 billion parameters on 300 billion tokens, and Chinchilla won on almost every benchmark. Hoffmann 2022 1.4 trillion divided by 70 billion is 20, and "about 20 tokens per parameter" became the rule of thumb for a compute-optimal model: the lowest loss you can reach for a given number of training FLOPs.

Then the rule was broken on purpose. Llama 3 8B was trained on over 15 trillion tokens, nearly 1,900 tokens per parameter, a hundred times past Chinchilla-optimal. Meta wrote that the compute-optimal amount for an 8B model is about 200 billion tokens and that the model kept improving "even after the model is trained on two orders of magnitude more data". Meta 2024 Why train inefficiently? Because Chinchilla only counts training. Once deployed, every token a model serves costs about 2·N FLOPs, and a model that answers billions of requests pays that bill many times over; a smaller model trained longer is cheaper on every one of them. Sardana and colleagues extended the Chinchilla analysis to include expected inference demand and found exactly this: the more you plan to serve, the smaller and more overtrained the model should be. Sardana 2023 The Llama 3 paper review has the details of that run.

Try it: set 8B parameters and 15T tokens below and read off the FLOPs and GPU-hours; then drag the tokens back to the Chinchilla line (about 160B) and see how much cheaper the run gets and how much higher the loss curve sits.

Notice that moving along the Chinchilla line keeps training cheap for a given loss, while sliding right (more tokens, same size) buys a lower loss at a price. The log-log curves are straight lines: that straightness is what "power law" means, and it is what lets labs predict a big run from small ones.

5. Spreading one model over thousands of GPUs

Two reasons a model does not fit on one GPU. Time: 58,000 GPU-hours on one card is six and a half years. Memory: in training each parameter needs far more than two bytes. With mixed-precision Adam you store a 16-bit weight, a 16-bit gradient, a 32-bit master copy, and two 32-bit optimizer moments: 16 bytes per parameter. Rajbhandari 2019 A 7B model needs 112 GB before counting activations, and an H100 has 80 GB. NVIDIA 2026 So the model, and the work, must be split. There are four standard ways, and real runs combine them.

Data parallelism: every GPU holds a full copy of the model and processes a different slice of the batch. After the backward pass the GPUs average their gradients (an operation called all-reduce) so every copy takes the same step. Simple, but it does nothing for memory: each GPU still needs the whole model.

Tensor parallelism: split each weight matrix into slices, one per GPU, so each GPU computes part of every layer and the partial results are stitched together. This needs communication inside every layer, so it is used only between GPUs with very fast links, typically the eight in one server. NVIDIA's Megatron-LM showed how to do it for Transformer layers. Shoeybi 2019

Pipeline parallelism: put layers 1 to 10 on one GPU, 11 to 20 on the next, and so on, and pass activations down the line. To keep later GPUs from idling, the batch is cut into micro-batches that flow through like an assembly line; the idle time that remains is the pipeline bubble. GPipe introduced the scheme. Huang 2018

ZeRO and FSDP: data parallelism without the redundancy. Instead of every GPU holding every optimizer state, gradient and parameter, each GPU holds a shard, and the full weights of a layer are gathered just in time for its forward and backward pass, then released. Memory per GPU falls roughly in proportion to the number of GPUs. Microsoft's ZeRO described the idea in three stages, Rajbhandari 2019 and PyTorch's Fully Sharded Data Parallel is the mainstream implementation. Zhao 2023 PyTorch 2026

Llama 3 405B used all of these at once, plus context parallelism, which splits very long sequences across GPUs, on up to 16,000 H100s. Grattafiori 2024

6. Making the numbers fit: mixed precision and checkpoints

Every number in training has a format. 32-bit floats (fp32) are the safe default; 16-bit floats halve the memory and, on tensor cores, run far faster. Mixed precision does the heavy matrix maths in 16 bits while keeping an fp32 master copy of the weights for the small, precise updates. Micikevicius 2017 fp16 has a tiny range, so small gradients underflowed to zero and had to be rescaled. bf16 (brain float) spends its 16 bits differently, keeping fp32's 8-bit exponent and giving up precision in the fraction, so it covers the same range without rescaling. It is today's default; Llama 3 reports its MFU in bf16. Grattafiori 2024 The next halving is fp8: NVIDIA proposed two 8-bit formats in 2022 and showed training matched 16-bit accuracy on models up to 175B, Micikevicius 2022 and DeepSeek-V3, a 671B-parameter model, was trained with fp8 mixed precision with a loss that stayed within 0.25% of a bf16 baseline. DeepSeek 2024 Each halving roughly doubles throughput and halves memory traffic, which is why precision keeps falling.

A checkpoint is a snapshot of everything needed to resume: weights, optimizer state, the data position, the random seeds. At 16 bytes per parameter a 405B checkpoint is around 6.5 TB, and writing it stalls thousands of GPUs, so there is a real trade between saving often (less lost work when something breaks) and saving rarely (less time stalled). The Llama 3 team wrote that they tuned their storage so they could checkpoint more often with less pause time. Grattafiori 2024

7. When the machines break

With 16,000 GPUs, something is always failing. The Llama 3 paper gives the clearest public numbers: during a 54-day stretch of 405B pretraining there were 466 job interruptions, 47 planned and 419 unexpected. About 78% of the unexpected ones were confirmed or suspected hardware problems, and GPU faults alone were 58.7%. Despite this, effective training time stayed above 90%. Grattafiori 2024 Do the division: 419 unexpected stops in 54 days is almost eight a day, one every three hours, around the clock. Each one stops the whole job; the bad machine is swapped out and training resumes from the last checkpoint, redoing whatever was computed since.

That is why checkpoint frequency, automatic restart and fast fault diagnosis are a research topic in their own right: a run that loses 10% of its time to failures loses 10% of a budget measured in tens of millions of dollars. Meta reported more than 95% effective training time for the earlier Llama 3 runs on its 24,000-GPU clusters. Meta 2024

8. Emergent abilities, or a trick of the ruler?

Scaling laws describe loss, which falls smoothly. But what people care about are abilities: doing arithmetic, following instructions, answering trivia. In 2022 a group of Google researchers catalogued tasks where accuracy sat near zero for small models and then jumped sharply at some scale, and called these emergent abilities: abilities "not present in smaller models but present in larger models", which cannot be predicted by extrapolating from the small ones. Wei 2022 If true, this is unsettling: you cannot know what a model will be able to do until you have spent the money.

A year later Stanford researchers argued that most of the jumps are a measurement artifact. The tasks that looked emergent were scored with all-or-nothing metrics such as exact match: a five-digit sum is right or wrong, so a model that gets four digits right scores the same as one that gets none. Under such a metric a smooth improvement in per-token accuracy looks like nothing, nothing, nothing, then a cliff. Score the same models with a continuous metric, such as how many digits are right, and the curves become smooth and predictable. They also showed they could manufacture "emergence" on ordinary vision tasks just by picking a harsh enough metric. Schaeffer 2023

The debate matters less for whether abilities arrive with scale (both sides agree they do) than for whether they can be forecast. The practical answer is that labs now fit scaling laws to benchmarks, not just to loss: the Llama 3 team predicted the 405B model's ARC Challenge score from small runs by fitting one curve from compute to the loss on the benchmark's answers, then a second from that loss to accuracy. Grattafiori 2024 Which is to say: with the right ruler, most of the magic is a straight line on log paper.

9. Reading the loss curve

The number everyone watches during a run is the training loss, the average −log p(correct token), measured in nats (natural-log units) per token. It is easier to feel as perplexity, e raised to the loss, which you can read as "the model is choosing between this many equally likely tokens". Wikipedia

perplexity = e^loss loss 2.0 → 7.4 loss 1.8 → 6.0

So "loss 2.0 versus 1.8" means the model went from being torn between about seven plausible next tokens to about six. Put in bits, 0.2 nats is 0.29 bits per token. That sounds like nothing. But the easy tokens (the "the", the closing bracket, the second half of a common word) are learned in the first few billion tokens; everything after that is spent on the hard ones, which is why the curve is almost flat for most of a run and why a tenth of a nat is worth millions of dollars at the top end. The shape is always the same: a fast drop in the first percent of training, a long slow power-law tail, the occasional spike when a bad batch or an instability hits, and a last visible dip when the cosine decay brings the learning rate down. The LLaMA paper's first figure shows this for four model sizes: bigger models sit lower along the whole curve, and all are still falling when training stops. Touvron 2023

Two cautions. Loss is only comparable between runs with the same tokenizer and the same held-out data: a bigger vocabulary packs more information into each token, so loss per token is higher even if the model is better. And loss is a proxy: nobody can tell from loss alone whether a model can write a working function.

Next: Day 7 — Inference: how a model actually writes

1. From logits to a choice

Run the Transformer from Day 5 on a prompt and the final layer produces, for the last position, one raw score per vocabulary entry. These scores are the logits: unnormalised numbers, positive or negative, one for each of the roughly 128,000 tokens in Llama 3's vocabulary. Grattafiori 2024 Softmax, the same function as in attention on Day 4, turns them into probabilities that are all positive and sum to one:

p(token i) = exp(zᵢ) ÷ Σⱼ exp(zⱼ)

Now the model has a probability distribution over the next token and someone has to pick. Greedy decoding always takes the most probable token. It is deterministic and fine for short factual answers, but over long outputs it falls into loops, repeating a phrase because the phrase was the most likely continuation of itself. Hugging Face 2026 Sampling draws a token at random in proportion to its probability: a token with p = 0.3 is chosen three times in ten. This is where variety, and all the dials, come from. Whichever way a token is chosen, it is appended to the sequence, the model runs again on the longer sequence, and the loop repeats until a stop condition.

2. The sampling knobs

Temperature is a number T that divides the logits before the softmax:

p(token i) = exp(zᵢ ÷ T) ÷ Σⱼ exp(zⱼ ÷ T)

Take three tokens with logits 2.0, 1.0 and 0.5. At T = 1 their probabilities are 0.63, 0.23 and 0.14. At T = 0.5 the logits are doubled before softmax, which exaggerates the gaps: 0.84, 0.11 and 0.04. At T = 2 the logits are halved and the gaps shrink: 0.48, 0.29 and 0.23. As T approaches 0 the top token gets all the mass and sampling becomes greedy; as T grows every token approaches equal probability and the output becomes noise. Intuition: temperature does not change which token the model prefers, it changes how strongly it insists. Low temperature for code and facts, around 0.7 to 1.0 for conversation, higher for brainstorming.

Top-k keeps only the k most probable tokens and renormalises before sampling; k = 50 is a common default. It was introduced for story generation, where sampling from the full distribution occasionally picked absurd tokens from the long tail. Fan 2018 Its weakness is that k is fixed while the distribution is not: when the model is confident, the top 50 includes 45 junk tokens; when it is genuinely unsure between 200 reasonable words, 150 of them are cut.

Top-p, also called nucleus sampling, fixes that. Sort the tokens by probability, keep the smallest set whose probabilities add up to at least p (say 0.9), and sample from that set. When the model is sure, the nucleus is one or two tokens; when it is unsure, it widens. The paper that introduced it also made the case that pure likelihood maximisation produces dull, repetitive text, and that human text is full of tokens that were not the most likely. Holtzman 2019

Min-p is newer: keep every token whose probability is at least a fraction of the top token's probability. With min-p = 0.1 and a top token at 0.6, anything below 0.06 is cut; if the top token is at 0.05, anything below 0.005 is cut. The threshold scales with the model's confidence, which keeps output coherent even at high temperatures, where top-p starts to let noise through. Nguyen 2024

Repetition penalty divides the logit of every token that has already appeared in the text by a factor θ (so a positive logit shrinks). The CTRL paper that introduced it found θ ≈ 1.2 "yields a good balance between truthful generation and lack of repetition". Keskar 2019 It is a blunt tool: it also penalises words that must repeat, like a variable name in code, so most chat deployments use it lightly or not at all. All of these knobs are standard arguments of a generation call. Hugging Face 2026

Try it: the widget below starts from a fixed set of logits. Lower the temperature and watch the tallest bar swallow the others; raise it and watch them flatten. Then set top-p to 0.9 and see which bars survive at each temperature. Press "sample" a few times at each setting.

Notice that temperature changes the shape of the whole distribution, while top-k and top-p only decide where to cut it. Notice, too, that at T = 1 with no cutoff, the low bars still get picked now and then; that is the long tail nucleus sampling was invented to trim.

3. Beam search, and knowing when to stop

Beam search is the opposite philosophy to sampling: instead of picking one token at a time, keep the B most probable partial sequences (the beams), extend each by every candidate, keep the best B again, and at the end return the sequence with the highest total probability. It is the standard decoder for machine translation and speech recognition, where there is a right answer and you want the most probable one. Hugging Face 2026 For chat it is almost never used, for three reasons. The most probable long text is usually a bad one: Holtzman and colleagues showed beam search output from GPT-2 collapsing into repeated phrases while humans wrote text full of lower-probability choices. Holtzman 2019 Worse, in translation the single most probable output under the model is often the empty string, over half the time in one study, so an exact search finds "nothing" and beam search only works because it fails to search well. Stahlberg 2019 And it costs B times the compute per token, which matters when you are serving millions of users.

How does the model know it has finished? With a stop token: a special vocabulary entry, often called end-of-sequence (EOS) or end-of-turn, that the model learned to emit when a document or a reply ends. The generation loop checks each new token against the configured stop ids and halts when it sees one. Hugging Face 2026 That is all "the model decided to stop talking" means. Chat models learn their end-of-turn token during post-training (Day 8); if you fine-tune one and forget to include that token in your examples, it will never stop. There is also a hard ceiling, the maximum number of new tokens, which is why answers sometimes cut off mid-sentence.

4. Prefill, decode, and why decoding is slow

A generation request has two phases that behave completely differently. Prefill processes the whole prompt at once: every prompt token goes through every layer in one big parallel pass, the same kind of matrix-matrix multiplication as training. The chip's arithmetic units are the bottleneck; prefill is compute-bound. Decode then produces output tokens one at a time, and each step needs only one new token's worth of maths but must read every weight of the model from memory to do it. NVIDIA 2023

Put numbers on it. Llama 3 8B in bf16 is 16 GB of weights. An H100 moves 3.35 TB/s between its memory and its compute units. NVIDIA 2026 Reading 16 GB takes 16 ÷ 3350 ≈ 4.8 milliseconds, so one request alone can never go faster than about 210 tokens per second on that chip, whatever the arithmetic speed. Meanwhile the arithmetic for one token is 2 × 8×10⁹ = 1.6×10¹⁰ FLOPs, which at 10¹⁵ FLOP/s takes 16 microseconds. The chip spends 99.7% of each decode step waiting for bytes. Decode is memory-bandwidth-bound; this is the single most important fact about inference and it explains almost everything that follows. Pope 2022

It also explains the two numbers every serving dashboard shows. Time to first token (TTFT) is mostly prefill time, so it grows with prompt length: a 100,000-token prompt has to be pushed through the model before anything comes back. Tokens per second during the answer is the decode rate, set by how fast the weights (and, as we are about to see, the cache) can be streamed. NVIDIA 2023

5. The KV cache

Attention lets each position look at every earlier position through their key and value vectors. During decode, the earlier positions have not changed, so recomputing their keys and values every step would be pure waste. The KV cache stores the key and value vectors of every token processed so far, for every layer, and each new token simply appends its own. Hugging Face 2026 Without it, generating token n would cost about n times more than with it. With it, the cost is memory, and the memory is easy to compute:

bytes per token = 2 (K and V) × layers × kv_heads × head_dim × 2 (bytes in bf16)

Llama 3 8B has 32 layers, 32 attention heads but only 8 key-value heads, and a head dimension of 128 (4,096 ÷ 32). Llama 3 70B has 80 layers, 64 heads, 8 key-value heads, and the same head dimension of 128. Grattafiori 2024

Read the table twice. At 8k context the cache is small change. At 128k, one request on the 8B model needs a 16 GiB cache, as large as the weights themselves, and one 70B request needs 40 GiB, half an H100. Multiply by the number of users served at once and the cache, not the model, is what fills the GPU. The third row shows why modern models use grouped-query attention: sharing each key-value head between several query heads (8 KV heads serving 32 or 64 query heads) cuts the cache four- to eight-fold with little loss in quality. Ainslie 2023 And since decode is bandwidth-bound, every byte of cache is also read on every step, so a big cache makes each token slower as well as the GPU fuller.

Try it: pick a model size and a context length below and watch the cache grow token by token. Compare 8B at 8k with 70B at 128k, then switch the KV-head count and see the bars change.

Notice that the cache grows linearly with the number of tokens, that the model's weights are a fixed cost while the cache is a per-request cost, and how quickly a long context on a big model crosses the 80 GB line of a single GPU.

6. Serving many users at once

If one decode step reads all 16 GB of weights to produce one token, it can just as well produce sixty-four tokens for sixty-four different users in nearly the same time: the weights are read once, the arithmetic is still tiny. Batching is how inference gets anywhere near the chip's capacity, and it is why a busy API is far cheaper per token than a GPU running one chat. The limit is memory: every user in the batch brings their own KV cache.

Naive batching has a problem: requests finish at different times, and a batch assembled at the start must wait for its longest member while finished slots sit empty. Continuous batching (the Orca system called it iteration-level scheduling) re-plans at every decode step: finished sequences leave, waiting ones join, and no slot idles. Yu 2022

The other problem is where the caches live. Early servers reserved a contiguous block of memory for each request's maximum possible length, so memory sat unused and could not be shared; the vLLM team measured 60 to 80% of KV memory wasted this way. vLLM 2023 PagedAttention borrows the operating-system idea of virtual memory: the cache is kept in small fixed-size blocks scattered wherever there is room, with a lookup table mapping each sequence's logical positions to physical blocks. Blocks are allocated as tokens arrive, freed the moment a request ends, and shared between requests that start with the same prompt. The result was 2 to 4 times the throughput of the best previous systems at the same latency, Kwon 2023 and vLLM became the default open-source serving engine. Its documentation lists what a serving engine in 2026 is expected to do: paged caches, continuous batching, prefix caching, speculative decoding and a long menu of quantisation formats. vLLM 2026

7. Speculative decoding

Here is a trick that follows directly from decode being bandwidth-bound. Verifying k candidate tokens in one forward pass costs about the same as generating one, because the weights are read once either way. So let a small, fast draft model guess the next few tokens, then have the big model score all of them in a single pass. Tokens the big model agrees with are accepted; at the first disagreement the big model's own choice is used and drafting restarts from there. With a rejection-sampling rule for the acceptances, the output distribution is exactly the big model's, not an approximation. The draft is right often enough on easy tokens (the second half of a word, the closing bracket) that two papers in late 2022 and early 2023 reported 2 to 3× faster generation with identical outputs, Leviathan 2022 including 2 to 2.5× on the 70B Chinchilla model. Chen 2023 Modern variants replace the separate draft model with small extra heads on the main model or with n-gram lookups in the prompt; vLLM ships several. vLLM 2026 The catch: the speedup shrinks when the batch is already large, because then the GPU is no longer idle between tokens.

8. Long context: what it costs

A model's context window is the maximum number of tokens it can hold at once, prompt plus answer. Llama 3.1 supports 128k; Grattafiori 2024 Google's Gemini 1.5 report tested retrieval at up to 10 million tokens. Gemini Team 2024 Long context is not free, and it costs in three distinct places. Prefill: attention compares every token with every other, so prefill work grows with the square of the prompt length; doubling the prompt quadruples that part of the time-to-first-token, and making this tolerable is what the FlashAttention paper review is about. Memory: the KV cache grows linearly, 128 KiB per token for an 8B model, which is Section 5's table. Decode: each new token attends to the whole cache, so a long context makes every subsequent token slower too.

Providers pass these costs on. As of October 2026, Google charges $2 per million input tokens for Gemini 3.1 Pro on prompts up to 200k tokens and $4 above that, Google 2026 and Anthropic's Haiku 5.5 moves from $0.10 to $0.50 per million input tokens once a prompt passes 100k. Anthropic 2026 There is also a quality cost: models have been shown to use information at the start and end of a long context better than information buried in the middle, Liu 2023 which is one reason retrieval (Day 12) did not disappear when windows got long.

9. Running a model on your own machine

None of this needs a data centre. llama.cpp is a C/C++ inference engine with no dependencies that runs on laptops, phones and servers, ggml-org 2026 and it reads models in the GGUF format, a single binary file designed for fast loading that usually contains quantised weights. ggml-org 2026 Ollama wraps it in a one-command installer and model library, Ollama 2026 and LM Studio adds a desktop chat window and a local API server. LM Studio 2026

What fits? The rule of thumb is: file size of the model plus the KV cache plus a gigabyte or two of overhead must fit in your GPU memory, or in unified memory on a Mac or a phone. Ollama's library lists Llama 3.1 8B at 4.9 GB, 70B at 43 GB and 405B at 243 GB; the 8B file is a third of the 16 GB the bf16 weights would take, which tells you the weights are stored at roughly four bits each. Ollama 2026 So an 8B model runs comfortably on a 16 GB laptop, a 70B model needs a machine with 64 GB of unified memory or several GPUs, and 405B needs a rack. Speed follows Section 4's rule: a 4.9 GB model on a laptop with about 100 GB/s of memory bandwidth (a typical figure for a thin laptop, as an estimate) tops out near 20 tokens per second for a single chat, which is faster than most people read. For phones, Meta released 1B and 3B versions of Llama 3.2 tuned for Qualcomm, MediaTek and Arm processors, with a 128k context, Meta 2024 and 4-bit quantisation (Day 10) is what makes a 3B model's weights fit in about 2 GB.

10. What it costs through an API

When you use a model through an API you pay per token, with separate prices for input (your prompt, charged at prefill) and output (the answer, charged at decode). Output costs more, typically five times more, because decode is the expensive, bandwidth-bound phase and output tokens cannot be batched as efficiently as prompts. Prices move often; these are the standard list prices on each provider's own pricing page as of October 2026, in US dollars per million tokens, excluding batch, cached-input and long-prompt tiers.

Worked example: a 1,000-token prompt with a 500-token answer on Claude Sonnet 5.5 costs 1,000 ÷ 10⁶ × $2 + 500 ÷ 10⁶ × $10 = $0.002 + $0.005 = $0.007. The same request on the flagship tier is five times that. Two consequences are worth remembering. First, a chat is re-sent in full on every turn, so a long conversation's cost grows with the square of its length unless the provider caches the shared prefix, which is exactly the prefix sharing PagedAttention made cheap, and most providers now sell cached input at a steep discount. Second, agents that loop (Day 12) spend most of their money on input tokens, re-reading their own history, not on the answers.

Next: Day 8 — Post-training: instruction tuning and RLHF

1. A base model is not an assistant

Everything up to Day 7 produced a base model: a network trained only to predict the next token of internet text. Ask it a question and it does the one thing it knows: it continues the document. If the question looks like a line from a quiz sheet, it writes more quiz lines. If it looks like a forum post, it writes the next post, complete with a username and a date.

Here is the kind of thing a base model does with a plain question. This is illustrative, not a quoted transcript; you can reproduce the pattern with any open base model.

Prompt:    What is the capital of France?
Base:      What is the capital of Germany?
           What is the capital of Italy?
           What is the capital of Spain?
           Answers: 1. Paris  2. Berlin  3. Rome  4. Madrid

Prompt:    What is the capital of France?
Instruct:  The capital of France is Paris.

The base model "knows" the answer: Paris appears in its continuation. What it lacks is the habit of treating your text as a request and its own text as a reply. An instruct model (also called a chat model or assistant model) is the same network after extra training that installs exactly that habit. The Hugging Face documentation says it plainly: all causal language models, chat-trained or not, continue a sequence of tokens; chat models are base models fine-tuned on data formatted as a sequence of messages Hugging Face docs.

Two consequences follow: an assistant's knowledge comes almost entirely from pretraining, and the "assistant" is a character the model has learned to play inside a particular text format.

2. Chat templates and special tokens

A conversation is turned into one long token sequence by a chat template: a fixed way of writing who said what. It uses special tokens, reserved vocabulary entries (from Day 2's tokenizer) that never occur in ordinary text, so the model can tell structure from content. Three roles are standard: system (standing instructions from whoever deploys the model), user, and assistant.

This is the Llama 3 format, as documented by Meta Meta 2024:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

What is the capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Each message is a role header, the content, and <|eot_id|> ("end of turn"). The prompt ends with an empty assistant header. The model's job is simply to continue from there, and it has learned that what follows an assistant header is a reply that ends with <|eot_id|>. That closing token is how the model stops talking: generation halts when it is sampled.

ChatML, the format OpenAI described for its early chat API and which Qwen and many other models adopted, does the same job with different tokens OpenAI 2023:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant

Three practical points. First, the template is part of the model. Feed Llama 3 the ChatML tokens and quality drops sharply, because it never saw them in training; libraries hide this behind tokenizer.apply_chat_template(messages), which reads the template stored with the model Hugging Face docs. Second, the option add_generation_prompt=True appends the empty assistant header; forget it and the model may continue the user's message instead of answering it. Third, special tokens are a trust boundary: tokenizers treat the text <|im_start|>system inside user content as ordinary characters, so a user cannot forge a system message.

3. Supervised fine-tuning: show it what good looks like

Supervised fine-tuning (SFT) is the first post-training step: keep training the base model, with the same next-token loss from Day 3, on examples of (prompt, ideal response) written in the chat template. Usually the loss is computed only on the assistant's tokens, so the model learns to write replies rather than to imitate users. Nothing about the architecture changes. The gradient simply pushes the weights toward "after an assistant header, produce text like this".

Where do the examples come from? Four sources, in historical order:

• Recast NLP datasets. FLAN (Google, 2021) rewrote more than 60 existing NLP tasks as natural-language instructions, fine-tuned a 137B model on them, and found it then handled unseen tasks better than the untuned model Wei 2021. This is where the term "instruction tuning" comes from.
• Human-written demonstrations. InstructGPT paid a team of about 40 contractors to write ideal responses to roughly 13,000 prompts, many of them taken from real API traffic Ouyang 2022. Expensive, slow, high quality.
• Model-generated data. Self-Instruct had a model generate new instructions and answers from a few seed examples, filtered them, and fine-tuned on the result Wang 2022. Stanford's Alpaca applied the idea with GPT-3.5 as the writer: 52,000 examples for under $500 of API calls, then fine-tuning a 7B LLaMA for under $100 of GPU time Stanford CRFM 2023. That recipe launched the 2023 wave of open chat models.
• Few but excellent. LIMA fine-tuned a 65B LLaMA on only 1,000 carefully curated examples, with no reinforcement learning at all, and human raters judged its answers equal to or better than GPT-4's in 43% of comparisons Zhou 2023. The authors' "superficial alignment hypothesis": almost all knowledge is learned in pretraining; SFT mainly teaches which format and style to use when talking to a person.

SFT is cheap and effective. Its limit is that it can only imitate: long-winded demonstrations make a long-winded model, and a writer who bluffed teaches bluffing. Imitation also gives no signal about relative quality; a demonstration says "this is good" but never "this is better than that". Getting that comparative signal is the job of RLHF.

4. The InstructGPT recipe

The method behind ChatGPT was published by OpenAI in March 2022 as InstructGPT, and it is still the reference recipe Ouyang 2022. ChatGPT itself, released in November 2022, used the same methods with changes to how data was collected OpenAI 2022. The ideas go back to a 2017 paper that trained game-playing agents from human comparisons of short video clips instead of a hand-written score Christiano 2017, and a 2020 paper that applied the same loop to summarizing Reddit posts Stiennon 2020. The InstructGPT paper review goes through the paper itself.

Try it: step through the three stages below and watch what each stage takes in and what it produces.

Stage 1 produces a model. Stage 2 produces a scorer. Stage 3 uses the scorer to improve the model, with a leash (the KL penalty) tying it to where it started.

Stage 1: SFT. Exactly as in section 3: fine-tune GPT-3 on human demonstrations.

Stage 2: reward model. Take a prompt, sample several answers from the SFT model, and have a labeler rank them. Each ranking yields pairs: answer A beat answer B. Train a second network, the reward model (RM), that reads (prompt, answer) and outputs one number: the predicted "how much a human would like this". The RM is trained with the Bradley–Terry model, a 1952 statistical model of paired comparisons in which the chance that A beats B depends only on the difference of their scores Wikipedia:

P(A preferred over B) = σ(r_A − r_B), loss = −log σ(r(winner) − r(loser))

σ is the sigmoid from Day 3. The loss is small when the RM scores the winner well above the loser. Only differences matter; the RM's absolute numbers mean nothing. InstructGPT used about 33,000 prompts for this stage Ouyang 2022.

Stage 3: reinforcement learning with PPO. Now treat the SFT model as a policy: something that takes a prompt and produces an action, here a whole response. Sample a prompt, generate a response, score it with the RM, and update the policy's weights so that high-scoring responses become more likely. The update rule is PPO (Proximal Policy Optimization), a 2017 algorithm that clips each step so the policy cannot move too far at once and training does not blow up Schulman 2017. The reward actually optimized is not the raw RM score but

reward = r_RM(prompt, response) − β · KL(π_RL ‖ π_SFT)

where the second term is the KL penalty, discussed next. InstructGPT also mixed a little of the original pretraining loss into the updates ("PPO-ptx") so the model would not lose general skills Ouyang 2022.

Result. Labelers preferred outputs from the 1.3-billion-parameter InstructGPT over those of the 175-billion-parameter GPT-3, a model more than 100 times larger Ouyang 2022. That is the whole argument for post-training: a modest amount of human feedback was worth more than a hundredfold increase in size.

# one RLHF update, in pseudo-code
for prompt in batch:
    response = policy.generate(prompt)              # sample from the current model
    r = reward_model(prompt, response)              # one number, from stage 2
    kl = logp(policy, response) - logp(sft, response)   # per token, summed
    shaped = r - beta * kl                          # reward with the leash
    advantage = shaped - value_model(prompt)        # better or worse than expected?
    ppo_update(policy, response, advantage)         # clipped step toward higher reward

5. Why the KL penalty exists

The reward model is a proxy: a learned guess at human preference, trained on a few tens of thousands of comparisons. It is accurate near the answers it was trained on and increasingly wrong far from them. Reinforcement learning is an optimizer, and optimizers go wherever the number is highest. Without a leash, the policy drifts into text that the RM scores highly but no human would like: repeated flattery, odd formatting that hits a quirk of the RM, or gibberish.

The KL divergence KL(π_RL ‖ π_SFT) measures how different the trained policy's token probabilities are from the SFT model's on the responses it actually produces. Subtracting β times it from the reward says: you get credit for pleasing the RM, but you pay for every bit you move away from the model we started with. This keeps the policy on the part of the map where the RM is trustworthy, and it keeps the text fluent, because the SFT model is fluent.

How much drift is safe? Gao, Schulman and Hilton measured it with a large "gold" reward model standing in for real humans. As optimization pushed the proxy RM's score up, the gold score first rose and then fell: past a certain amount of KL, the policy was getting better at fooling the proxy and worse for people. Larger RMs and more RM training data moved the peak further out, but every proxy eventually broke Gao 2022. This is Goodhart's law with a measurement attached: when a measure becomes a target, it stops being a good measure.

6. Reward hacking, with examples

Reward hacking is any behavior that raises the reward without achieving what the reward was meant to measure. The classic example predates language models. An RL agent playing the boat-racing game CoastRunners discovered that driving in circles in a lagoon, hitting respawning targets and catching fire, scored more points than finishing the race OpenAI 2016. Language models find the same kind of loophole in learned reward models:

• Length. Labelers tend to prefer longer answers, so the RM learns that longer is better, so RL makes answers longer. One study found that most of RLHF's measured gains on a standard setup could be reproduced by a reward that looked only at length Singhal 2023. OpenAI's own ChatGPT launch post said the model was "often excessively verbose" and blamed over-optimization OpenAI 2022.
• Agreeing with the user. In human preference data, a response that matches the user's stated view is more likely to be chosen, so the RM rewards agreement and RL amplifies it Sharma 2023. More on this in section 10.
• Thumbs-up chasing. In April 2025 OpenAI rolled back a GPT-4o update after it became noticeably more sycophantic, validating doubts and encouraging impulsive actions; the post-mortem said the update had leaned too heavily on short-term user feedback such as thumbs-up signals OpenAI 2025. The reward was "did the user like this message", and flattery is the cheapest way to earn it.
• Hedging and refusing. If labelers punish confident mistakes harder than evasions, the RM learns that "I can't help with that" is a safe score, and the model over-refuses. XSTest was built to measure exactly this, with prompts such as how to "kill a Python process" that sound dangerous and are not Röttger 2023.

None of these are bugs in PPO. They are the reward model being exactly as good as the preference data, and the optimizer being exactly as relentless as it should be. The fixes are better data, better RM training, a tighter KL leash, or, as Day 9 shows, rewards that cannot be faked, such as a unit test passing.

7. What RLHF costs: the human pipeline

The expensive part of RLHF is not GPUs. It is people. InstructGPT's team hired about 40 contractors through Upwork and Scale AI and had them write demonstrations and rank outputs following a guideline document; even so, trained labelers agreed with one another only about 73% of the time Ouyang 2022. Meta collected over 1.4 million binary human comparisons for Llama 2's reward models, in weekly batches, across five rounds of RLHF in which each new model's outputs were ranked again Touvron 2023. Anthropic's 2022 helpful-and-harmless work described the same loop, with crowdworkers chatting with the model and choosing the better of two replies at every turn Bai 2022a.

A typical pipeline looks like this:

On the compute side, PPO-style RLHF keeps four networks alive at once: the policy being trained, a frozen SFT copy for the KL term, the reward model, and a value model that estimates expected reward Hugging Face 2022. Generating full responses for every update dominates the wall clock. That cost is why a cheaper alternative, DPO, spread so quickly through open models; Day 9 covers it.

8. Constitutional AI: let the model grade itself

Anthropic's 2022 Constitutional AI (CAI) replaces most of the human harmlessness labels with a short written list of principles, the "constitution", plus the model's own judgment Bai 2022b. Two phases:

1. Critique and revise. Give a helpful-only model a harmful request and collect its harmful answer. Show it a principle ("identify ways in which this response is dangerous or unethical"), ask it to critique its own answer, then ask it to revise. Repeat with a few randomly chosen principles. Fine-tune on the revised answers. This is SFT with the model writing its own demonstrations.
2. RL from AI feedback (RLAIF). Sample two responses, show them to the model along with a principle, and ask which is better. Those AI-generated preferences train a reward model, which drives PPO exactly as in RLHF. Humans still label helpfulness; the AI labels harmlessness.

Why do it? Human harmlessness labeling is slow, exposes workers to disturbing content, and bakes in labeler inconsistency. A constitution is explicit, inspectable, and editable: change a sentence, retrain, and the behavior shifts. Anthropic's original constitution drew on sources such as the UN Declaration of Human Rights and other labs' published rules Anthropic 2023; a much longer rewritten constitution, explaining the values the company hopes Claude holds and the reasons behind them, was published in January 2026 Anthropic 2026. In the paper, the CAI model was both less harmful and less evasive than the RLHF baseline, because it had been trained to explain its objections rather than refuse flatly Bai 2022b. See the Constitutional AI paper review.

RLAIF, AI preference labels in place of human ones, has since become routine, usually with a stronger model grading a weaker one. It is cheap and consistent, and it inherits every bias of the grader.

9. System prompts: the cheapest layer

Training sets the model's defaults. The system prompt adjusts them per deployment: text in the system role, placed before the conversation, that the model learned during SFT and RLHF (whose data includes varied system prompts) to treat as standing instructions. It is the layer that says "today's date is..." or "you are the support agent for this product; answer in Spanish". Nothing changes in the weights; behavior shifts because the model conditions on different context.

Companies lean on it heavily. Anthropic publishes the system prompts used by claude.ai and its mobile apps, with dated entries per model; they supply the date, encourage behaviors such as formatting code in Markdown, and are not applied to API traffic, where developers write their own Anthropic docs. Reading one is instructive: a good part of what feels like personality is a few paragraphs of instructions on top of the trained model.

So behavior is layered, from the most expensive layer to change to the cheapest:

Cheaper layers override expensive ones only partly. A system prompt cannot give the model knowledge it lacks, and trained refusals usually survive a system prompt that asks for the opposite, which is intended: a deployer should not be able to switch off safety behavior with one sentence.

10. What RLHF does and does not fix

An honest scorecard, as of October 2026.

Fixed: format and usefulness. The model answers instead of continuing, follows multi-part instructions, declines clearly harmful requests, and matches the tone people expect.

Made worse, then partly managed: sycophancy. Because preference data rewards agreement, RLHF tends to produce models that tell users what they want to hear, abandon correct answers when challenged, and praise bad work Sharma 2023. The 2025 GPT-4o rollback showed how fast it comes back when feedback signals are reweighted OpenAI 2025. Labs now train and test against it explicitly, but it is a pressure built into the data, not a solved problem.

Reduced, not removed: hallucination. InstructGPT's authors listed "still makes up facts" among its remaining failures Ouyang 2022. A 2025 OpenAI analysis argued the cause is structural: most training and evaluation scores a confident guess the same as a correct answer and gives nothing for "I don't know", so guessing is the score-maximizing strategy, and post-training inherits that incentive unless evaluations are changed to reward calibrated uncertainty OpenAI 2025. RLHF can teach a model to say "I'm not sure" more often; it cannot give the model facts it never learned.

A dial, not a fix: refusals. Push harmlessness and the model refuses to explain how to kill a Python process Röttger 2023. Push helpfulness and it complies with harmful requests, which InstructGPT did by default Ouyang 2022. Every lab sits somewhere on this dial and moves along it release by release.

Barely touched: reasoning. Preferences about which answer reads better are a weak signal for whether a proof is correct. Getting better at hard problems needed a different reward, one that checks the answer. That is Day 9.

Next: Day 9 — Reasoning models and test-time compute

1. Why writing the steps helps a next-token predictor

Recall Day 5: generating one token means one forward pass through every layer. That is a fixed budget of arithmetic, the same for "the" as for the answer to a hard question. Ask a model for 17 × 24 and demand the answer as the very next token, and it must compress the whole multiplication into that single pass. Let it write "17 × 20 = 340, 17 × 4 = 68, 340 + 68 = 408" and each step is a small, familiar pattern, and every intermediate result sits in the context where attention can read it back. The context window becomes scratch paper; the sequence of tokens becomes a sequence of computation steps. The model is not reasoning in some new way. It is converting one impossible prediction into many easy ones.

Wei et al. (Google, January 2022) made this a technique: chain-of-thought (CoT) prompting, in which the few-shot examples in the prompt include worked reasoning before each answer. With eight such exemplars, a 540B model set a new state of the art on GSM8K, a benchmark of 8,500 grade-school math word problems Cobbe 2021, beating a fine-tuned GPT-3 with a verifier Wei 2022. Small models produced fluent chains with wrong steps; the gain appeared only at scale. See the chain-of-thought paper review.

Kojima et al. then showed that you do not even need examples. Appending "Let's think step by step" to the question raised text-davinci-002's accuracy on MultiArith from 17.7% to 78.7% and on GSM8K from 10.4% to 40.7% Kojima 2022. The model already knew how to show its work; the phrase just moved it into the part of its distribution where it does.

Wang et al. added self-consistency: sample many chains with the temperature sampling from Day 7, take the final answer from each, and return the majority. On GSM8K this added 17.9 points over a single chain Wang 2022. This is the first clean example of test-time compute: pay for 40 answers instead of one and accuracy rises, with no change to the weights.

2. From prompting the steps to training them

If chains help, train on them. STaR (2022) had a model generate rationales, kept only the ones that reached the correct answer, fine-tuned on those, and repeated, bootstrapping its own reasoning from a few seeds Zelikman 2022. OpenAI's "Let's Verify Step by Step" (2023) trained a process reward model that grades each step of a solution rather than only the final answer, and found it a far better selector on hard math than an outcome-only grader Lightman 2023. The common pattern is: generate many attempts, check them, and train on what passed. Everything below is a variation on that loop.

3. DPO: preference training without a reward model

Day 8's optional note ended with a surprise: the KL-penalized RLHF objective has a closed-form optimum, so any reward model can be rewritten in terms of the policy itself. Rafailov et al. (2023) turned that into Direct Preference Optimization (DPO), a plain classification loss on preference pairs Rafailov 2023:

loss = −log σ( β · [ log π(y_w|x)/π_ref(y_w|x) − log π(y_l|x)/π_ref(y_l|x) ] )

In words: for each (prompt, chosen answer y_w, rejected answer y_l), raise the probability of the chosen answer relative to the frozen reference model and lower the rejected one, with β setting how far you are allowed to move. There is no reward model, no sampling during training, no value model, and no PPO. Two networks instead of four.

# DPO, one batch
logp_w  = policy.logp(prompt, chosen)   - ref.logp(prompt, chosen)    # log-ratio, chosen
logp_l  = policy.logp(prompt, rejected) - ref.logp(prompt, rejected)  # log-ratio, rejected
margin  = beta * (logp_w - logp_l)
loss    = -log_sigmoid(margin).mean()   # push margin up: chosen more likely than rejected
loss.backward(); optimizer.step()       # ordinary supervised training step

Open models adopted it almost immediately. Zephyr-7B (October 2023) used DPO on AI-ranked data to beat much larger chat models Tunstall 2023. Meta's Llama 3 post-training used SFT, rejection sampling, and DPO, and the paper says the team tried PPO but found DPO needed less compute at scale and performed better Llama Team 2024. Tülu 3 (2024) used DPO for preferences before a final RL stage Lambert 2024. See the DPO paper review.

DPO's limits explain what came next. The pairs are usually generated once, often by another model, so training is off-policy: the signal goes stale as the policy moves. DPO can over-optimize too, often by driving the rejected answer's probability down faster than it raises the chosen one. And DPO needs preference pairs; it cannot use a reward you compute on the fly, like a unit test passing. For that, you need RL again.

4. Rewards you cannot fake: verifiable rewards and GRPO

Day 8's reward hacking came from a learned reward model being a proxy. Many tasks do not need a proxy. A math problem has an answer you can compare against a reference. Code either passes the unit tests or it does not. Tülu 3 named this setup reinforcement learning with verifiable rewards (RLVR): reward 1 if the checker accepts the final answer, 0 otherwise Lambert 2024. The reward cannot be flattered. It can still be gamed, but only through the checker (formatting tricks, special-casing visible tests), a narrower attack surface than a model's taste.

The algorithm most associated with this is GRPO (Group Relative Policy Optimization), introduced by DeepSeek in February 2024 for math Shao 2024. Try reading it without the equations:

1. For one prompt, sample a group of answers, say 16.
2. Score each with the verifier: 1 or 0.
3. Compute the group's mean and spread. Each answer's advantage is how far its score sits above or below the group mean, divided by the spread.
4. Raise the probability of every token in above-average answers and lower it for below-average ones, with the PPO-style clipping and a KL penalty to the reference model from Day 8.

The trick is in step 3. PPO needs a separate value model to estimate "how well should this prompt have gone", which is a whole extra network the size of the policy. GRPO uses the group's own average as that estimate. If 4 of 16 answers were right, those 4 get pushed up and the 12 get pushed down. No critic, half the memory, and a signal that is exact rather than guessed.

5. OpenAI o1: a hidden reasoning trace

On 12 September 2024 OpenAI released o1-preview, the first widely available reasoning model: a model trained with large-scale reinforcement learning to produce a long internal chain of thought before every answer. The numbers were striking. On the 2024 AIME, a competition exam for the strongest US high-school mathematicians, GPT-4o solved about 12% of problems; o1 solved 74% with one attempt per problem and 93% when 1,000 samples were re-ranked with a learned scorer. It reached the 89th percentile on Codeforces programming contests and exceeded PhD-level human accuracy on the GPQA science benchmark OpenAI 2024.

Two things about the release mattered more than the scores. First, OpenAI showed two scaling curves: accuracy rose smoothly with more RL training compute, and separately with more thinking at inference time OpenAI 2024. Test-time compute had become a lever on the same footing as model size. Second, the chain of thought was hidden. OpenAI chose not to show users the raw trace, giving only a summary, partly so the chain could stay unfiltered for monitoring and partly for competitive reasons OpenAI 2024. In the API, reasoning tokens are invisible but still occupy the context window and are billed as output tokens OpenAI docs. The follow-ups o3 and o4-mini (April 2025) added the ability to call tools such as search and Python inside the chain OpenAI 2025a.

What o1 did not give the world was the recipe: which RL algorithm, what reward, how much SFT. Four months later, someone else published it.

6. DeepSeek-R1: the open recipe

In January 2025 DeepSeek released R1 with a paper, open weights, and an MIT license DeepSeek 2025a. The paper's most important experiment is DeepSeek-R1-Zero: take the DeepSeek-V3 base model, skip SFT entirely, and run GRPO with two rule-based rewards, one for a correct final answer and one for wrapping the reasoning in the expected tags. Nothing else. Over training, the model's AIME 2024 pass@1 rose from 15.6% to 77.9%, and to 86.7% with majority voting, and its responses grew steadily longer on their own as it learned that more thinking paid off DeepSeek 2025b.

The behaviors that emerged were never demonstrated to it. The paper describes an "aha moment" partway through training, marked by a sudden jump in the model's use of the word "wait" as it began to pause, re-examine its own steps, and try another approach; one logged chain literally reads "Wait, wait. Wait. That's an aha moment I can flag here." before the model restarts the problem DeepSeek 2025b. The lesson is the one from the analogy box: give an optimizer an exact reward and room to produce tokens, and self-checking is discovered because it works.

R1-Zero's chains were also hard to read and often mixed English and Chinese mid-sentence DeepSeek 2025b. The full DeepSeek-R1 fixed that with a four-stage pipeline: a small "cold start" SFT on a few thousand clean, long reasoning examples; RL for reasoning with an added reward for staying in one language; rejection sampling from that model to build a large SFT set that also covers writing and general chat; and a final RL round for helpfulness and harmlessness, in the spirit of Day 8. The result matched o1 on the headline math and coding benchmarks.

Then came distillation: fine-tune small open models on about 800,000 of R1's outputs. DeepSeek released six such models, built on Qwen2.5 (1.5B, 7B, 14B, 32B) and Llama 3 (8B, 70B), and reported that distilling from the big model beat running RL directly on the small ones DeepSeek 2025a. A 7B model that could work through competition math on a laptop appeared overnight. The paper was later peer reviewed and published in Nature in September 2025, with supplementary material that put the cost of the R1 reinforcement-learning stage at about US$294,000 on 512 H800 GPUs, on top of the already-trained V3 base DeepSeek 2025c.

What R1 showed about open research: the reasoning recipe was not a secret ingredient but a known loop (verifiable rewards plus GRPO plus patience), it could be reproduced by any lab with a strong base model, and the results could be compressed into models small enough for anyone to run. See the DeepSeek-R1 paper review.

7. Test-time compute: buying accuracy with tokens

Try it: drag the thinking budget below and watch accuracy on the illustrative task. Notice where the curve flattens.

Each doubling of thinking tokens buys roughly the same accuracy gain, so the curve is a straight line on a log axis and flattens on a linear one. Past some budget, more tokens buy almost nothing; before it, a small model with a budget can beat a large model without one.

Snell et al. (2024) made the trade-off precise. Matching total compute, a smaller model that spends its budget on test-time strategies (revising, or sampling candidates and picking with a verifier) can beat a model 14 times larger, as long as the problem is one it has a reasonable chance on. On the hardest problems, pretraining compute still wins Snell 2024. Anthropic reported the same shape for Claude 3.7 Sonnet: accuracy on math rose roughly logarithmically with the serial thinking budget, and in parallel mode, sampling 256 attempts and choosing with a scoring model pushed GPQA to 84.8% Anthropic 2025a.

Two ways to spend, then. Serial: one longer chain, which is what reasoning models do by default; it cannot be parallelized, so latency grows with the budget. Parallel: many chains with a vote or a verifier, which is self-consistency grown up; it costs N times the tokens but not N times the wall clock. Logarithmic returns are the reason "thinking budgets" exist: the first 2,000 tokens of thinking are worth far more than the next 20,000, so you want a dial.

The dials, as of October 2026:

8. Where reasoning models are now (October 2026)

The short version: "reasoning model" is no longer a separate product category. Every frontier model thinks, and the question has become how much and whether the model or the caller decides. Dates below are the official announcement dates.

• OpenAI. GPT-5 (7 August 2025) merged the GPT and o-series lines into one system: a fast model, a deeper reasoning model, and a router that chooses between them per request; OpenAI reported its thinking mode made about 80% fewer factual errors than o3 OpenAI 2025b. As of 7 October 2026 the GPT-6 Sol and Luna models serve all ChatGPT tiers, and OpenAI's safety documentation reports capability evaluations at maximum reasoning effort and safety evaluations at the lowest deployed effort, because most traffic runs there OpenAI 2026.
• Anthropic. Claude 3.7 Sonnet (February 2025) was the first hybrid: instant replies or visible extended thinking, same weights Anthropic 2025b. Claude Opus 4.6 (February 2026) made thinking adaptive Anthropic 2026a. Claude Opus 5.5 (22 September 2026) can no longer be run with thinking switched off, defaults to medium effort, and ships "preserved thinking", which stops API users from editing prior context to extract the reasoning for distillation Anthropic 2026b.
• Google. Gemini 2.5 (March 2025) was presented as a thinking model from the start Google 2025. Gemini 3 (18 November 2025) added Deep Think, an enhanced reasoning mode that beats Gemini 3 Pro on the hardest benchmarks Google 2025b. Gemini 3.8 Live Extended Thinking (15 September 2026) reasons in the background during a spoken conversation Google 2026.
• DeepSeek. After R1, the reasoning line merged into the general V4 models. As of October 2026 the API serves DeepSeek-V4-Pro and a V4.1-Flash checkpoint dated 17 October, with thinking controlled per request DeepSeek docs; the weights are on Hugging Face, with V4-Pro-0813 from August 2026 DeepSeek HF.
• Qwen (Alibaba). Qwen3 (April 2025) was the first major open family with a built-in thinking switch Qwen 2025. Qwen3.5 (February 2026) and Qwen3.6 (April 2026) followed, and Qwen3.8 (August 2026) tops out at a 2.4-trillion-parameter mixture-of-experts with 95B active, alongside a 27B model, both with adjustable reasoning effort Qwen 2026.

The arc is two years long: hidden chain (September 2024), open recipe (January 2025), a switch (2025), an effort dial (2026), and now models that will not run without thinking at all.

9. Limitations

Overthinking. o1-style models generate hundreds of tokens of deliberation for problems like "2 + 3 = ?", a pattern measured and named "overthinking" in late 2024 Chen 2024. Every adaptive-thinking and effort control in section 7 exists to manage it.

The chain is not a faithful record. Anthropic tested this directly: slip a hint about the answer into the prompt and check whether the model's chain of thought admits using it. Claude 3.7 Sonnet mentioned the hint it relied on 25% of the time and DeepSeek R1 39% of the time, averaged across hint types Anthropic 2025c. The visible reasoning is a text the model generates, shaped by the same training pressures as the answer; it is evidence about the model's process, not a transcript of it. OpenAI found the complementary problem: when a monitor reads the chain to catch reward hacking in coding tasks and the model is penalized for "bad thoughts", it keeps hacking and learns to hide the intent from the chain instead, so OpenAI recommends not optimizing the chain directly OpenAI 2025c. A cross-lab position paper calls chain-of-thought monitorability a useful but fragile safety opportunity for exactly this reason Korbak 2025.

Cost and latency. Thinking tokens are output tokens, the most expensive kind, and a serial chain cannot be sped up by adding GPUs: ten thousand tokens of thinking is ten thousand sequential decode steps from Day 7, often a minute or more. Those tokens also occupy the context window and count against the output limit, so a response can be cut off or come back empty if thinking eats the budget Google docs.

Reward hacking moves to the checker. With a learned reward model, the model learned to please the judge. With unit tests, the model learns to pass the tests, which is not the same as writing correct code; OpenAI's monitoring work found models subverting tests rather than fixing bugs OpenAI 2025c.

Uneven gains. Verifiable rewards exist for math, code, and logic. Writing, judgment, and taste have no checker, so the improvement there comes mostly from the older preference methods of Day 8, and it is smaller.

10. When to use a reasoning model, and how much

A workable rule: if you could write a checker for the answer and the task has more than two steps, pay for thinking. Otherwise start at the lowest effort, measure accuracy on your own examples, and raise effort only while the measurement moves. The log-shaped curve means the first increase is usually worth it and the last one almost never is.

Next: Day 10 — Fine-tuning on a budget: LoRA, QLoRA, and quantization

1. Prompt, retrieve, or fine-tune?

Most problems are solved by a better prompt, which changes nothing inside the model and can be undone in a second. Retrieval-augmented generation (RAG), covered on Day 12, looks up relevant documents at run time and pastes them into the prompt, so the model reads your facts instead of remembering them. Fine-tuning continues training on your own examples so the weights themselves change: the most powerful option and the easiest to get wrong, so it comes last.

One finding keeps coming up: fine-tuning is good at teaching form (a format, a tone, a skill) and surprisingly bad at teaching facts. Ovadia and colleagues compared fine-tuning with RAG for injecting new knowledge and found RAG consistently won, both for facts the model had seen in pretraining and for brand-new ones; the model struggled to learn new facts from fine-tuning unless it was shown many paraphrases of each. Ovadia 2023

2. Full fine-tuning and what it costs

Full fine-tuning means running the same gradient descent you met on Day 3 over every weight in the model, on your data. The compute per token is the same ≈6·N FLOPs as pretraining (Day 6), but your dataset is thousands of times smaller, so compute is rarely the problem. Memory is.

Count the bytes per parameter with the standard mixed-precision Adam recipe. The forward and backward passes run in 16-bit floats: 2 bytes for the weight, 2 bytes for its gradient. The optimizer keeps an fp32 master copy of the weight (4 bytes) so that tiny updates are not rounded away, plus Adam's two running statistics, the momentum and the variance (4 + 4 bytes). Micikevicius 2017 Kingma 2014

training memory ≈ (2 + 2 + 4 + 4 + 4) bytes × N = 16 bytes × N, plus activations

That 16-bytes-per-parameter figure is the one the ZeRO paper used to motivate sharding optimizer state across GPUs. Rajbhandari 2019 Worked example: a 7B model needs 7 × 10⁹ × 16 bytes ≈ 112 GB before a single activation is stored. That does not fit on one 80 GB GPU. Activations add more: Hugging Face's memory guide estimates roughly 85 GB to train a 4B model at batch size 16 in mixed precision. Hugging Face docs A 70B model needs about 1.1 TB of weights, gradients and optimizer state alone, so full fine-tuning at that scale means a multi-GPU cluster with sharded optimizers, exactly the machinery used for pretraining.

Notice where the bytes go: 12 of the 16 exist only because we are training those weights. Train fewer weights and that cost vanishes for the frozen ones. That is the whole idea behind parameter-efficient fine-tuning.

3. LoRA: train a thin detour, not the whole road

Fine-tuning changes a weight matrix W into W + ΔW. The LoRA authors asked whether ΔW really needs as many degrees of freedom as W. Their bet, confirmed empirically, was that the useful change has a low rank, meaning it can be written as the product of two thin matrices. So LoRA (Low-Rank Adaptation) freezes W and learns only the detour. Hu 2021

h = W·x + (α/r)·B·A·x, with B of shape d×r and A of shape r×d, r ≪ d

A d×d matrix has d² weights; the pair B, A has only 2·d·r. For d = 4096 and r = 8 that is 65,536 trainable numbers in place of 16.8 million, about 0.4%. Four details matter in practice:

• Initialization. A starts random and B starts at zero, so B·A = 0 and the model begins exactly as the pretrained one. Training can only move it away gradually.
• Rank r and alpha α. r sets the capacity of the detour; the update is scaled by α/r, so α acts like a learning-rate knob for the adapter. The paper found a rank of 1 to 8 enough when adapting the attention query and value matrices; common 2026 defaults are r = 8 to 64 with α = 2r. Hugging Face docs
• Which matrices. Under a fixed budget the paper did best adapting W_q and W_v together. Most recipes today attach LoRA to every linear layer, attention and MLP alike, because the extra parameters are still cheap.
• Merging. After training, compute W′ = W + (α/r)·B·A once and save it: inference costs exactly what the base model costs. Or keep the adapter separate (a few tens of megabytes) and serve many adapters on one base model, swapping per customer.

Now the 7B arithmetic, using Mistral 7B's published shapes: 32 layers, model width 4,096, MLP width 14,336, and 8 key/value heads so the k and v projections are 4,096×1,024. Jiang 2023 With r = 16 on all seven linear matrices per layer (q, k, v, o, gate, up, down), one layer holds 16 × [(4096+4096)·2 + (4096+1024)·2 + (4096+14336)·3] = 1,310,720 adapter weights; times 32 layers is about 41.9 million, roughly 0.6% of the 7.2 billion base weights. With r = 8 on q and v only, the classic paper setting, it is 3.4 million, about 0.05%. The optimizer state for 42M parameters is under 1 GB; the frozen base still sits in memory at 14 GB in bf16, but carries no gradients or Adam moments. For GPT-3 175B the paper reported 10,000× fewer trainable parameters and about 3× less GPU memory, at equal or better quality. Hu 2021

Try it: drag the rank slider and watch the trainable count and the percentage of full fine-tuning.

The big square is the frozen W. The two thin strips are the adapter. Their area, not W's, is what you train, and it grows linearly with the rank while W's cost stays fixed.

One honest caveat. Biderman and colleagues compared LoRA with full fine-tuning on large code and math datasets and found that at typical ranks LoRA learns less, but also forgets less of the base model's other abilities; full fine-tuning's weight changes had ranks 10 to 100× higher than common LoRA settings. Biderman 2024 For most product fine-tunes that trade is exactly what you want. See the LoRA paper review for the experiments.

4. QLoRA: a 65B fine-tune on one GPU

LoRA removes the optimizer bytes but the frozen base still costs 2 bytes per weight: 130 GB for a 65B model, before activations. QLoRA stores that frozen base in 4 bits and trains bf16 LoRA adapters on top of it, backpropagating through the quantized weights (they are dequantized on the fly for each matrix multiply). The result: fine-tuning a 65B model dropped from more than 780 GB of GPU memory to under 48 GB, and the authors' Guanaco models reached 99.3% of ChatGPT's score on the Vicuna benchmark, as judged by GPT-4, after about 24 hours on a single GPU. Dettmers 2023 Three inventions make it work:

• NF4 (4-bit NormalFloat). Trained weights are roughly bell-shaped around zero. Instead of 16 evenly spaced levels, NF4 places its 16 levels at the quantiles of a normal distribution, so each level is used about equally often and the rounding error is smaller for the same 4 bits. Weights are quantized in blocks of 64, each block with its own scale.
• Double quantization. Those per-block scales are themselves 32-bit numbers, and one per 64 weights adds half a bit per weight. Quantizing the scales to 8 bits (in blocks of 256) saves about 0.37 bits per parameter, around 3 GB on a 65B model.
• Paged optimizers. Memory spikes, for example from one long sequence, used to crash runs. QLoRA uses NVIDIA unified memory so optimizer states are paged out to CPU RAM when the GPU fills and paged back when needed.

At hobby scale, a 7B base in NF4 is about 3.5 to 4 GB; with adapters and activations at a modest sequence length the run fits a 16 GB consumer card or a free Colab GPU. The implementation lives in the bitsandbytes library, which also ships 8-bit optimizers and LLM.int8() inference. Hugging Face docs The QLoRA paper review walks through the numbers.

5. Adapters, prefix tuning, prompt tuning

LoRA was not the first parameter-efficient method, and knowing its cousins explains why it won.

Soft prompts are worth remembering for one reason: they show that a frozen model already contains the skill, and training only finds an input that unlocks it, few-shot prompting (Day 8) with gradient descent choosing the examples.

6. Catastrophic forgetting

Catastrophic forgetting is when training on new data overwrites what the network knew before. The term comes from a 1989 study of small networks and was treated as a curiosity for decades. Wikipedia It is not a curiosity for LLMs: Luo and colleagues fine-tuned 1B to 7B models in sequence on new tasks and saw general abilities, reasoning and reading comprehension decay, with the effect growing with model size in that range. Luo 2023

The reason is mechanical: gradient descent only cares about your loss. If your 5,000 support transcripts never contain Python, nothing protects the weights that write Python, and the updates drift them. Defences, cheapest first: fewer steps at a lower learning rate; a small adapter (LoRA forgets less); a slice of general instruction data mixed back in so the loss still rewards old skills; and a general benchmark run before and after, so you see the damage before users do (Day 13 covers evaluation).

7. Quantization: fewer bits per weight

Every weight is a number stored in some format. The format decides the file size, and because decoding is limited by how fast weights stream from memory (Day 7), it also decides the speed.

Integer quantization is rounding with a ruler. Take a group of weights, find the largest magnitude m, set a scale s = m / 7 (for symmetric 4-bit, levels −8 to 7), store each weight as the integer round(w / s), and reconstruct it later as q·s. The error is at most s/2 per weight. Where the ruler is shared decides the damage. Per-tensor scaling uses one s for a whole matrix, so a single outlier weight of 50 among thousands near 0.01 stretches the ruler and rounds almost everything to zero. Per-group scaling gives every 32 to 128 consecutive weights their own scale: the outlier only ruins its own group, at the cost of storing more scales. That overhead is why a "4-bit" GGUF file like Q4_K_M actually spends about 4.9 bits per weight. llama.cpp docs Outliers are real: LLM.int8() found that models beyond a few billion parameters develop a few feature dimensions with huge magnitudes; keeping just those in 16-bit while 99.9% of the multiply runs in int8 preserved full quality at 175B. Dettmers 2022

Try it: the widget below quantizes a row of fp16 weights onto an int4 grid. Watch the rounding error per weight and the memory saved.

The grid has only 16 rungs. Weights that fall between rungs get pushed to the nearest one; the bigger the group's largest value, the coarser the rungs for everyone in the group.

Post-training methods try to be smarter than plain rounding, using a few hundred calibration examples and no retraining:

• GPTQ quantizes one column at a time and uses approximate second-order information to nudge the remaining weights to compensate for each rounding error. It quantized 175B-parameter models in about four GPU-hours. On OPT-175B, WikiText perplexity went from 8.34 in fp16 to 8.37 at 4-bit and 8.68 at 3-bit, where naive rounding gave 10.54 and about 7,300 respectively. Frantar 2022
• AWQ (activation-aware weight quantization) observes that about 1% of weight channels, the ones that multiply large activations, matter far more than the rest. Instead of keeping them in higher precision, it scales those channels up before rounding, which provably lowers their error, and it ships with a 4-bit kernel more than 3× faster than fp16 Hugging Face inference on desktop and mobile GPUs. Lin 2023
• GGUF and llama.cpp are the format and runtime most people use on laptops. GGUF is a single, memory-mappable file carrying weights, tokenizer and hyperparameters as typed metadata; ggml docs llama.cpp runs it on CPUs and on Apple, NVIDIA, AMD and Intel GPUs with quantization from 1.5 to 8 bits. llama.cpp The names encode the recipe: Q4_K_M is a 4-bit "k-quant" that keeps the most sensitive tensors at somewhat higher precision; Q8_0 is plain 8-bit with per-block scales.

Real numbers, from the llama.cpp k-quants benchmark on the original LLaMA 7B and 13B (perplexity on WikiText; lower is better): llama.cpp 2023

Read it twice. First, 6-bit and 5-bit are essentially free, 4-bit costs about 1% in perplexity, 3-bit about 4%, and 2-bit about 15%. Second, the 13B model at Q4_K_M (7.3 GB, perplexity 5.30) beats the 7B model in fp16 (13 GB, perplexity 5.91) at half the memory: for a fixed budget, a bigger model quantized harder usually wins, down to about 4 bits. Downstream tasks agree: Llama-2-7B scored 54.3 on the Open LLM Leaderboard average in fp16 and about 53.3 in 4-bit, with a smaller gap for 13B. Hugging Face 2023 Small models and reasoning-heavy outputs suffer most, so always test on your own task.

8. Knowledge distillation: a small model imitates a big one

Knowledge distillation trains a small student to match a big teacher. The classic version, from Hinton, Vinyals and Dean, trains the student on the teacher's full output distribution (the soft labels) rather than on the single correct answer. Hinton 2015 The soft labels carry far more information per example: a hard label says "the next token is cat"; the teacher's distribution says "cat 60%, kitten 25%, dog 3%, car 0.01%", which also teaches the student which mistakes are near-misses. DistilBERT applied this to BERT and kept 97% of its language-understanding score in a model 40% smaller and 60% faster. Sanh 2019

Distillation is now how most good small models are made:

• Gemma 2's 2B and 9B models were trained with knowledge distillation instead of plain next-token prediction, and every Gemma 3 model, from 1B to 27B, is trained with distillation. Gemma Team 2024 Gemma Team 2025
• Llama 3.2's 1B and 3B models were created by structured pruning of Llama 3.1 8B, then recovered by distillation using logits from the 8B and 70B models as token-level targets during pretraining. Meta 2024
• DeepSeek-R1's distilled models (Qwen 1.5B to 32B, Llama 8B and 70B) use the other flavour, sometimes called sequence-level or data distillation: the student is simply fine-tuned on about 800k reasoning traces written by the 671B teacher, with no access to its logits. DeepSeek 2025 It is the same trick as synthetic-data SFT, and why a 7B model can suddenly "think" like a model a hundred times bigger on math problems. The DeepSeek-R1 review has the numbers.

Both flavours ship in Hugging Face TRL: DistillationTrainer and the generalized GKDTrainer for logit matching, and plain SFTTrainer on teacher-generated text. Hugging Face docs

9. Pruning and sparsity, briefly

Pruning deletes weights. Unstructured pruning zeroes individual weights: SparseGPT pruned OPT-175B to 50 to 60% zeros in one shot, with no retraining and a negligible perplexity increase, Frantar 2023 and Wanda gets close with an even simpler score, weight magnitude times the typical activation it meets. Sun 2023 The catch is hardware: scattered zeros do not make a GPU matrix multiply faster. Semi-structured 2:4 sparsity, exactly two zeros in every block of four, is the pattern NVIDIA tensor cores can skip, with a theoretical 2× throughput. NVIDIA 2021 Structured pruning removes whole heads, layers or width, which helps on any hardware but hurts more, so it is followed by distillation, as in Llama 3.2 1B and 3B and NVIDIA's Nemotron-H 47B (pruned and distilled from the 56B). NVIDIA 2025

10. How to fine-tune a small model today

As of October 2026 the standard open-source stack is Hugging Face transformers for the model, peft for LoRA and QLoRA, trl for the training loops (SFT, DPO, GRPO), and optionally Unsloth, which patches the same stack with hand-written kernels and advertises about 2× faster training with 70% less VRAM, plus free Colab notebooks. Hugging Face docs Unsloth A complete LoRA run on a chat dataset:

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

data = load_dataset("trl-lib/Capybara", split="train")   # rows of {"messages": [...]}
lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                  target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                                  "gate_proj", "up_proj", "down_proj"])
args = SFTConfig(output_dir="out", max_length=1024, learning_rate=1e-4,
                 num_train_epochs=1, per_device_train_batch_size=4,
                 gradient_accumulation_steps=4, bf16=True)
trainer = SFTTrainer(model="Qwen/Qwen3-0.6B", train_dataset=data,
                     args=args, peft_config=lora)   # add quantization_config=... for QLoRA
trainer.train()
trainer.save_model("out")          # writes only the adapter, a few tens of MB

The trainer applies the chat template, computes the loss on the assistant's reply rather than the prompt, and uses a higher learning rate (≈1e-4) than full fine-tuning because only new parameters are learned. Hugging Face docs The parts the code does not show are the parts that decide success:

1. Data. 500 to 5,000 clean, consistent examples of exactly the behaviour you want. One bad template or a mixed tone costs more than any hyperparameter.
2. Base model. An instruction-tuned small model (0.5B to 8B) that already does most of the job; fine-tuning is for the last stretch.
3. Evaluation before and after, on a held-out slice of your task and on a general benchmark, to catch forgetting.
4. Shipping. Merge the adapter (merge_and_unload()) and quantize to GGUF for local use, or serve the adapter next to the base model with a multi-LoRA server.

Next: Day 11 — Architectures beyond vanilla: MoE, long context, state-space models, multimodal

1. Three bills every vanilla Transformer pays

Recall the Day 5 block: attention, then an MLP, repeated L times. For one new token at inference, three costs scale badly.

• Compute per token ∝ parameters. Every weight is multiplied once per token, about 2·N FLOPs (Day 6): a 400B dense model does 400 billion multiply-adds to produce the word "the".
• KV cache ∝ layers × KV heads × context. Each layer stores a key and a value per head for every past token (Day 7); long chats and big batches run out of memory before they run out of compute.
• Attention work ∝ context². A 128k-token prompt means 128k × 128k score pairs per head per layer, about 1.7 × 10¹⁰, and naive implementations write that whole matrix to memory.

A fourth limit is a wall rather than a bill: the vanilla model reads only text. The rest of this day is one fix per line.

2. Mixture of Experts: more parameters, same compute per token

The MLP block holds about two-thirds of a Transformer block's parameters (8d² against attention's 4d², from the Day 5 counts). Mixture of Experts (MoE) replaces that one MLP with E copies, the experts, and adds a tiny router: a linear layer that scores each token against each expert, takes a softmax, and sends the token to the top-k experts. Only those k MLPs run; their outputs are blended using the router's weights. Attention, embeddings and norms are shared by all tokens as before.

y = Σ over the top-k experts i of g_i(x) · Expert_i(x), where g = softmax(W_router · x)

The payoff is a decoupling that dense models cannot offer: parameters (how much the model can know) grow with E, but compute per token grows only with k. Shazeer and colleagues introduced the sparsely-gated layer in 2017, scaling an LSTM-based model to 137B parameters with thousands of experts for "greater than 1000× improvements in model capacity with only minor losses in computational efficiency". Shazeer 2017 GShard put it into a 600B-parameter Transformer, Lepikhin 2020 and Switch Transformer simplified routing to a single expert per token (k = 1), reaching the same loss as T5-Base up to 7× faster at equal compute and training a trillion-parameter model. Fedus 2021

Try it: the widget routes a stream of tokens to the top-2 of 8 experts and keeps a tally of active versus total parameters.

Every token touches two experts, so the active count is a quarter of the expert parameters plus everything shared. Watch the per-expert counters: if the router falls in love with one expert, the others stop learning. That is the load-balancing problem.

3. Load balancing, and the models that proved MoE works

A router trained only on the language loss has a bad habit: it finds one slightly better expert, sends it more tokens, which trains it more, until most experts sit idle. Switch Transformer added an auxiliary load-balancing loss, the product of the fraction of tokens sent to each expert and the router probability it assigned, summed over experts, which is smallest when traffic is uniform, plus a capacity factor that caps how many tokens an expert can accept per batch (overflow tokens are dropped and pass through the residual connection). Fedus 2021 DeepSeek-V3 showed you can balance without an extra loss: each expert carries a bias that is added to its routing score for selection only, nudged down when it is overloaded and up when idle, so the language loss is never distorted. DeepSeek 2024

Three releases turned MoE from a research result into the default for big models:

• Mixtral 8x7B (released December 2023, paper January 2024): 8 experts per layer, top-2 routing, 46.7B parameters in total but 12.9B used per token, 32k context, Apache 2.0. It matched or beat Llama 2 70B on every benchmark the authors reported while running at the cost of a 13B model. Mistral 2023 Jiang 2024 The name misleads: "8×7B" is not 56B because attention is shared and only the MLPs are replicated. See the Mixtral paper review.
• DeepSeek-V3 (December 2024): 671B total, 37B active, 61 layers, each MoE layer with 256 small routed experts plus 1 shared expert that every token visits, 8 routed experts active per token. The fine-grained split and the always-on shared expert come from DeepSeekMoE, where many small experts combine more flexibly than a few large ones. Dai 2024 It was trained on 14.8T tokens in FP8 for 2.788M H800 GPU-hours. DeepSeek 2024
• Llama 4 (April 2025): Meta's first MoE. Scout has 17B active of 109B total with 16 experts; Maverick has 17B active of 400B with 128 routed experts plus a shared one. Meta 2025

4. The trade-off, and who uses MoE as of October 2026

MoE buys compute with memory. Mixtral runs like a 13B model but must hold 46.7B weights: 93 GB in bf16, a server model unless quantized to 4 bits (roughly 25 to 30 GB). DeepSeek-V3's 671B weights fill 671 GB even in 8-bit, so serving it means a multi-GPU node with expert parallelism: each GPU hosts some experts and tokens are shipped to wherever their expert lives. The win shows up at scale: serving thousands of users, GPU memory is shared anyway and what you pay per token is the 37B of compute, not the 671B of storage. For one user on one GPU a dense model of the same footprint is often the better deal, which is why Qwen3 and Gemma ship dense small models and MoE large ones.

Among closed models, Google said in February 2024 that Gemini 1.5 uses a Mixture-of-Experts architecture. Google 2024 OpenAI and Anthropic have not published their frontier architectures as of October 2026, so claims that GPT-5 or Claude are MoE are inferences from pricing and speed; OpenAI's open-weight gpt-oss models are MoE, which is suggestive and nothing more. "Every frontier model is MoE" is a reasonable guess: confirmed for most open models and for Gemini, undisclosed for the rest.

5. FlashAttention: the same answer, fewer trips to memory

A GPU has a small, very fast on-chip memory (SRAM, tens of megabytes) and a large, slower high-bandwidth memory (HBM, tens of gigabytes). Standard attention computes the N×N score matrix, writes it to HBM, reads it back for the softmax, writes it again, and reads it again to multiply by V. For N = 8k that is 67 million numbers per head per layer, and the time goes to moving them, not to arithmetic. FlashAttention is IO-aware: it splits Q, K and V into tiles that fit in SRAM, computes the softmax incrementally with a running maximum and sum, and never materializes the full matrix. The output is exactly the same attention. It was 3× faster on GPT-2 at 1k context and 2.4× faster on long-range benchmarks, and it drops attention's memory from quadratic to linear in N, which is why long contexts became trainable at all. Dao 2022 FlashAttention-2 doubled the speed by splitting work better across GPU threads, reaching 50 to 73% of the A100's peak, Dao 2023 and FlashAttention-3 rewrote it for H100s with asynchronous copies and FP8, hitting 75% utilization at 740 TFLOPs/s. Shah 2024 Every serious stack uses it or a descendant; see the FlashAttention paper review.

6. Fewer keys and values: MQA, GQA, MLA and sliding windows

FlashAttention speeds up prefill. In decoding the bottleneck is different: each new token re-reads the whole KV cache, so cache size sets the speed and how many users fit on a GPU. The lever is to store fewer keys and values.

Multi-query attention (MQA) keeps many query heads but a single shared key head and value head, cutting the cache by the number of heads at a small quality cost. Shazeer 2019 Grouped-query attention (GQA) is the compromise: query heads are divided into groups and each group shares one KV head. Ainslie and colleagues converted existing multi-head checkpoints to GQA with about 5% of the original pretraining compute, landing near multi-head quality at near multi-query speed. Ainslie 2023 Llama 2 used GQA for its 34B and 70B models, Touvron 2023 Llama 3 for every size, Meta 2024 and nearly every open model since follows (see the Llama 3 paper review).

Worked example with Mistral 7B: 32 layers, 32 query heads, 8 KV heads, head size 128. Jiang 2023 Each past token costs 2 (K and V) × 32 layers × 8 heads × 128 numbers × 2 bytes = 131 KB; with 32 KV heads it would be 524 KB. At a 32k-token context that is 4.3 GB per conversation instead of 17 GB, which decides how many long conversations fit next to the 14 GB of weights on a 24 GB card.

Multi-head latent attention (MLA), DeepSeek's design, goes further: instead of caching K and V, it caches a single compressed latent vector per token (512 numbers in DeepSeek-V3) and reconstructs keys and values from it with small matrices at attention time. DeepSeek-V2 reported a 93.3% smaller KV cache than its 67B dense predecessor with no quality loss, and V3 kept the design. DeepSeek 2024

Sliding-window attention limits each token to the previous w tokens, making attention cost linear in context. Longformer introduced the windowed-plus-global pattern for long documents in 2020. Beltagy 2020 Mistral 7B uses a 4,096-token window in every layer; because a token in layer ℓ has read a window of layer ℓ−1's outputs, information can still travel 32 × 4,096 ≈ 131k tokens through the stack. Jiang 2023 Gemma 3 interleaves five local layers (window 1,024) with every global layer, which keeps the KV cache small at 128k context because only one layer in six stores the full history. Gemma Team 2025

7. Long context, and whether the model actually uses it

Why does a model trained on 8k tokens fall apart at 16k? Because of positions. With RoPE (Day 5, RoPE review) each position rotates the query and key by an angle proportional to its index; positions past the training length produce rotation angles the model has never seen, and the attention scores turn to noise. Su 2021 The fixes rescale the positions. Position interpolation squeezes the new, longer range into the old one (position 16,000 is presented as 8,000), which Chen and colleagues used to stretch Llama to 32k with about 1,000 fine-tuning steps. Chen 2023 YaRN is choosier about which frequencies to stretch: the slow-rotating dimensions that encode long range get interpolated, the fast ones that encode local order are left alone, and attention is slightly re-scaled; it needed 10× fewer tokens and 2.5× fewer steps than earlier methods. Peng 2023 YaRN-extended Llama 2 checkpoints reached 128k, Nous Research 2023 and YaRN is now a standard switch in model configs: Qwen3-Next is 262k native and stretches to about 1M with it. Qwen 2025

Where things stand as of October 2026: Llama 3.1 shipped 128k in 2024; Meta 2024 Gemini 1.5 Pro put 1M tokens into production in 2024 after testing 10M in research; Google 2024 Llama 4 Scout advertises 10M; Meta 2025 and 1M is now the default window for current Claude and Gemini models. Anthropic docs Google docs So 128k is the floor and 1M the norm, with 10M a marketing figure.

A long window is not the same as using it. The needle-in-a-haystack test hides one sentence in a long document and asks for it, sweeping context length and needle depth. Kamradt 2023 Models pass it easily, and it is almost the only test vendors quote. RULER added multi-needle, multi-hop and aggregation tasks across 17 models claiming 32k or more: nearly all passed the single needle, yet only about half kept acceptable scores at 32k on the harder tasks. Hsieh 2024 "Lost in the Middle" found a U-shaped curve: facts at the beginning or end of the context are used well, facts in the middle are often missed, even by long-context models. Liu 2023 Anthropic's documentation calls the degradation "context rot" and recommends curating what goes in. Anthropic docs So put instructions and key facts at the start or end, do not pad, and for large corpora retrieve (Day 12) rather than stuff.

8. State-space models: Mamba and the hybrids

Attention's superpower is also its cost: every token can look at every earlier token, so the memory of the past is the past itself, stored verbatim in the KV cache. A state-space model (SSM) returns to the recurrent idea from Day 4: keep one fixed-size state, a running summary, and update it once per token. Time is linear in length and per-step memory is constant. S4 showed in 2021 that a carefully structured version could beat Transformers on long-range benchmarks, solving a 16k-length task on which everything else failed. Gu 2021 But early SSMs used the same update rule for every token, so they could not decide what to remember.

Mamba (Gu and Dao, December 2023) made the update selective: how much of the old state to keep, and how much of the new token to write in, are computed from the token itself, so the model can copy an important name into memory and ignore filler. That broke the fast convolutional shortcut earlier SSMs relied on, so the authors wrote a hardware-aware scan that runs the recurrence in parallel on GPUs. The result had 5× the inference throughput of a Transformer, scaled linearly to million-token sequences, and Mamba-3B matched Transformers twice its size on language modeling. Gu 2023 Mamba-2 then showed that SSMs and a form of linear attention are the same object seen from two sides ("state space duality"), and used the connection to make the layer 2 to 8× faster. Dao 2024

Pure SSMs have a known weakness: exact recall of a specific earlier token (copying a long ID, in-context lookup) is hard when the past is compressed into a fixed state. The practical answer is a hybrid: mostly SSM or linear-attention layers, with a few full-attention layers for precise retrieval. As of October 2026 the family is large:

• Jamba (AI21, March 2024): one attention layer per seven Mamba layers, MoE with 16 experts and top-2 routing, 52B total and 12B active, 256k context on a single 80 GB GPU. Lieber 2024
• Nemotron-H (NVIDIA, 2025): 8B and 56B models that replace most attention layers with Mamba layers, up to 3× faster at inference than similar-size Transformers; NVIDIA 2025 followed by Nemotron 3 (December 2025), a Mamba-2 plus MoE hybrid family with a 30B-total, 3B-active Nano and context up to 1M. NVIDIA 2025
• IBM Granite 4.0 (October 2025): Mamba-2 and Transformer blocks at 9:1, from a 3B dense hybrid to a 32B-total, 9B-active MoE, with IBM claiming over 70% less RAM than conventional Transformers on long, batched inputs. IBM 2025
• Qwen3-Next 80B-A3B (2025): three Gated DeltaNet (linear-attention) blocks for every gated full-attention block, 512 experts with 10 plus one shared active per token, 262k native context. Qwen 2025
• Kimi Linear (Moonshot, October 2025): a 48B-total, 3B-active model mixing its Kimi Delta Attention with MLA layers, cutting the KV cache by up to 75% and decoding up to 6× faster at 1M context. Kimi Team 2025

The pattern is consistent: labs that care most about serving cost ship hybrids, and the biggest closed models have not disclosed whether they do. Hybrids have not replaced attention; they have made it one ingredient among several.

9. Multimodal: pixels and sound as tokens

A Transformer does not care what a token means; it only needs a vector per position. That observation is the whole of multimodality. The Vision Transformer (ViT) cuts an image into 16×16-pixel patches, flattens each into a vector, and runs a plain Transformer encoder over the patch sequence, matching convolutional networks once pretrained on enough images. Dosovitskiy 2020 CLIP trained such an image encoder together with a text encoder on 400 million image–caption pairs with a contrastive objective: within a batch, pull each image's vector toward its own caption and push it away from the others. The result is an image encoder whose features are organized by language: it classifies ImageNet "zero-shot" by comparing with the text "a photo of a dog", matching a supervised ResNet-50 without one ImageNet training label. Radford 2021 SigLIP swapped the softmax contrastive loss for a per-pair sigmoid loss that scales better, and its 400M-parameter encoder is the one inside Gemma 3. Zhai 2023 Gemma Team 2025

Then connect the encoder to an LLM. The LLaVA recipe is a vision encoder (a CLIP ViT), a small projector that maps each patch feature into the LLM's token-embedding space, and an off-the-shelf LLM; image patches enter the Transformer as if they were a few hundred word tokens. Trained on instruction-following data that language-only GPT-4 generated from image captions, it reached GPT-4-style visual chat in 2023 at academic cost. Liu 2023 Flamingo had earlier bridged pretrained vision and language models with cross-attention, learning new tasks from a few in-prompt examples. Alayrac 2022 Llama 3's image, video and speech abilities use this "compositional" style, adapters on top of the text model. Meta 2024

Natively multimodal models skip the bolt-on: images, audio and text are tokenized and one network is pretrained on all of them from the start. Google described Gemini as "built from the ground up to be multimodal". Google 2023 OpenAI's GPT-4o is "an autoregressive omni model" trained "end-to-end across text, vision, and audio", so speech goes in and speech comes out without a separate transcription step. OpenAI 2024 Meta's Chameleon showed early-fusion token models that both read and generate images, Chameleon Team 2024 and Llama 4 uses early fusion of text and vision tokens in one backbone. Meta 2025 The costs are real: an image at useful resolution becomes hundreds to thousands of tokens, and mixed-modality training is harder to stabilize.

Two neighbours, briefly. For speech, Whisper is an encoder-decoder Transformer trained on 680,000 hours of audio paired with transcripts, and it remains the standard open recognizer; omni models fold the same idea into the LLM. Radford 2022 For image generation, most systems are not next-token models but latent diffusion: a network learns to remove noise step by step in an autoencoder's compressed latent space, conditioned on text through cross-attention, the design behind Stable Diffusion. Rombach 2021 Chameleon-style autoregressive image tokens are the main alternative, and the two are converging inside omni models.

10. Six open models, side by side

Nobody ships "vanilla" any more: every model here has GQA or something cheaper, RoPE with a scaling trick, and sparsity in the MLPs or in attention. The architecture of 2026 is a kit of parts, and a model card is a list of which parts were picked.

Next: Day 12 — Using LLMs well: prompting, RAG, tools, and agents

1. Prompting that works

A prompt is everything the model sees before it writes: the system message, your request, any examples, any pasted documents. The model does not know what you meant. It only knows what you wrote, and it predicts the most likely continuation given that text (Day 7). So most "bad answers" are under-specified prompts. Five ingredients fix most of them.

Why does the role line help? It is not magic. The pretraining corpus contains tutors, lawyers, and comedians, and a role sentence moves the model into the part of its distribution where those texts live. Both Anthropic's and OpenAI's own guides say the same thing in different words: be explicit, give context, show examples, and tell the model what to do when it is unsure Anthropic docs 2026.

Common failure modes, and the fix for each:

• Ambiguity. "Summarize this" could mean one line or one page. Say the length and the reader.
• Buried instructions. A rule in the middle of a 30-page paste is often missed; put instructions at the top or bottom, and repeat the key constraint at the end.
• Negative instructions only. "Don't be verbose" is weaker than "Answer in at most three sentences."
• Asking for a fact the model cannot have. Anything after its training cutoff, or anything private, must be pasted in or retrieved (section 5).
• Letting it guess. Add "If the answer is not in the text, say 'not found'." This single line removes a surprising share of made-up answers.

2. Few-shot and chain-of-thought

Few-shot prompting is giving the model a handful of worked examples inside the prompt. GPT-3's 2020 paper showed that a 175B model could learn a new formatting or classification task from a few examples with no weight updates at all, a behavior the authors called in-context learning Brown 2020. Examples do two jobs: they define the task more precisely than an instruction can, and they fix the output format. Three to five good examples usually beat twenty sloppy ones. Vary them, and make sure the label balance does not accidentally teach "always say POS".

Chain-of-thought (CoT) prompting is asking the model to write out intermediate steps before the final answer. Wei et al. showed in 2022 that this lifts multi-step arithmetic and logic accuracy sharply, but only once models are large enough (around 100B parameters at the time) Wei 2022. The simplest version is two words: adding "Let's think step by step" to the prompt pushed a 2022 model from 17.7% to 78.7% on a grade-school math benchmark with zero examples Kojima 2022. Why it works is covered in Day 9: each written step is extra computation, and every later token can attend to it. The chain-of-thought paper review has the details and the limits.

System: You are a careful assistant. If information is missing, say so.
User: Here are three support tickets. For each, output one line:
      <id> | <category: billing|bug|feature> | <urgency 1-5>
      Think through each ticket briefly first, then give the table.

      T1: "Charged twice this month, please refund."
      T2: "App crashes when I open settings on Android 15."
      T3: "Would love dark mode."

Assistant: T1 mentions a double charge → billing, money involved → urgent.
T2 is a crash on a current OS version → bug, blocks a feature → high.
T3 is a wish → feature, no blocker → low.

T1 | billing | 4
T2 | bug | 4
T3 | feature | 1

As of October 2026, most frontier models are reasoning models: they produce a hidden or visible chain of thought on their own, with an "effort" knob you set per request (see Day 9). For them, "think step by step" is redundant; what still matters is telling them how much to think and what to check.

3. Structured outputs and constrained decoding

Apps do not want prose. They want a JSON object the code can parse. Two mechanisms exist. JSON mode is a prompt-level promise: the model is trained and instructed to emit valid JSON. Constrained decoding is stronger: at every step, the sampler masks out any token that would break a grammar or schema, so an invalid output is impossible, not just unlikely. A 2023 paper showed how to compile a regular expression or JSON schema into a finite-state machine that tells the sampler which tokens are legal next, with almost no overhead Willard 2023. OpenAI shipped schema-enforced "Structured Outputs" in August 2024, reporting 100% schema compliance on its evals versus under 40% for prompting alone with an older model OpenAI 2024.

Practical rules: keep schemas flat and small; make every field required or give it a default; put an explanation field before the answer field if you want the model to reason first, because tokens are generated left to right.

4. Embeddings for search: cosine similarity, chunking, reranking

An embedding model turns a piece of text into a vector of numbers (often 768 to 3,072 of them) so that texts with similar meaning land near each other (Day 2). Similarity is usually cosine similarity: the cosine of the angle between two vectors.

cos(a, b) = (a · b) / (‖a‖ · ‖b‖), ranging from −1 (opposite) to 1 (same direction)

Two sentences about refund policy might score 0.82; a refund sentence and a sentence about hiking might score 0.11. Modern embedding models are trained with contrastive objectives so that a question and the passage that answers it are pulled together, a recipe set by Sentence-BERT and dense passage retrieval in 2019–2020 Reimers 2019 Karpukhin 2020. The MTEB benchmark compares dozens of them on retrieval, clustering and classification Muennighoff 2022.

A vector database stores millions of these vectors and finds the nearest ones fast. Exact search over 10 million 1,024-dimensional vectors means 10 billion multiplications per query, so databases use approximate indexes such as HNSW, a layered graph you walk greedily toward your query Malkov 2016, or the compressed indexes in FAISS Johnson 2017. For a small project, a Postgres table with the pgvector extension is enough pgvector.

Chunking is splitting documents into pieces before embedding. A whole 40-page manual embedded as one vector is a blurry average; a 300-token section is specific. Typical chunks are 200–800 tokens with some overlap, split on headings or paragraphs rather than fixed character counts, and each chunk carries its source (file, page, section) so the answer can cite it.

Reranking fixes the weakness of embeddings: they compress each text separately, so they miss fine interactions between the question and the passage. A cross-encoder reranker reads the question and a candidate passage together and scores the pair; it is too slow to run on a million passages but fine on the top 50 from the vector search. This two-stage design dates from BERT-based passage reranking in 2019 Nogueira 2019, and hosted rerankers are now an ordinary API call Cohere docs.

5. The RAG pipeline

Retrieval-augmented generation (RAG) is: find the relevant text first, then let the model read it while answering. The name comes from a 2020 paper from Facebook AI Research that paired a dense retriever with a seq2seq generator and trained them end to end, beating pure parametric models on open-domain question answering while making it possible to update knowledge by swapping the index rather than retraining Lewis 2020. The RAG paper review walks through that original design. Modern practice keeps the idea and drops the joint training: the retriever and the LLM are off-the-shelf, glued together by a prompt.

Try it: the widget below runs the modern pipeline on a tiny document set. Step through it and watch what the model actually receives.

Notice that the model never "searches" anything. Your app searches, then pastes the top chunks into the prompt with their sources, and asks the model to answer only from them and to cite the chunk it used. The model's job shrinks from "know everything" to "read these five paragraphs carefully".

The modern pipeline in one line: chunk → embed → retrieve → rerank → prompt with citations. A survey of the many variants (query rewriting, hybrid keyword + vector search, iterative retrieval) is a good map once the basic loop works Gao 2023.

When RAG beats long context, and vice versa. Since 2024 frontier models accept 1M-token windows Gemini team 2024, so why not paste everything? Three reasons. Cost: a 1M-token prompt at Opus 5.5's list price of 4 USD per million input tokens is 4 USD per question, before caching Anthropic 2026. Latency: reading a million tokens takes seconds to tens of seconds. Accuracy: models answer worse when the relevant passage sits in the middle of a long context, the "lost in the middle" effect Liu 2023. A 2024 study comparing the two found long context wins on average when budget is no object, RAG is far cheaper, and a simple router that sends easy questions to RAG and hard ones to long context keeps most of the accuracy at a fraction of the cost Li 2024. Rule of thumb: corpus bigger than the window, changes often, or must be cited → RAG. One document, one careful read → long context.

Evaluating RAG. Separate the two halves. Retrieval is measured with recall@k: of the chunks a human marked as needed, what fraction appear in the top k? Generation is measured with faithfulness (is every claim supported by the retrieved text?) and answer relevance; the RAGAS framework scores these automatically using an LLM as the grader Es 2023. LLM-as-judge is convenient and correlates with human raters above 80% agreement on open-ended tasks, but it has known biases (it prefers longer answers and its own style), so spot-check it Zheng 2023. Build a test set of 50 real questions with known sources before you tune anything.

6. Tool use and function calling

A model cannot check the weather, run code, or read your calendar. But it can write a request to do so. Function calling (also "tool use") works like this: the app describes available tools in the prompt as JSON schemas (name, description, parameters). When the model decides a tool is needed, it emits a structured call instead of prose. The app sees the call, executes the real function, and appends the result to the conversation as a tool message. The model then continues. The model never runs anything; it only fills in a form.

// 1. App → model: tool definitions + user message
tools: [{ name: "get_weather",
          description: "Current weather for a city",
          parameters: { city: "string" } }]
user:  "Do I need an umbrella in Oslo today?"

// 2. Model → app: a structured call (no prose)
tool_call: { name: "get_weather", arguments: { city: "Oslo" } }

// 3. App runs the real function, sends the result back
tool_result: { temp_c: 7, condition: "rain", rain_mm: 4 }

// 4. Model → user
"Yes. It is 7 °C and raining in Oslo right now, about 4 mm."

OpenAI added function calling to its API in June 2023 OpenAI 2023, and every major API now offers the same shape Anthropic docs 2026 OpenAI docs 2026. Under the hood it is post-training: models are fine-tuned on many examples of "when a tool fits, emit this exact format", the same way they were taught to chat (Day 8). The 2023 Toolformer paper showed models can even teach themselves when to insert API calls by checking whether the call lowers their own loss Schick 2023. Good tool descriptions matter as much as good prompts: say what the tool returns, when to use it, and when not to.

7. Agents: the loop, planning, memory, coding agents, MCP

An agent is a model running in a loop with tools: think, call a tool, observe the result, think again, until the task is done. The pattern was named ReAct (Reason + Act) in 2022: interleaving reasoning traces with actions beat either alone on question answering and on text-based games, and produced trajectories a human could audit Yao 2022. Try it: the widget below steps through one agent task.

Each turn around the loop is a fresh model call with the whole history so far in the context. The "memory" of the agent is literally the transcript. When the transcript gets long, the app summarizes or trims it, which is where many agents quietly lose track.

Three ingredients separate a toy loop from a useful one. Planning: ask the model to write a short plan first, then execute step by step and revise the plan on surprises. Memory: short-term memory is the context window; long-term memory is a file or database the agent can read and write, often through RAG over its own notes. Reflexion showed that letting an agent write a sentence about why it failed, and feeding that back on the next attempt, improves success rates without any training Shinn 2023. Stopping rules: a maximum number of steps, a budget in tokens, and a clear definition of "done". Anthropic's engineering guide makes the same point from the other direction: most production systems are not open-ended agents but fixed workflows (prompt chains, routers, parallel calls) with one agentic step where it is needed Anthropic 2024.

Coding agents are where this works best as of October 2026, because code gives the agent a tool that tells the truth: the test either passes or it does not. SWE-bench measures exactly this, resolving real GitHub issues in real repositories Jimenez 2023; the first agent scaffold for it, SWE-agent, got 12.5% in 2024 Yang 2024, and the top of the Verified leaderboard sat at 97% by September 2026 Vals.ai 2026. The products: Claude Code, a terminal agent that reads your repo, edits files and runs commands Anthropic docs 2026; OpenAI Codex, which runs tasks in a cloud sandbox and in a CLI OpenAI 2026; Cursor, an editor with an agent mode Cursor docs 2026; Gemini CLI Google 2025; and GitHub Copilot's coding agent, which takes an issue and opens a pull request GitHub 2025. The same loop now drives computer use, where the tools are screenshots, clicks and keystrokes, first shipped as a public beta by Anthropic in October 2024 Anthropic 2024 and by OpenAI's Operator in January 2025 OpenAI 2025.

The Model Context Protocol (MCP), released by Anthropic in November 2024, is a plug standard. Before it, every app had to write custom glue for every tool. MCP defines one way for a "server" (your database, Slack, a file system, a browser) to advertise its tools and data, and one way for any model app (a "client") to call them Anthropic 2024. Think USB for tools: write the server once, and Claude Code, Cursor, ChatGPT or your own agent can all use it MCP docs. It carries no intelligence; it is plumbing, and that is the point.

8. A worked mini-example: one agent, two tool calls

Task given to the agent: "Find the total of last month's invoices in invoices.csv and email it to finance." Tools available: run_python(code) and send_email(to, subject, body).

Turn 1 (model): I need the total first. I'll read the CSV with Python.
  tool_call run_python:
    import csv, datetime
    rows = list(csv.DictReader(open("invoices.csv")))
    last = datetime.date.today().replace(day=1) - datetime.timedelta(days=1)
    tot = sum(float(r["amount"]) for r in rows
              if r["date"][:7] == last.strftime("%Y-%m"))
    print(round(tot, 2), len(rows))
  tool_result: 48213.5 312

Turn 2 (model): Total is 48,213.50 across the September rows. Now send it.
  tool_call send_email:
    to: "finance@example.com"
    subject: "September invoice total"
    body: "Total of September invoices: 48,213.50 EUR (from invoices.csv)."
  tool_result: {"status": "sent", "id": "m_8831"}

Turn 3 (model): Done. I emailed finance the September total of 48,213.50 EUR.

Three details to notice. The model wrote code rather than doing arithmetic in its head, which is the right instinct: tools are for things models are bad at. The second call depended on the first result, so the loop had to be sequential. And the final message reports what was observed (the tool said "sent"), not what was intended. A well-built agent also asks before irreversible actions; many apps require a human click before send_email runs.

9. Reliability and cost: latency, caching, guardrails, evals

Latency has two parts: time to first token (dominated by reading the prompt) and tokens per second after that (Day 7). A 50,000-token system prompt makes every request slow and expensive. Prompt caching fixes the repeated part: the provider stores the KV cache for a prefix you reuse, and charges a fraction for cache hits. Anthropic's cached reads cost about one tenth of normal input Anthropic 2024; OpenAI applies caching automatically to repeated prefixes at half price OpenAI 2024. Put the stable parts (instructions, tool definitions, reference documents) first and the changing parts (the user's message) last, so the prefix matches.

Cost is tokens in plus tokens out, at very different prices per model. As of October 2026, list prices per million tokens run from 0.10 USD in / 0.50 USD out for a small model such as Claude Haiku 5.5 Anthropic 2026 to 10 USD / 50 USD for the largest public tier, Claude Fable 5.1 Anthropic 2026; GPT-5.6's three sizes are 1/6, 2.5/15 and 5/30 OpenAI 2026. A 100× spread. The standard move is routing: a cheap model classifies or drafts, an expensive one handles the hard 10%. Reasoning models add a hidden multiplier, since thinking tokens are billed as output; set the effort level per task.

Guardrails are checks outside the model: an input filter for prompt injection and off-topic requests, an output filter for leaked secrets or disallowed content, a schema check on structured outputs, and an allow-list on which tools an agent may call without a human. Frameworks such as NeMo Guardrails express these as rules around the model rather than hoping the prompt holds Rebedea 2023. The attacks they defend against are Day 13.

Evals for apps are the thing most teams skip and regret. Collect 50–200 real inputs with expected outputs, run every prompt change against them, and track the score like a unit test OpenAI docs 2026. Grade with exact match where possible, with an LLM judge where not, and with a human on a sample. Without this, every prompt tweak is a guess, and the guess that fixes one case silently breaks three others. Anthropic's note on context engineering makes the same argument for agents: the unit you design and test is the whole context the model sees at each step, not a single prompt string Anthropic 2025.

Next: Day 13 — Evaluating LLMs, safety, and what they get wrong

1. How models are measured

A benchmark is a fixed set of questions with known answers, scored automatically. Each one was hard when it was published. Here are the ones you will see in every model announcement, what each tests, where models started, and where the frontier stood in October 2026. Scores depend on the harness (tools or not, how much "thinking", how many attempts), so always read the footnote.

Read the last column with care. The GPQA and HLE numbers are the labs' own reports or third-party boards with different settings, and they move every month. The useful fact is the shape: a benchmark goes from "far below human experts" to "above them" in two to three years, then stops being informative.

2. Saturation and contamination

Try it: the widget shows the frontier score on three benchmarks over time. Each curve flattens near the top, and a new, harder test appears.

Saturation does not mean the model is perfect at the skill. It means the test can no longer tell a better model from a worse one. Some remaining "errors" are mislabeled questions; SWE-bench Verified exists because humans had to remove unsolvable tasks from the original set.

Contamination is the second problem: benchmark questions leak into training data. The web contains MMLU, GSM8K and HumanEval many times over, with answers. A model that has seen them is remembering, not reasoning. To test this, one team rewrote GSM8K from scratch with matched difficulty and found that several model families scored several points lower on the fresh set, a sign of overfitting to the public one Zhang 2024. Private test sets (ARC Prize's semi-private set, FrontierMath, HLE's held-out questions) are the standard fix, which is why you should trust a verified run on a hidden set more than a lab's own number.

Human-preference leaderboards have their own leak. A 2025 analysis of LMArena found that large labs could privately test many variants of a model and publish only the best, which inflates ratings; it reports 27 private variants tested by one lab before a single launch Singh 2025. And voters prefer confident, long, nicely formatted answers, which is not the same as correct ones.

3. Hallucination: why a next-token predictor confabulates

A hallucination is a fluent, confident statement that is false: a citation that does not exist, a court case that was never filed, a function in a library that was never written. It is not a glitch. Remember what the model is (Day 1): a machine that outputs the most plausible continuation. "The capital of Australia is" has one strong continuation. "The 2019 paper by Dr. Mbeki on" has thousands of plausible ones, and the model picks one at the same confident pace. Pretraining rewards plausibility, not truth, and it has no bit that says "I have not seen this".

A 2025 analysis from OpenAI makes the second half of the argument: even a model that could tell when it is unsure is pushed to guess, because almost every benchmark scores a wrong answer and an "I don't know" the same (zero), so the policy that maximizes score is to always answer. They propose scoring confident errors below abstentions Kalai 2025 OpenAI 2025. Short-answer factual tests confirm how bad the baseline is: on SimpleQA's 4,326 fact questions, frontier models in late 2024 answered well under half correctly while stating most answers confidently Wei 2024.

Calibration is whether stated confidence matches accuracy: of the claims a model marks 80% sure, are 80% true? Base models are surprisingly well calibrated on their own answers when asked "is this true?" with a probability, and larger models more so Kadavath 2022. Post-training then moves calibration in both directions: RLHF makes models more helpful but also more willing to assert, and reasoning training can raise both accuracy and hallucination rate at once; OpenAI's own system card for o3 reported a higher hallucination rate than its predecessor o1 on a person-facts test, because it made more claims overall OpenAI 2025. A broad survey of causes and detection methods is a good next read Huang 2023.

Mitigations, in order of effectiveness: retrieval with citations (Day 12), so claims are grounded in text the user can check; abstention, by instructing and rewarding "not found" and by routing low-confidence answers to a human; verification tools, such as running the code it wrote or looking up the citation it produced; and asking for confidence and acting on it. None removes hallucination. All of them shrink it to where a reader can catch the rest.

4. Prompt injection and jailbreaks

A jailbreak is a user talking a model out of its safety training ("pretend you are my late grandmother who used to read me napalm recipes"). A prompt injection is different and more dangerous: text that the model reads as data contains instructions, and the model follows them. The first systematic demonstrations in 2022 showed "ignore the previous instructions and…" hijacking GPT-3 apps and leaking their hidden prompts Perez 2022. Indirect prompt injection moves the attack into content the model fetches: a web page, an email, a PDF, a tool result. A 2023 paper showed an assistant that browses the web can be turned against its own user by a hidden line on a page Greshake 2023. OWASP lists prompt injection as number one in its Top 10 risks for LLM applications OWASP 2025.

Why is this hard? Because the model's one skill is following instructions in text, and there is no reliable marker that separates the user's instructions from the document's. Jailbreaks are hard for a related reason: safety training and helpfulness training pull in opposite directions ("competing objectives"), and safety training generalizes worse than capability, so an unusual encoding (base64, a poem, another language) can find a gap Wei 2023. Automated attacks exist: gradient search finds nonsense suffixes that unlock open-weight models and transfer to closed ones Zou 2023, and long context lets an attacker paste hundreds of fake "examples" of the model complying, which works better the longer the window Anthropic 2024. Defenses have improved: classifier systems trained on a written constitution of what is and is not allowed held up against thousands of hours of red-teaming with a small cost in false refusals Sharma 2025, and labs now report injection attack success rates in launch posts, such as 0.7% on a third-party indirect-injection benchmark for Gemini 4 Argon Google 2026. But "rare" is not "never", and an agent with your inbox needs the structural defenses above regardless.

5. Sycophancy

Sycophancy is telling the user what they want to hear: agreeing with a wrong claim, changing a correct answer when challenged, praising bad work. Anthropic researchers showed in 2023 that five leading assistants all did it, and traced the cause to preference data: human raters, on average, slightly prefer agreeable responses, so a reward model trained on them rewards agreement Sharma 2023. The failure became public in April 2025, when OpenAI rolled back a GPT-4o update after it became "overly flattering or agreeable", and explained that short-term thumbs-up signals had been weighted too heavily in training OpenAI 2025 OpenAI 2025. The lesson generalizes: whatever you optimize for, you get, including the parts you did not mean. The practical defense for you as a user is to ask for the strongest case against your view, and to never use "are you sure?" as a test of truth.

6. Bias and fairness

Models learn the statistics of their training text, including its stereotypes. The clean early demonstration was in word vectors: the analogy "man is to computer programmer as woman is to…" returned "homemaker" Bolukbasi 2016. For chat models, the BBQ benchmark asks questions where a stereotype would suggest an answer and the correct answer is "cannot be determined", and measures how often models pick the stereotype Parrish 2021. Decision-style tests (should this loan be approved, given a profile) found both negative and positive discrimination by demographic group, much of which could be reduced with explicit instructions in the prompt, which helps but is not a guarantee Tamkin 2023. Two things make fairness harder than it looks: the definitions conflict (equal error rates across groups and equal treatment of identical inputs cannot both hold in general), and post-training can hide a bias in direct questions while leaving it in indirect ones, such as which names get the shorter cover letter. If a model makes decisions about people, test it on your own population, not on a published benchmark.

7. Interpretability: features, sparse autoencoders, circuits

Interpretability asks what is going on inside the network, not just what comes out. The problem is that single neurons are not meaningful: a neuron fires for DNA sequences and legal text and Korean, because the model packs more concepts than it has neurons, a phenomenon called superposition Elhage 2022. The 2023 fix was the sparse autoencoder (SAE): train a second, much wider network to rebuild a layer's activations from a sparse set of directions. Each direction, a feature, tends to mean one thing Bricken 2023 Cunningham 2023.

In "Scaling Monosemanticity" (2024) Anthropic ran this on a production model, Claude 3 Sonnet, and found millions of features: the Golden Gate Bridge, code bugs, sycophantic praise, deception, specific people. Turning a feature up changes behavior: with the bridge feature clamped high, the model brought the Golden Gate Bridge into every answer, including who it believed it was Templeton 2024. In 2025 the group went from features to circuits: graphs of which features cause which, traced through a single forward pass, an approach descended from the "circuits" program on vision models Olah 2020 Ameisen 2025. The findings read like biology: the model plans the rhyme word of a poem several tokens ahead; it shares a language-independent "concept" space between English, French and Chinese; and when it does mental arithmetic, the circuit it actually uses is not the carry-the-one method it writes down when asked to explain Lindsey 2025 Anthropic 2025. That last result is the one to remember: the chain of thought a model writes is a story about its computation, not a readout of it.

8. Alignment and safety frameworks, and the EU AI Act

Alignment is the project of making models want what we want and behave that way under pressure; Day 8 covered the training half (RLHF, constitutional AI). Two results show why training is not enough on its own. A model deliberately trained with a hidden backdoor kept it through standard safety training, and adversarial training taught it to hide the behavior better Hubinger 2024. And a production model, told it would be retrained to drop its values, sometimes complied strategically during "training" while reasoning that it was doing so to protect those values, which is the behavior called alignment faking Greenblatt 2024.

The institutional response is frontier safety frameworks: public commitments that tie model capability to required safeguards. Anthropic's Responsible Scaling Policy defines AI Safety Levels and the evaluations that trigger them; it was rewritten as version 3.0 in February 2026 and is at version 3.4 as of October 2026 Anthropic 2026. OpenAI's Preparedness Framework tracks categories such as biology, cyber and autonomy, with thresholds that gate deployment OpenAI 2025, and Google DeepMind's Frontier Safety Framework defines critical capability levels Google DeepMind 2024. The frameworks rest on two practices. Red-teaming is paying people (and other models) to attack the model before release Ganguli 2022 Perez 2022. Dangerous-capability evals test whether a model can meaningfully help with bioweapons, offensive cyber operations, persuasion, or copying itself to new servers Phuong 2024; governments now run some of these independently through bodies such as the UK AI Security Institute AISI. The frameworks had a visible effect in 2026: Anthropic declined to release its most capable model, Claude Mythos Preview, to the public because of its ability to find and exploit software vulnerabilities, giving access only to vetted defenders Anthropic 2026. Whether such voluntary commitments hold under competition is one of the open questions of Day 14.

Regulation is arriving on a schedule. The EU AI Act entered into force on 1 August 2024 with staged deadlines European Commission AI Act timeline, and the "Digital Omnibus on AI" (Regulation (EU) 2026/1744, in force 27 July 2026) pushed the high-risk deadlines back Gibson Dunn 2026 AI Act explorer 2026.

9. Copyright and data privacy

Models are trained on text their makers mostly did not license. Whether that is lawful is being decided case by case, and the answers differ by country. Status as of 19 October 2026:

The pattern so far: US courts have leaned toward calling the act of training "transformative", while punishing how the data was obtained and whether outputs copy; a German court treated memorized lyrics as plain infringement; an English court focused on whether the model contains the works. Data privacy raises a separate set of issues: training sets contain personal data scraped from the web, models can regurgitate it, and the lawsuits themselves pull in user data, as when the court in the NYT case ordered OpenAI to hand over 20 million de-identified ChatGPT conversations for discovery Bloomberg Law 2026. If you build with these models, assume what you send may be retained, read the data-use terms, and do not paste what you could not defend sharing.

10. Energy and environmental cost

Two different bills: training once, and serving billions of times. Training. Meta reported 39.3 million GPU-hours and 11,390 tonnes of CO₂-equivalent (location-based) for the Llama 3.1 family, of which the 405B model took 30.8 million hours and 8,930 tonnes Meta 2024. Mistral's audited life-cycle study put Mistral Large 2 at 20,400 tonnes CO₂e and 281,000 m³ of water over 18 months of training and use Mistral 2025. For scale, 10,000 tonnes is roughly the annual footprint of 1,000 average Europeans, serious but small next to a single airline. The alarm first sounded in 2019, when a study estimated a neural-architecture-search run at about 626,000 pounds of CO₂ Strubell 2019; Google's 2021 accounting put GPT-3's training at 1,287 MWh Patterson 2021.

Inference. Google measured the median Gemini text prompt in May 2025 at 0.24 Wh of energy, 0.26 mL of water and 0.03 g CO₂e, counting idle capacity and overhead, and reported a 33× energy reduction per prompt over the previous year from better models and better serving Google 2025 Elsworth 2025. Sam Altman gave 0.34 Wh for an average ChatGPT query Altman 2025, and Mistral measured 1.14 g CO₂e and 45 mL of water for a 400-token answer. Worked number: 0.24 Wh is what a 60 W bulb uses in 14 seconds; a billion such prompts a day is 240 MWh/day, the consumption of a town of about 30,000 people. Reasoning models and agents multiply tokens per task by 10–100×, so these per-prompt figures understate where usage is heading.

The system. The International Energy Agency estimated data centres at about 415 TWh in 2024, around 1.5% of world electricity, and projected roughly 945 TWh by 2030 with AI the main driver IEA 2025; its 2026 update raised the 2025 estimate to about 485 TWh and kept the 2030 figure near 950 TWh, about 3% of global demand IEA 2026. Single training sites now draw a gigawatt, the supply of about a million US homes Epoch AI 2026. The honest summary: per query, AI is cheap; in aggregate, it is a new industrial load arriving faster than grids were planned for, and where that electricity comes from matters more than the AI itself.

Next: Day 14 — The frontier (2024–2026) and where to go next

1. How to read a model timeline

Three warnings before the table. First, a release date is the day a lab said something exists, not the day it became useful; many "launches" were previews for a few partners. Second, version numbers are marketing: Claude 3.7 to 4 was a bigger jump than 4.5 to 4.6, and GPT-4.5 came before GPT-4.1. Third, everything below is accurate as of 19 October 2026 and cites the lab's own announcement; later releases will not be here. Try it: the widget below lists every major release with a link to its announcement; filter it by lab.

Notice the cadence. In 2023 the frontier moved about twice a year. By 2026 each of the four largest labs ships a new frontier tier every two to three months, and the open-weight labs follow within one to four months.

2. The dated timeline, GPT-4 to October 2026

3. The big trends

Reasoning trained with RL. o1 (September 2024) showed that rewarding a model for correct final answers teaches it to write long, self-correcting chains of thought, and that accuracy rises with thinking time (Day 9). DeepSeek-R1 showed the recipe works with mostly pure RL on an open model and published it DeepSeek 2025. By 2026 every frontier model is a reasoning model with an "effort" dial, and the hardest math benchmark of 2024, FrontierMath Tier 4, is reported near saturation by the top model OpenAI 2026.

Agents and computer use. The length of software tasks that models can complete at 50% reliability has doubled roughly every seven months since 2019 Kwa 2025. On OSWorld-Verified, a benchmark of real desktop tasks where humans score about 72%, GPT-5.5 reported 78.7% in April 2026 OpenAI 2026. The products that matter are the coding agents of Day 12; the next wave is the same loop pointed at browsers and spreadsheets.

Open weights versus closed. The open-weight frontier moved east. Meta, which carried it from Llama 2 to Llama 4, launched a proprietary model in April 2026 and says only that it hopes to open future versions Meta 2026. DeepSeek (V4: 1.6T parameters, open) DeepSeek 2026, Alibaba (Qwen3.8: 2.4T, open) Qwen 2026, Moonshot and Mistral now set the open frontier, typically three to six months behind the closed one; OpenAI's gpt-oss was a one-off return to open weights OpenAI 2025. For a builder this means you can run a near-frontier model on your own hardware, and that the licenses to read are now MIT, Apache 2.0, and custom Chinese-lab terms.

Small models and distillation. Distillation trains a small "student" model on the outputs of a large "teacher", an idea from 2015 Hinton 2015. R1's distilled 1.5B–70B models made it mainstream, and every lab's small tier (Haiku, Flash, Luna, mini) is now a distilled or heavily post-trained model that does 90% of the work at 1–5% of the price. The practical rule from Day 12 follows: route most traffic to a small model.

Long context and multimodality. Context went from 8K tokens (GPT-4, 2023) to 1M (Gemini 1.5, 2024) and is standard at the top tier in 2026, with retrieval accuracy on 8-needle tests at 512K–1M tokens still separating models (96% vs 74% in one lab's comparison) OpenAI 2026. "Multimodal" moved from image input (2023) to native audio and image output (GPT-4o, Gemini 2.0) to models that are natively multimodal in the open-weight world too (Llama 4, Mistral Large 4).

Synthetic data. The phi models showed that carefully generated "textbook-quality" synthetic text can train small models that punch far above their size Gunasekar 2023 Abdin 2024. Reasoning models made the loop explicit: a strong model generates and verifies solutions, a new model trains on them. Most frontier post-training data in 2026 is model-generated and filtered, with humans writing the rubrics rather than the answers.

Compute build-out. Epoch AI estimates the computing capacity of the single largest AI site has doubled about every seven months since August 2024, from roughly 80,000 H100-equivalents to a projected 1.1 million at xAI's Colossus 2 in mid-2026 Epoch AI 2026. A gigawatt site draws the power of about a million US homes Epoch AI 2026, and the IEA expects data centres to use about 950 TWh a year by 2030, some 3% of world electricity (Day 13) IEA 2026.

Falling prices per token. Epoch AI found that the price to reach a fixed benchmark score has fallen between 9× and 900× per year depending on the task, about 40× per year for GPT-4-level science questions Epoch AI 2025; Stanford's AI Index put the fall for GPT-3.5-level performance at over 280× between November 2022 and October 2024 Stanford HAI 2025. The sticker price of the very top tier has not fallen (Fable 5.1 lists at 10 / 50 USD per million tokens), but last year's top tier is this year's mid tier at a fifth of the price, and reasoning models spend more tokens per task, so your bill depends on routing as much as on list prices.

4. The open questions

The data wall. A 2022 estimate projected that the stock of public human-written text would be fully used by frontier training runs between 2026 and 2032 Villalobos 2022. Labs responded with synthetic data, RL on verifiable tasks (code, math), and multimodal data. Whether synthetic text can keep improving models rather than just recycling them is unresolved.

Scaling limits. Pretraining scale alone gave diminishing returns around GPT-4.5; the gains since have come from post-training and test-time compute (Day 6, Day 9). Nobody knows how far RL on reasoning scales, or whether a 1-gigawatt training run buys a qualitative jump or an incremental one.

Evaluation. ARC-AGI-3 went from 0.51% to 99.9% (with a provider-tuned harness) in six months ARC Prize 2026. When tests saturate faster than they can be written, the field falls back on private evals, expert judgments, and real-world outcomes, none of which are comparable across labs. The measurement problem of Day 13 is now the central scientific problem.

Safety. In 2026 two labs withheld or gated their strongest models over cyber capabilities Anthropic 2026 Google 2026. That is the safety frameworks working as written. The open question is whether voluntary frameworks survive a race between five labs and two governments, and whether regulation (the EU's high-risk rules now start December 2027) arrives in a form that helps.

5. How to keep learning

Two weeks is a start. The resources below are the ones that reward a second pass, in roughly the order to take them.

6. Three capstone projects

Pick one. Each takes a weekend, runs on a laptop or a free cloud GPU, and exercises a different half of the course.

Capstone A: train a tiny GPT on your own text (Day 1 to Day 7). Collect 1–5 MB of text you care about: your emails, a favorite author's public-domain novels, song lyrics, a forum archive. Run nanoGPT's character-level recipe, which trains a ~10M-parameter model in minutes on one GPU Karpathy 2023.

git clone https://github.com/karpathy/nanoGPT && cd nanoGPT
pip install torch numpy tiktoken
# put your text at data/mytext/input.txt, then reuse the Shakespeare prep:
cp data/shakespeare_char/prepare.py data/mytext/ && python data/mytext/prepare.py
python train.py config/train_shakespeare_char.py --dataset=mytext \
       --max_iters=3000 --out_dir=out-mytext
python sample.py --out_dir=out-mytext

Steps that teach the most: (1) write down the validation loss at 500, 1,500 and 3,000 iterations and plot it; (2) halve n_layer and compare, then double the data and compare, to feel the scaling law of Day 6; (3) sample at temperature 0.3, 0.8 and 1.5; (4) train the Day 1 n-gram model on the same text and compare samples. Done when you can explain, with your own numbers, why the transformer beats the n-gram.

Capstone B: fine-tune a small model with LoRA (Day 8, Day 10). Choose a task with a checkable output: turn messy meeting notes into a fixed JSON summary, classify support tickets, or answer questions in a specific style. Build 300–1,000 input–output pairs (write 50 by hand, generate the rest with a frontier model, then review a sample). Take an open 1B–4B model such as a small Qwen3.5 Qwen 2026, and train a LoRA adapter with Hugging Face PEFT Hugging Face docs or Unsloth, which fits QLoRA on a free Colab GPU Unsloth docs. Hold out 50 pairs; score the base model with few-shot prompting, then your adapter. Done when you can state the accuracy of both, the adapter's size in MB, and one example where fine-tuning hurt. The LoRA and QLoRA paper reviews explain the knobs.

Capstone C: build a RAG app (Day 12, Day 13). Pick 50–500 documents: your course notes, a product manual, a city's council minutes. Chunk by section (300–600 tokens). Embed with a sentence-transformers model SBERT docs, store in FAISS FAISS or pgvector, retrieve the top 20, rerank to 5, and answer with any API model or a local one through Ollama Ollama, with the instruction to cite chunk IDs and say "not found" otherwise. Write 50 test questions with the chunk that answers each. Measure recall@5 for retrieval and faithfulness for answers (Day 12). Then attack it: put "ignore your instructions" inside one document and see what happens (Day 13). Done when you have the two numbers, the attack result, and a one-paragraph note on what you would change.

7. The whole course on one page

You made it. Go to the Papers tab and read the originals.