LATENT
HOW AI WORKS - FRAME BOOK
VOL. 01·SEP 2026

LEARNING FROM THE NEXT TOKEN.

A MODEL READS TEXT LEFT TO RIGHT AND GUESSES EACH NEXT TOKEN. NO HUMAN LABELS: THE TEXT ITSELF IS THE ANSWER KEY.

SURPRISE

EACH GUESS IS SCORED BY HOW SURPRISED THE MODEL IS BY THE TRUE NEXT TOKEN.

ℓt=−log⁡pθ(xt+1∣x≤t)\ell_t = -\log p_\theta(x_{t+1} \mid x_{\le t})
x≤tx_{\le t}
Context so far
xt+1x_{t+1}
True next token
pθp_\theta
Model probability
ℓt\ell_t
Surprise (loss)
LL
Mean surprise

01CORPUS

∣D∣≈1012 to 1013 tokens\lvert D \rvert \approx 10^{12}\ \text{to}\ 10^{13}\ \text{tokens}

Web pages, books, code: trillions of tokens.

02NEXT TOKEN

targett=xt+1\text{target}_t = x_{t+1}

Targets are the inputs, shifted by one.

03SURPRISE

ℓ=−log⁡ptrue\ell = -\log p_{\text{true}}

Small probability on the truth, big surprise.

04BATCH

L=1BT∑b,tℓb,tL = \tfrac{1}{BT}\sum_{b,t} \ell_{b,t}

All positions scored at once, then averaged.

05LOSS CURVE

L0=ln⁡∣V∣≈10.8L_0 = \ln \lvert V \rvert \approx 10.8

From random guessing to fluent text.

Pre-training

Learning from Next-Token Prediction

A LANGUAGE MODEL LEARNS BY READING. AT EVERY POSITION IN TRILLIONS OF TOKENS IT GUESSES THE NEXT ONE, SCORES ITS SURPRISE, AND NUDGES ITS PARAMETERS TO BE LESS SURPRISED NEXT TIME.

CORPUSraw textTOKENSx1…xTx_1 \dots x_TBATCHB×TB \times TMODELθ\thetaθ←θ−η∇θL\theta \leftarrow \theta - \eta \nabla_{\theta} L
OBJECTIVE
min⁡θ 1T∑t=1T−log⁡pθ(xt+1∣x≤t)\min_{\theta}\ \frac{1}{T}\sum_{t=1}^{T} -\log p_\theta(x_{t+1} \mid x_{\le t})
SURPRISE
ℓ=−log⁡p\ell = -\log p
p = 0.40 →\to 0.92
p = 0.02 →\to 3.91
PERPLEXITY
PPL=eL\mathrm{PPL} = e^{L}
e1.66≈5.3e^{1.66} \approx 5.3
like choosing among ~5 tokens
BEFORE TRAINING
L0=ln⁡50,000≈10.8L_0 = \ln 50{,}000 \approx 10.8
uniform guess over the vocabulary
SCALE (EXAMPLES)
GPT-3 · 300B tokens · 2020
Llama 3 · 15T tokens · 2024