LOSS FALLS ON A STRAIGHT LINE.

ON LOG–LOG AXES, LOSS DROPS AS A POWER LAW OF MODEL SIZE, DATA, AND COMPUTE: SMOOTH ENOUGH THAT SMALL RUNS PREDICT BIG ONES.

COMPUTE-OPTIMAL FRONTIER

EACH THIN CURVE IS ONE MODEL SIZE, TRAINED ON MORE AND MORE TOKENS. THE LOWEST LOSS FOR EACH BUDGET TRACES THE STRAIGHT CYAN LINE.

C \approx 6\,N D

$N$: Parameters
$D$: Training tokens
$C$: Compute (FLOPs)
$L$: Test loss
$E$: Irreducible loss

01PARAMETERS

L(N) - E \approx A\,N^{-\alpha}

Bigger models: loss falls on a line.

02DATA

L(D) - E \approx B\,D^{-\beta}

More tokens: the same straight line.

03COMPUTE-OPTIMAL

C \approx 6\,N D

Each budget has one best model size.

0420 TOKENS PER PARAMETER

D_{\text{opt}} \approx 20\,N_{\text{opt}}

Same compute: smaller model, more tokens.

ON THE LINE: COMPUTE-OPTIMAL
ABOVE IT: TOO FEW TOKENS PER PARAMETER
DASHED: SAME COMPUTE

05EMERGENCE?

P(\text{all } k \text{ right}) = p^{\,k}

Sudden jumps can hide smooth progress.

DEBATED: SOME ABILITIES LOOK SUDDEN;
SMOOTHER METRICS OFTEN SHOW STEADY GAINS.
$p = 0.64 \Rightarrow p^{10} = 0.01$
$p = 0.94 \Rightarrow p^{10} = 0.54$

ScalingLaws

More Data, More Compute, Lower Loss

LOSS FALLS SMOOTHLY AND PREDICTABLY AS MODELS, DATA, AND COMPUTE GROW. FOR A FIXED BUDGET, THE BEST RESULT BALANCES MODEL SIZE AGAINST TRAINING TOKENS: ROUGHLY 20 TOKENS PER PARAMETER.

LOSS OVER THE N–D PLANETOKENS

D

PARAMS

N

20 : 1C = constCHINCHILLAGOPHER

LOSS MODEL: $L(N, D) = E + \dfrac{A}{N^{\alpha}} + \dfrac{B}{D^{\beta}}$
COMPUTE: $C \approx 6\,N D$; FLOPs $\approx$ 6 × parameters × tokens
COMPUTE-OPTIMAL: $N_{\text{opt}} \propto C^{0.5}$; $D_{\text{opt}} \propto C^{0.5}$
RULE OF THUMB: $D_{\text{opt}} \approx 20\,N_{\text{opt}}$
EXAMPLES: Chinchilla · 70B · 1.4T · 20:1; Gopher · 280B · 0.3T · ~1:1; GPT-3 · 175B · 0.3T · ~2:1
IN PRACTICE: Small models are often trained far past 20:1: they are cheaper to run.

LOSS FALLS ON A STRAIGHT LINE.

COMPUTE-OPTIMAL FRONTIER

01/PARAMETERS

02/DATA

03/COMPUTE-OPTIMAL

04/20 TOKENS PER PARAMETER

05/EMERGENCE?