Context Window & KV Cache — What the Model Can Remember · LATENT

WHAT THE MODEL CAN REMEMBER.

A MODEL KEEPS NO NOTES BETWEEN WORDS. ALL IT HAS IS THE TEXT IN FRONT OF IT — THE CONTEXT WINDOW — AND THAT WINDOW HAS A FIXED SIZE.

LOOK BACK, NEVER AHEAD.

EACH NEW TOKEN ATTENDS TO EVERY EARLIER TOKEN IN THE WINDOW. THEIR KEYS AND VALUES WAIT IN A CACHE, SO NOTHING IS COMPUTED TWICE.

WINDOW SHOWN: 12 TOKENS REAL MODELS: 8K – 1M+ TOKENS

01CONTEXT WINDOW

n_{\text{in}} + n_{\text{out}} \le N_{\max}

Everything must fit. The oldest falls out.

02CAUSAL MASK

A = \mathrm{softmax}\!\left(\dfrac{QK^{\top}}{\sqrt{d_k}} + M\right)

M_{ij} = -\infty \ \text{if}\ j > i

Each token sees only the past.

03KV CACHE

\underbrace{\tfrac{n(n+1)}{2}}_{\text{no cache}} \;\text{vs}\; \underbrace{n}_{\text{cache}}

Keep past keys and values. Reuse them.

04CACHE MEMORY

\text{KV} = 2 \cdot L \cdot h_{kv} \cdot d_{\text{head}} \cdot n \cdot b

L{=}32,\ d_{\text{head}}{=}128,\ b{=}2

Memory grows with every token.

05PREFILL · DECODE

\text{prefill} \propto n^{2} \qquad \text{decode} \propto n \text{ per token}

Prompt in parallel. Answer one token at a time.