LATENT
HOW AI WORKS - FRAME BOOK
VOL. 01·SEP 2026

WORDS STEER THE NOISE.

TEXT-TO-IMAGE MODELS START FROM NOISE, AS IN 009. EVERY STEP OF DENOISING LOOKS BACK AT THE PROMPT.

aglowingwiresphere,lineartCLIPc\mathbf{c}LATENTzt∈R64×64×4z_t \in \mathbb{R}^{64 \times 64 \times 4}STEP50 / 5050 / 50STEP 50 / 5050 / 50zTz_Tz0z_001020304050

PROMPT IN. PICTURE OUT.

EACH REGION OF THE LATENT ASKS WHICH WORDS MATTER, THROUGH CROSS-ATTENTION.

STABLE DIFFUSION 1.X

prompt = 77 tokens

latent = 64 × 64 × 4

image = 512 × 512 × 3

steps = 20–50

w = 7.5

01PROMPT → VECTORS

77 × 768
c=Etext(prompt)∈R77×768\mathbf{c} = E_{\text{text}}(\text{prompt}) \in \mathbb{R}^{77 \times 768}

The prompt becomes 77 context vectors.

a glowing wire sphere, line art⟨start⟩aglowingwiresphere,lineart⟨end⟩… ×68CLIP TEXT ENCODERPROMPT (9)PADDING (68)

02LATENT SPACE

DOWNSAMPLE 8×
z=E(x),x≈D(z)z = \mathcal{E}(x), \quad x \approx \mathcal{D}(z)
786,432→16,384  (48×)786{,}432 \to 16{,}384 \ \ (48\times)

Denoise a small latent, not pixels.

xx512 × 512 × 3E\mathcal{E}D\mathcal{D}zz64 × 64 × 4

03CROSS-ATTENTION

softmax(QKT/d) V\mathrm{softmax}(QK^{T}/\sqrt{d})\,V
Q=WQ φ(zt),  K=WK c,  V=WV cQ = W_Q\,\varphi(z_t),\ \ K = W_K\,\mathbf{c},\ \ V = W_V\,\mathbf{c}

Image regions query the prompt's words.

glowingwiresphereartQ: FROM THE LATENT · K, V: FROM THE WORDS

04DENOISER

U-NET (SD 1.X)
ϵ^=ϵθ(zt, t, c)\hat{\epsilon} = \epsilon_\theta(z_t,\ t,\ \mathbf{c})
zt−1=step(zt, ϵ^)z_{t-1} = \mathrm{step}(z_t,\ \hat{\epsilon})

Predict the noise, remove a little.

ztz_tϵ^\hat{\epsilon}c\mathbf{c}tt64264^232232^216216^2828^2

05GUIDANCE

w = 7.5
ϵ^=ϵ∅+w (ϵc−ϵ∅)\hat{\epsilon} = \epsilon_{\varnothing} + w\,(\epsilon_{\mathbf{c}} - \epsilon_{\varnothing})

Two guesses; lean toward the prompt.

ϵ∅\epsilon_{\varnothing}ϵc\epsilon_{\mathbf{c}}ϵ^\hat{\epsilon}7.514w = 1w = 7.5w = 14

06DECODE

64 → 512
x=D(z0)∈R512×512×3x = \mathcal{D}(z_0) \in \mathbb{R}^{512 \times 512 \times 3}

One decoder pass: latent to pixels.

z0z_064 × 64 × 4D\mathcal{D}512 × 512 × 3

Text-to-Image

Guiding Diffusion with Words

A TEXT ENCODER TURNS THE PROMPT INTO VECTORS. A DENOISER CLEANS A SMALL LATENT STEP BY STEP, READING THOSE VECTORS EVERY TIME, AND A DECODER PAINTS THE PIXELS.

zTz_TNOISEztz_tz0z_0LATENTxxIMAGEPROMPTc\mathbf{c}D\mathcal{D}
TEXT ENCODER
c=Etext(prompt)\mathbf{c} = E_{\text{text}}(\text{prompt})
CLIP, AS IN 019
LATENT (VAE)
z=E(x),x≈D(z)z = \mathcal{E}(x), \quad x \approx \mathcal{D}(z)
TRAINING
L=E ∥ϵ−ϵθ(zt, t, c)∥2\mathcal{L} = \mathbb{E}\,\big\lVert \epsilon - \epsilon_\theta(z_t,\ t,\ \mathbf{c}) \big\rVert^{2}
PROMPT DROPPED ~10% OF THE TIME
GUIDANCE
ϵc=ϵθ(zt, t, c)\epsilon_{\mathbf{c}} = \epsilon_\theta(z_t,\ t,\ \mathbf{c})
ϵ∅=ϵθ(zt, t, ∅)\epsilon_{\varnothing} = \epsilon_\theta(z_t,\ t,\ \varnothing)
TYPICAL w = 7.5
SAMPLING
zT→zT−1→⋯→z0z_T \to z_{T-1} \to \cdots \to z_0
zTz_T = PURE NOISE · 20–50 STEPS
BUILDS ON
← 009 DIFFUSION
← 019 VISION TRANSFORMERS
SAME DENOISING LOOP, NOW STEERED BY WORDS.