01NEURON

A NEURON RECEIVES MULTIPLE INPUTS, WEIGHTS THEM, ADDS A BIAS, AND PRODUCES A SINGLE ACTIVATION VALUE.

02ACTIVATION

THE WEIGHTED SUM PASSES THROUGH A NONLINEAR ACTIVATION FUNCTION, INTRODUCING EXPRESSIVITY.

03SOFTMAX

THE MODEL PRODUCES A LOGIT FOR EACH VOCABULARY TOKEN. SOFTMAX CONVERTS LOGITS INTO A PROBABILITY DISTRIBUTION, AND THE NEXT TOKEN IS CHOSEN FROM THE HIGHEST PROBABILITY.

x_1

x_2

x_3

x_n

1

\vdots

w_1

w_2

w_3

w_n

b

\Sigma

z

z = \sum_{i=1}^{n} w_i x_i + b

z

y

\sigma(\cdot)

y = \sigma\left(\sum_{i=1}^{n} w_i x_i + b\right)

logits

z_i

probabilities

p_i

\mathrm{softmax}(z_i)

p_i = \dfrac{\exp(z_i)}{\sum_j \exp(z_j)}

next token

\texttt{=\,argmax}(p_i)

04ACTIVATION FUNCTIONS

y

z

10-1-3-2-10123Sigmoid

\sigma(z)

Tanh

\tanh(z)

ReLU

\max(0, z)

\sigma(z) = \dfrac{1}{1 + e^{-z}}

(sigmoid)

\tanh(z) = \dfrac{e^{z} - e^{-z}}{e^{z} + e^{-z}}

(tanh)

f(z) = \max(0, z)

(ReLU)

05FROM LOGITS TO NEXT TOKEN

Logits and softmax probabilities of the next token (the next token is “the”)
token	logit z	probability p
the	2.2	0.40
cat	1.1	0.14
sat	0.5	0.07
on	-0.1	0.04
a	0.0	0.05
mat	1.8	0.27
dog	-1.2	0.01
runs	-0.7	0.02

Neuron,Activation,Softmax

From Inputs to Next Token

A VISUAL SEQUENCE SHOWING HOW A NEURON TRANSFORMS INPUTS THROUGH A WEIGHTED SUM AND ACTIVATION, AND HOW LOGITS BECOME PROBABILITIES VIA SOFTMAX TO SELECT THE NEXT TOKEN.

LOGITS $z_i$
SOFTMAX $p_i$
NEXT TOKEN

INPUTS
$x_i$
WEIGHTED SUM
$z = \sum_{i} w_i x_i + b$
ACTIVATION
$y = \sigma(z)$
LOGITS
$z_i$
SOFTMAX
$p_i = \dfrac{e^{z_i}}{\sum_j e^{z_j}}$
NEXT TOKEN
$t = \mathrm{argmax}(p_i)$

01/NEURON

02/ACTIVATION

03/SOFTMAX

04/ACTIVATION FUNCTIONS

05/FROM LOGITS TO NEXT TOKEN