ATTENTION CONNECTS TOKENS.

EACH TOKEN LOOKS AT OTHERS AND COMBINES INFORMATION USING WEIGHTED RELATIONSHIPS.

ATTENTION AS WEIGHTED CONNECTIONS.

THICKER LINES REPRESENT HIGHER ATTENTION WEIGHTS. EACH TOKEN PRODUCES QUERIES, KEYS AND VALUES TO COMPUTE ATTENTION SCORES.

01TOKENS

The

cat

sat

the

mat

A SEQUENCE OF TOKENS REPRESENTED AS VECTORS.

X = [x_1,\ x_2,\ \ldots,\ x_n]

n = 7

02Q, K, V

X

W^{Q}

W^{K}

W^{V}

Q

K

V

EACH TOKEN IS PROJECTED INTO QUERIES, KEYS AND VALUES.

Q = X\,W^{Q} \quad K = X\,W^{K}

V = X\,W^{V}

03ATTENTION WEIGHTS

The

cat

sat

the

mat

A = \mathrm{softmax}\left(\dfrac{QK^{T}}{\sqrt{d_k}}\right)

The

cat

sat

the

mat

The cat sat on the mat .

1.0

0.0

04PER-HEAD VIEW

Head 1

Head 2

Head 3

05WITH POSITION

The

cat

sat

the

mat

POSITIONAL ENCODING (SINE WAVES)

f = 1

f = 2

f = 4

f = 8

f = 16

The

cat

sat

the

mat

ADD POSITIONAL INFORMATION USING SINUSOIDAL ENCODING.

PE(pos,\ 2i) = \sin\left(pos\,/\,10000^{2i/d_{\text{model}}}\right)

PE(pos,\ 2i+1) = \cos\left(pos\,/\,10000^{2i/d_{\text{model}}}\right)

Attention &Position

Connecting Meaning and Order

ATTENTION LETS TOKENS EXCHANGE INFORMATION THROUGH WEIGHTED RELATIONSHIPS. POSITIONAL ENCODING GIVES TOKENS A SENSE OF ORDER USING SINE WAVES OF DIFFERENT FREQUENCIES.

TOKENS (INPUT)

EMBEDDINGS

X \in \mathbb{R}^{n \times d}

+ POSITION

X + PE

ATTENTION

A = \mathrm{softmax}

\left(QK^{T} / \sqrt{d_k}\right)

OUTPUT

Z \in \mathbb{R}^{n \times d}

ATTENTION WEIGHT

1.0

0.5

0.1

TOKEN

Current focus

Other token

MOTION NOTES

Build sequence
Increase weights
Add position(waves)
Combine (final)

ATTENTION CONNECTS TOKENS.

01/TOKENS

02/Q, K, V

03/ATTENTION WEIGHTS

04/PER-HEAD VIEW

05/WITH POSITION