FROM TEXT TO TOKENS TO EMBEDDINGS.

A KEY-FRAME SEQUENCE SHOWING HOW A PROMPT BECOMES TOKENS AND THEN A DISTRIBUTED REPRESENTATION IN VECTOR SPACE.

What isartificialintelligence?Wisai?technologyintelligencelanguagequestionxyz

TEXT BECOMES A SEQUENCE OF TOKEN VECTORS IN A HIGH-DIMENSIONAL SPACE.

MEANING EMERGES FROM GEOMETRY. SIMILAR TOKENS OCCUPY NEARBY REGIONS IN EMBEDDING SPACE.

t = 0.0s

A TYPED QUESTION AS A RAW STRING.

text = "What is artificial intelligence?"len(text) = 32 characters

t = 0.8s

THE TEXT BREAKS INTO DISCRETE TOKENS.

tokens = ["What", "is", "artificial",
          "intelligence", "?"]n_tokens = 5

t = 1.6s

EACH TOKEN MAPS TO A VOCABULARY ID.

vocab_id = [3209, 374, 6273, 1936, 30]token_index = [0, 1, 2, 3, 4]

t = 2.4s

TOKENS LIFT INTO D-DIMENSIONAL VECTORS.

x_i \in \mathbb{R}^d \qquad d_{\text{model}} = 768

X = [x_0,\ x_1,\ x_2,\ x_3,\ x_4] \qquad X \in \mathbb{R}^{5 \times 768}

t = 3.2s

TOKENS OCCUPY A SEMANTIC SPACE WHERE SIMILAR MEANINGS CLUSTER.

x_i \in \mathbb{R}^d

(e.g. d = 768)

\mathtt{similarity}(i,\ j) = \mathtt{cos}(x_i,\ x_j) = \dfrac{x_i \cdot x_j}{\lVert x_i \rVert\ \lVert x_j \rVert}

t = 3.2s

SIMILAR TOKENS HAVE HIGHER COSINE SIMILARITY.

S_{ij} = \cos(x_i,\ x_j)

S \in \mathbb{R}^{5 \times 5} \qquad S_{ij} \in [-1,\ 1]

Whatisartificialintelligence?Whatisartificialintelligence?technologyintelligencelanguagez (PC3)xx (PC1)y (PC2)

TOKENS LIFT INTO D-DIMENSIONAL VECTORS.

x_i \in \mathbb{R}^d \qquad d_{\text{model}} = 768

X = [x_0,\ x_1,\ x_2,\ x_3,\ x_4] \qquad X \in \mathbb{R}^{5 \times 768}

t = 3.2s

LATENTFrom Text to Tokens to Embeddings

A KEY-FRAME SEQUENCE SHOWING HOW A QUESTION IS TOKENIZED AND TRANSFORMED INTO A GEOMETRIC REPRESENTATION IN A HIGH-DIMENSIONAL SPACE.

TEXT
(sequence)

TOKENS
(discrete)

EMBEDDINGS
(vectors)

What is artificialintelligence?

V = \{\, 0,\ 1,\ \ldots,\ |V| - 1 \,\}

|V| \approx 50{,}000

x \in \mathbb{R}^d

d_{\text{model}} = 768

x = E[\texttt{token\_id}]

\cos(x_i,\ x_j) = \dfrac{x_i \cdot x_j}{\lVert x_i \rVert\ \lVert x_j \rVert}

N(x) = \{\, x_j \mid \cos(x,\ x_j) > \tau \,\}