How does AI turn words into numbers? From one-hot to semantic space
8 min readComputers only understand numbers. To a GPU, the word "cat" is meaningless—it cannot participate in matrix multiplication, gradient descent, or any mathematical operation. But if we represent "cat" as [0.23, -0.81, 0.45, ...], everything becomes computable.
This is the core task of an embedding: mapping discrete text symbols into a continuous numerical space. With numerical representations, AI can compute distances, directions, and relationships between words.
Type any word below to see it converted into a simulated embedding vector (consistent per word):
Like giving every person a GPS coordinate—once you have coordinates, you can compute distance and direction.
The simplest approach is one-hot encoding: with a vocabulary of N words, represent each word as a vector of length N with exactly one 1 and the rest 0s.
Problem 1: Extremely sparse. Modern models have vocabularies of 50,000 to 100,000+ tokens. A one-hot vector is 99.99% zeros—a massive waste of memory and compute.
Problem 2: No semantics. In one-hot space, "cat" and "dog" are just as far apart as "cat" and "rock"—there is no concept of similarity.
See the one-hot encoding for 5 words—click any row to highlight it:
One-hot is like assigning student IDs—student #1001 and #1002 are not "more similar" than #1001 and #9999.
In 2013, Word2Vec introduced a revolutionary idea: words appearing in similar contexts should have similar vector representations. By training on massive text corpora, the model automatically learned to place semantically related words close together in vector space.
The most famous discovery was vector arithmetic:
This means the model learned a "gender" direction in vector space! Modern Transformers go further—embeddings are trained end-to-end as the first layer of the model, giving each token a dense vector (e.g., 768 or 4096 dimensions).
2D projection showing parallel relationships between word vectors. Click word pairs to see vector arithmetic:
Word2Vec discovered that words with similar meanings live in the same "neighborhood" of semantic space.
Real embedding vectors live in 768 to 4096-dimensional space—far beyond what humans can directly visualize. But the core principle remains: similar meanings are close, different meanings are far apart.
Each dimension captures some abstract feature—no human-readable label, but together they encode rich semantic information. Think of it as a super coordinate system where each axis records some semantic property of the word.
2D projection of word vectors showing natural clusters. Hover to see distances:
In real models this is 4096-dimensional space, but the clustering principle is exactly the same—animals, vehicles, and food each form tight "neighborhoods."
High-dimensional space is like a super library—each book's position encodes hundreds of attributes: topic, language, difficulty, and more.
Computers cannot understand text directly. Embeddings map discrete symbols into continuous numerical vectors.
One-hot encoding wastes space and loses all semantic information—every word is equidistant.
Learned embeddings place words with similar meanings close together in vector space.
Every token passes through an embedding layer to get a dense vector—the model's first step to understanding language.
Embeddings are AI's first step to understanding the world—mapping everything into mathematical space, measuring meaning by distance.