Transformer Attention Mechanism

Why Do We Need Attention?

Consider this Chinese sentence:

"小明把苹果给了小红，因为她很饿"

(Xiaoming gave the apple to Xiaohong, because she was hungry)

Who does "she" refer to? You can answer "Xiaohong" without hesitation -- because your brain automatically performed a round of attention focusing: when you see "she", your cognitive system rapidly scans the context and locks onto the most relevant word.

But for traditional word-by-word processing models, this is not so simple. RNNs read sentences sequentially, and by the time they reach "she", the information about "Xiaohong" may have already faded. The revolutionary aspect of the attention mechanism is: it lets the model directly "see" every word in the entire sentence, regardless of distance.

The Q/K/V Trio

The core of attention lies in three vectors: Query (Q), Key (K), and Value (V). Each word generates all three vectors, and each has its own role.

Imagine walking into a library:

• Q (Query) -- The question in your mind: "I want to find books about Beijing cuisine"
• K (Key) -- The label on each book cover: Geography, Cuisine, History...
• V (Value) -- The actual content inside the book

You use Q to match each book's K, find the most relevant ones, and then retrieve the corresponding V to read. It's that simple!

How Are Attention Scores Computed?

With Q, K, and V in hand, attention scores are computed using an elegant formula:

Three steps to understand it:

1. Q · K^T -- Compute the "relevance" between every pair of words (higher dot product = more relevant)
2. ÷ √d_k -- Scaling factor to prevent excessively large values that cause softmax gradient vanishing
3. softmax -- Normalize into a probability distribution (all weights sum to 1)

Finally, use these weights to compute a weighted sum of V, producing a new representation infused with contextual information.

Multi-Head Attention: Understanding from Multiple Angles

A single set of Q/K/V may not be enough -- it's like looking at a problem from only one angle. Transformer uses multiple attention heads (Multi-Head Attention), each learning to focus on different patterns:

• Syntactic Head: Focuses on grammatical relationships like subject-verb and verb-object
• Semantic Head: Focuses on semantic associations (e.g., "Beijing" and "roast duck")
• Positional Head: Focuses on local relationships between neighboring words

Self-Attention: The Full Picture

Now let's put all the pieces together and see how a complete Transformer layer works. It's an elegant "understanding loop" -- first look at context (attention), then think deeply (feed-forward network), continuously deepening comprehension.

Each layer continues to deepen understanding based on the output of the previous layer. GPT-3 has 96 layers, meaning it goes through 96 such "understanding loops" -- no wonder it can comprehend such complex language!

Summary