Why are tokens cheaper in long conversations?
About 10 min readWhen each token enters the Attention layer, it is multiplied by two fixed weight matrices to produce a Key and a Value vector — these two vectors are what gets cached.
sin(30°) = 0.5 — once known, there's no need to recompute the trig function every time. KV Cache stores these "known results" in a lookup table for reuse.
Tokens with a matching prefix are read directly from cache; only new tokens need to be computed.
The model sees just a stream of tokens — it doesn't distinguish who said what. All tokens need K/V, and all are stored in cache. As conversation turns increase, the hit rate keeps climbing.
Cache hit = skip K/V matrix multiplication. Compute drops from O(N) to approximately O(1). Providers discount cached tokens accordingly, typically around 10% of the normal price.
It's not "a single matrix" — it's multiple sets of vectors stored separately by layer and attention head.
Each layer stores K/V independently
Multi-head attention, each head computes independently
Longer conversations use more memory
Usually fixed, determined by model architecture
📐 Actual size: 2 × 32 layers × 32 heads × 1000 tokens × 128 dims × 2 bytes (fp16) ≈ 500 MB — memory usage grows linearly with context length
K/V are outputs of a deterministic function. Both inputs (token embedding + weights) are fixed, so the result is always the same and can be safely reused.
Cache hit = skip matrix multiplication, saving GPU compute. Providers discount cached tokens at roughly 10% of the normal price.
Not an approximation — it's bit-for-bit identical floating-point values. Exactly the same as recomputation, zero error, no impact on output quality.
Both user messages and model replies are cached. The model doesn't distinguish who said what — all tokens need K/V and are treated equally.
A 4D tensor [L layers × H heads × N tokens × d dims]. Longer context means more memory — a tradeoff between memory and compute.
Put stable content (System Prompt / documents) at the beginning, append changing content at the end — this maximizes prefix cache hits.
KV Cache = trading memory for compute time · the higher the hit rate, the cheaper the tokens