The two phases of LLM text generation: "understanding" and "outputting"
10 min read
When you ask a large language model a question, it doesn't just "think and respond" -- it actually goes through two distinctly different phases. It's like taking an exam: first read the question, then write the answer.
The first phase is called Prefill, where the model "reads" all of your input at once. The second phase is called Decode, where the model begins "writing" its response one token at a time.
During the Prefill phase, the model processes all your input tokens in parallel. This means whether your prompt has 10 tokens or 1,000 tokens, they are all "understood simultaneously" -- like a GPU performing thousands of matrix multiplications at once.
During this process, the model computes a pair of vectors for each token: K (Key) and V (Value), which are stored in the KV Cache. This cache is used repeatedly during the subsequent Decode phase to avoid redundant computation.
Input tokens (all light up simultaneously after clicking play):
The computational cost of Prefill scales quadratically with input length (due to the Attention mechanism), so longer prompts significantly increase Prefill time. The good news: these computations are highly parallelizable, allowing GPUs to fully utilize their compute power.
After Prefill completes, the model enters the Decode phase. Here the model generates tokens one at a time: each newly generated token is fed back into the model as input to produce the next token. This is known as auto-regressive generation.
The key advantage: when generating a new token, the model doesn't need to reprocess all previous tokens -- it directly reads the information already stored in the KV Cache. This means only one forward pass is needed each time.
The bottleneck of the Decode phase isn't computation -- processing just 1 token at a time requires very little compute -- but rather memory bandwidth. For each generated token, the entire model's weights must be read from memory into the compute units. The larger the model, the slower the read.
Now that we understand the two phases, we can understand two key performance metrics:
TTFT (Time To First Token) -- the wait time from sending a request to seeing the first output token. This is primarily determined by the Prefill phase. The longer the prompt, the larger the TTFT.
TPS (Tokens Per Second) -- the speed at which the model outputs text. This is determined by the per-token generation time in the Decode phase and is largely independent of prompt length.
Drag the slider to change prompt length and observe how TTFT changes:
Prefill and Decode face entirely different hardware bottlenecks. Understanding this explains why different hardware architectures perform differently across the two phases.
Massive parallel matrix operations
GPU running at full capacity
Only processes 1 token at a time
but must read all model weights
Processes all input tokens in parallel and builds the KV Cache. Compute-intensive, GPU running at full load.
Auto-regressive token-by-token generation, reusing the KV Cache. Memory bandwidth-intensive, low compute utilization.
TTFT depends on Prefill speed, TPS depends on Decode speed. They require different optimization strategies.
KV Cache connects the two phases, trading space for time and avoiding redundant computation during Decode.
"Understanding Prefill and Decode explains why LLMs 'wait first, then stream fast' -- and how to optimize them."
Next, we'll explore how ANE uses hybrid inference to accelerate Prefill by 11.3x