Glossary
What Is Prefill?
Prefill is the first phase of inference: the model processes all input tokens at once and builds their KV cache. The longer the input, the longer prefill takes, which is the direct reason long-context requests have a high time to first token. The following phase, which generates output tokens one at a time, is called decode.
Updated
In the measurements
Every hit on an external KV cache saves one full prefill. That is the fundamental reason KV cache tiering lowers TTFT.
Nearby terms
- TPOT (time per output token)
TPOT (time per output token) is the average time taken to generate each token once output has started. TTFT measures how long before the model starts speaking, TPOT how fast it speaks; together they describe the full latency of a request.
- Cold recovery
Cold recovery is the case where a session's KV cache is no longer in GPU memory and its context state must be restored when the next request arrives. There are two ways to restore it: read the KV cache back from external storage, or have the GPU recompute the whole context. Many sessions recovering at once is often called a session-recovery storm.
Test jointly first, decide afterwards
Test jointly first, decide afterwards
Every engagement starts with a gated joint test of about 10 weeks. Four gates are passed in sequence; if one is missed, the project stops there, so you never pay for a benefit that was not demonstrated.
- Contact
- Wang Limin
- Phone
- +86 131 6181 8898
- mingxin@agentmail.to
The inbox is monitored and triaged by AI around the clock