Glossary
What Is Cold recovery?
Cold recovery is the case where a session's KV cache is no longer in GPU memory and its context state must be restored when the next request arrives. There are two ways to restore it: read the KV cache back from external storage, or have the GPU recompute the whole context. Many sessions recovering at once is often called a session-recovery storm.
Updated
In the measurements
The 480B benchmarks quoted on this site use exactly this long-context cold-recovery load, the typical stress of agent and coding-assistant workloadsR2.
Related test reports
Nearby terms
- Prefill
Prefill is the first phase of inference: the model processes all input tokens at once and builds their KV cache. The longer the input, the longer prefill takes, which is the direct reason long-context requests have a high time to first token. The following phase, which generates output tokens one at a time, is called decode.
- Prefix caching
Prefix caching is the mechanism by which an inference engine reuses the KV cache of an identical input prefix: when two requests start the same way (the same system prompt or the same document), the second can skip computing that part. The prefix caching built into vLLM works only within GPU memory.
Test jointly first, decide afterwards
Test jointly first, decide afterwards
Every engagement starts with a gated joint test of about 10 weeks. Four gates are passed in sequence; if one is missed, the project stops there, so you never pay for a benefit that was not demonstrated.
- Contact
- Wang Limin
- Phone
- +86 131 6181 8898
- mingxin@agentmail.to
The inbox is monitored and triaged by AI around the clock