Glossary
What Is Prefix caching?
Prefix caching is the mechanism by which an inference engine reuses the KV cache of an identical input prefix: when two requests start the same way (the same system prompt or the same document), the second can skip computing that part. The prefix caching built into vLLM works only within GPU memory.
Updated
In the measurements
An external KV cache extends the mechanism beyond GPU memory: evicted prefixes can still be hit, and they can be shared across instancesR3.
Related test reports
Nearby terms
- Cold recovery
Cold recovery is the case where a session's KV cache is no longer in GPU memory and its context state must be restored when the next request arrives. There are two ways to restore it: read the KV cache back from external storage, or have the GPU recompute the whole context. Many sessions recovering at once is often called a session-recovery storm.
- LMCache
LMCache is an open-source KV cache library that plugs into vLLM. It moves KV cache that does not fit in GPU memory down to host RAM and external storage and reads it back on a hit. It is a mainstream open-source implementation of KV cache tiering.
Test jointly first, decide afterwards
Test jointly first, decide afterwards
Every engagement starts with a gated joint test of about 10 weeks. Four gates are passed in sequence; if one is missed, the project stops there, so you never pay for a benefit that was not demonstrated.
- Contact
- Wang Limin
- Phone
- +86 131 6181 8898
- mingxin@agentmail.to
The inbox is monitored and triaged by AI around the clock