Glossary
What Is KV cache?
The KV cache is the intermediate state an LLM keeps during inference: the key and value tensors computed at each attention layer for every token already processed. With it, the model does not have to recompute the preceding context when generating the next token. The longer the context, the more GPU memory the KV cache occupies.
Updated
In the measurements
When GPU memory runs out, KV cache is evicted. A returning session must then either be recomputed in full or read back from external storage. In measurements on a 480B model, reading back is 8.6–20× faster than recomputingR2.
Related test reports
Nearby terms
- KV cache tiering
KV cache tiering places KV cache on different media according to how hot it is: GPU memory holds active sessions, host RAM holds recent ones, and external all-flash storage holds all historical sessions and shares them across instances. The first two tiers are managed by the inference framework; the third needs a dedicated storage device.
Test jointly first, decide afterwards
Test jointly first, decide afterwards
Every engagement starts with a gated joint test of about 10 weeks. Four gates are passed in sequence; if one is missed, the project stops there, so you never pay for a benefit that was not demonstrated.
- Contact
- Wang Limin
- Phone
- +86 131 6181 8898
- mingxin@agentmail.to
The inbox is monitored and triaged by AI around the clock