# What Is KV cache?

> The KV cache is the intermediate state an LLM keeps during inference: the key and value tensors computed at each attention layer for every token already processed. With it, the model does not have to recompute the preceding context when generating the next token. The longer the context, the more GPU memory the KV cache occupies.

URL: https://vuke.top/en/glossary/kv-cache · Updated November 12, 2026

The KV cache is the intermediate state an LLM keeps during inference: the key and value tensors computed at each attention layer for every token already processed. With it, the model does not have to recompute the preceding context when generating the next token. The longer the context, the more GPU memory the KV cache occupies.

## In the measurements

When GPU memory runs out, KV cache is evicted. A returning session must then either be recomputed in full or read back from external storage. In measurements on a 480B model, reading back is 8.6–20× faster than recomputing [R2](https://vuke.top/en/evidence/R2).

Source: [R2](https://vuke.top/en/evidence/R2)