# What Is KV Cache Tiering and When Does It Help?

> KV cache tiering keeps the KV cache that overflows GPU memory in host RAM and external flash, then reads it back instead of recomputing. Structure, gains and limits.

URL: https://vuke.top/en/insights/what-is-kv-cache-tiering · Updated November 12, 2026

Published: 2026-11-12

**Question**: What is KV cache tiering?

**Short answer**: KV cache tiering stores the KV cache that overflows GPU memory during LLM inference in host RAM and then in external all-flash storage, so a returning session reads it back instead of making the GPU recompute the whole context. Measured on a 480B model against full recompute without external storage, time to first token at p50 falls from 149.5 s to 11.85 s [R2](https://vuke.top/en/evidence/R2).

## Why tiering is needed

For every token an LLM processes it leaves a set of key and value tensors at each attention layer: the KV cache. With it, generating the next token does not require recomputing the preceding context. The longer the context, the larger the cache.

GPU memory has to hold the model weights and the KV cache of every active session. As sessions accumulate, older cache is evicted. When an evicted session sends another request, the model must prefill the whole context again and time to first token climbs sharply.

The idea behind tiering is simple: evicted KV cache is not thrown away, it is kept somewhere else.

## What each tier holds

Cache is placed on three media according to how hot it is. The inference framework manages the first two tiers; the third needs a dedicated storage device.

- GPU memory: sessions that are active now, managed by vLLM.
- Host RAM: sessions used recently, managed by LMCache.
- External all-flash storage: all historical sessions, which several inference instances can share.

## How large the measured gain is

The system under test is Qwen3-Coder-480B-FP8 in the TP=8 production topology on 8 × AMD MI308X, under a long-context cold-recovery load. There are two baselines and they have to be read separately.

Against full recompute without external storage: 8.6–20× faster [R2](https://vuke.top/en/evidence/R2).

Against a single local NVMe drive: 29–40% more throughput and 26–32% lower TTFT at p50 [R2](https://vuke.top/en/evidence/R2) [R3](https://vuke.top/en/evidence/R3). In other words, even when the KV cache is stored and read back either way, an external all-flash array is faster than one drive inside the server.

## When it does not apply

The gain comes from cache hits. In short, low-concurrency conversations, sessions are rarely evicted and rarely return, so this tier has little to read back. Workloads that recover sessions constantly, such as long contexts, multi-turn chat, agents and coding assistants, are where the gain shows.

That is why you should measure with your own session-length distribution and concurrency before deciding to buy.

## Related questions

### Does KV cache tiering require rewriting the inference engine?

No rewrite is needed. The measurements use the open-source vLLM + LMCache combination plus one LMCache parallel-read patch; the patch, orchestration scripts and raw data are packaged as R8 and can be rerun on your own cluster [R8](https://vuke.top/en/evidence/R8).

### How does KV cache tiering relate to prefix caching?

Prefix caching decides which KV cache later requests can reuse; tiering decides where that KV cache lives. They work together: after a prefix hit, tiering makes sure the matching cache can still be read back from RAM or external storage even if it has already left GPU memory.

## Related test reports

- [R2](https://vuke.top/en/evidence/R2) · FX100 KV Cache Benchmark (480B, TP8, Long Context) (8 × AMD Instinct MI308X · ROCm 7.2)
- [R3](https://vuke.top/en/evidence/R3) · FX100 KV Cache Benchmark Summary (480B, TP4×2, All Metrics) (8 × AMD Instinct MI308X · ROCm 7.2)
- [R8](https://vuke.top/en/evidence/R8) · FX100 KV Cache Test Code Export and Raw Benchmark Data (AMD Instinct MI308X · ROCm 7.2)

## External references

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180) · arXiv (SOSP '23). vLLM's PagedAttention memory management and the motivation for paging the KV cache
- [LMCache (GitHub repository)](https://github.com/LMCache/LMCache) · LMCache Project. LMCache source code, architecture and storage backends
- [vLLM Documentation](https://docs.vllm.ai/) · vLLM Project. Official documentation of vLLM deployment, prefix caching and tensor-parallel settings