Question
What is KV cache tiering?
Short answer
KV cache tiering stores the KV cache that overflows GPU memory during LLM inference in host RAM and then in external all-flash storage, so a returning session reads it back instead of making the GPU recompute the whole context. Measured on a 480B model against full recompute without external storage, time to first token at p50 falls from 149.5 s to 11.85 sR2.
Why tiering is needed
For every token an LLM processes it leaves a set of key and value tensors at each attention layer: the KV cache. With it, generating the next token does not require recomputing the preceding context. The longer the context, the larger the cache.
GPU memory has to hold the model weights and the KV cache of every active session. As sessions accumulate, older cache is evicted. When an evicted session sends another request, the model must prefill the whole context again and time to first token climbs sharply.
The idea behind tiering is simple: evicted KV cache is not thrown away, it is kept somewhere else.
What each tier holds
Cache is placed on three media according to how hot it is. The inference framework manages the first two tiers; the third needs a dedicated storage device.
- GPU memory: sessions that are active now, managed by vLLM.
- Host RAM: sessions used recently, managed by LMCache.
- External all-flash storage: all historical sessions, which several inference instances can share.
How large the measured gain is
The system under test is Qwen3-Coder-480B-FP8 in the TP=8 production topology on 8 × AMD MI308X, under a long-context cold-recovery load. There are two baselines and they have to be read separately.
Against full recompute without external storage: 8.6–20× fasterR2.
Against a single local NVMe drive: 29–40% more throughput and 26–32% lower TTFT at p50R2R3. In other words, even when the KV cache is stored and read back either way, an external all-flash array is faster than one drive inside the server.
When it does not apply
The gain comes from cache hits. In short, low-concurrency conversations, sessions are rarely evicted and rarely return, so this tier has little to read back. Workloads that recover sessions constantly, such as long contexts, multi-turn chat, agents and coding assistants, are where the gain shows.
That is why you should measure with your own session-length distribution and concurrency before deciding to buy.
Related questions
Does KV cache tiering require rewriting the inference engine?
No rewrite is needed. The measurements use the open-source vLLM + LMCache combination plus one LMCache parallel-read patch; the patch, orchestration scripts and raw data are packaged as R8 and can be rerun on your own clusterR8.
How does KV cache tiering relate to prefix caching?
Prefix caching decides which KV cache later requests can reuse; tiering decides where that KV cache lives. They work together: after a prefix hit, tiering makes sure the matching cache can still be read back from RAM or external storage even if it has already left GPU memory.
Related test reports
- R2FX100 KV Cache Benchmark (480B, TP8, Long Context)Qwen3-Coder-480B-FP8 deployed as one 8-GPU instance (TP=8), the standard production topology, under a long-context cold-recovery load. A three-way comparison of FX100, a single local NVMe drive and full recompute without external storage, stepped across concurrency 8 / 16 / 32. Most throughput and TTFT figures quoted on this site come from this report.
8 × AMD Instinct MI308X · ROCm 7.2 - R3FX100 KV Cache Benchmark Summary (480B, TP4×2, All Metrics)The same 8-GPU server redeployed as two 4-GPU instances (TP4×2), with every metric recorded across nine load levels: TTFT p50 / p90 / p99, TPOT, throughput and disk bandwidth. The report also verifies two things: the fs:// shared pool lets both instances hot-share one cache, and two independent runs at the same level deviate by no more than 5%.
8 × AMD Instinct MI308X · ROCm 7.2 - R8FX100 KV Cache Test Code Export and Raw Benchmark DataEverything a third party needs to rerun the tests: the LMCache parallel-read patch (git patch plus full before/after sources), the load client, orchestration scripts, forensic probe scripts and the raw benchmark data.
AMD Instinct MI308X · ROCm 7.2
External references
- 01Efficient Memory Management for Large Language Model Serving with PagedAttention · arXiv (SOSP '23)
vLLM's PagedAttention memory management and the motivation for paging the KV cache
- 02
- 03vLLM Documentation · vLLM Project
Official documentation of vLLM deployment, prefix caching and tensor-parallel settings
Further reading
Recompute or Read Back? KV Cache on a 480B Model
Full recompute versus reading KV cache back from FX100 on a 480B model at concurrency 16: time to first token, throughput, test conditions and how to reproduce.
SourceR2R3R4R8
When Does KV Cache Tiering Not Help?
The gain from KV cache tiering comes from cache hits. Short chats, low concurrency, never-repeated contexts and output-bound workloads benefit little. Here is why.
SourceR2
Local NVMe or an NVMe-oF Array for KV Cache?
Both avoid full recompute. Across three concurrency levels on a 480B model, an external FX100 array gives more throughput and lower TTFT than one local NVMe drive.
SourceR1R2R3