How KV cache tiering works

KV cache tiering places cache on three media according to how hot it is. The first two tiers are managed by the inference framework; the third needs a dedicated storage device, and that is what the FX series provides.

• Tier one, GPU memory: sessions that are active now. The lowest access latency of the three, and the least capacity.
• Tier two, host RAM: sessions used recently, managed by LMCache.
• Tier three, external flash: all historical sessions, shared by several inference instances.

Every hit on the external cache saves one full prefill. That is the whole reason time to first token goes down.

1. A request arrives and LMCache looks up existing KV cache by the prefix of the input tokens.
2. If it is in GPU memory, it is used directly.
3. If not, host RAM and external flash are checked in turn; on a hit the KV cache is read back into GPU memory and that part of prefill is skipped.
4. Only when no tier has it does the GPU compute it, and the new KV cache is written to the tiers for later use.

The system under test is Qwen3-Coder-480B-FP8 deployed as one 8-GPU instance (TP=8), the production topology, on 8 × AMD MI308X under a long-context cold-recovery load. There are two baselines with different denominators, and they must not be mixed in one sentence.

Against full recompute without external storage: 8.6–20× fasterR2. At concurrency 16, TTFT p50 falls from 149.5 s to 11.85 s.

Against a single local NVMe drive: 29–40% more throughput and 26–32% lower TTFT p50R2R3. In other words, even when the KV cache is stored and read back either way, keeping it on FX100 is faster than keeping it on one NVMe drive inside the server.

Intuition says a local drive sits closer to the GPU and should be faster. But time to first token is not decided by the small extra protocol latency; it is decided by whether data can be returned to GPU memory fast enough under concurrent read-back.

In the measured setup, the baseline is one local NVMe drive (PCIe Gen4, 2 TB), while FX100 is a 4-drive RAID0 all-flash pool reached over RoCEv2 on one 100 GbE portR2. In software, the LMCache parallel-read patch raises single-GPU cold-read bandwidth from 0.98 GB/s to 5.23 GB/sR1.

An external pool also does something a local drive cannot: it lets several inference instances share one cache. Cross-instance hot sharing through the fs:// shared pool has been verifiedR3.

The benefit comes from cache hits. In short, low-concurrency conversations, sessions are rarely evicted and rarely return, so the hit rate is low and this tier adds little.

Long contexts, multi-turn chat, agents and coding assistants recover sessions constantly and benefit the most. Which kind your workload is can only be known by measuring it with your own session-length distribution and concurrency. That is why we insist on testing before buying.

Adoption does not require rewriting the inference engine; it uses the open-source vLLM + LMCache combination. The measured versions are ROCm 7.2 and vLLM 0.20.1+rocm721, with LMCache built from upstream mainline source dated 2026-06-29R2.

The LMCache parallel-read patch we wrote is packaged as R8 together with the load client, orchestration scripts, forensic probe scripts and raw data, so a third party can rerun everything on its own clusterR8.