What Is KV cache tiering?

• R2FX100 KV Cache Benchmark (480B, TP8, Long Context)Qwen3-Coder-480B-FP8 deployed as one 8-GPU instance (TP=8), the standard production topology, under a long-context cold-recovery load. A three-way comparison of FX100, a single local NVMe drive and full recompute without external storage, stepped across concurrency 8 / 16 / 32. Most throughput and TTFT figures quoted on this site come from this report.2026-07-05
  8 × AMD Instinct MI308X · ROCm 7.2
• R3FX100 KV Cache Benchmark Summary (480B, TP4×2, All Metrics)The same 8-GPU server redeployed as two 4-GPU instances (TP4×2), with every metric recorded across nine load levels: TTFT p50 / p90 / p99, TPOT, throughput and disk bandwidth. The report also verifies two things: the fs:// shared pool lets both instances hot-share one cache, and two independent runs at the same level deviate by no more than 5%.2026-07-06
  8 × AMD Instinct MI308X · ROCm 7.2

• What Is KV Cache Tiering and When Does It Help?→

  KV cache tiering keeps the KV cache that overflows GPU memory in host RAM and external flash, then reads it back instead of recomputing. Structure, gains and limits.

• When Does KV Cache Tiering Not Help?→

  The gain from KV cache tiering comes from cache hits. Short chats, low concurrency, never-repeated contexts and output-bound workloads benefit little. Here is why.

• KV cache→

  The KV cache is the intermediate state an LLM keeps during inference: the key and value tensors computed at each attention layer for every token already processed. With it, the model does not have to recompute the preceding context when generating the next token. The longer the context, the more GPU memory the KV cache occupies.

• TTFT (time to first token)→

  TTFT (time to first token) is the time from sending a request to receiving the model's first token, and it decides how responsive a service feels. It is dominated by the prefill phase: the model must process the entire input before it can start producing output.

All terms→