Question
In which scenarios does KV cache tiering have no effect?
Short answer
When the cache is rarely hit. In workloads with short conversations, low concurrency or contexts that never repeat, sessions are seldom evicted from GPU memory and seldom return afterwards, so an external KV cache tier has little to read back and the gain is limited. Workloads whose bottleneck is the output phase also benefit little.
Where the gain comes from
KV cache tiering does one thing: after a session's KV cache has been evicted from GPU memory, it reads the cache back when the next request arrives and saves the prefill. Every hit saves one prefill; without a hit nothing is saved.
The large gains in our reports were measured under a long-context cold-recovery load: on a 480B model at concurrency 16, TTFT at p50 for a full recompute is 149.5 sR2. That kind of load is exactly where a hit is worth the most, and it does not stand for every workload.
Four kinds of workload that benefit little
If your business consists mainly of the following, be especially careful before buying.
- Short conversations: the context is short, prefill takes little time anyway, and reading back saves almost nothing.
- Low concurrency: GPU memory holds every active session, there is hardly any eviction, and so there is no cache that needs reading back.
- Contexts that never repeat: batch processing of unrelated documents, for example, where every request is new input with no reusable prefix.
- Output-bound tasks: when the output is long and time to first token is a small share of total time, the experience is decided by the speed of each generated token, which depends on compute.
How to tell which kind you have
There is no need to guess; it can be counted from production logs.
- The distribution of session context length: what share is long context.
- The return interval of sessions: how long users wait before the next turn, and whether the cache is still in GPU memory by then.
- How often KV cache is evicted from GPU memory: more eviction means more potential hits.
- The share of time to first token in total time: the higher it is, the more optimizing it is worth.
Our advice
If the indicators above show a workload dominated by long, multi-turn sessions at high concurrency, this tier is probably worth doing; if not, the budget is better spent elsewhere.
When in doubt, run a small joint test on your own workload first, with the pass conditions written down in advance. If the measurement shows no benefit, stop there. That is more reliable than either side's guess.
Related questions
Which kind are agents and coding assistants?
Usually the kind that benefits clearly. Their contexts are long, they run many turns, and sessions often return after being evicted, which is exactly where a cache hit is worth the most. How much you gain still has to be measured with your own session distribution.
Related test reports
External references
- 01Efficient Memory Management for Large Language Model Serving with PagedAttention · arXiv (SOSP '23)
vLLM's PagedAttention memory management and the motivation for paging the KV cache
- 02SGLang: Efficient Execution of Structured Language Model Programs · arXiv (NeurIPS '24)
Why prefix reuse works for multi-turn chat and shared-prefix workloads
Further reading
What Is KV Cache Tiering and When Does It Help?
KV cache tiering keeps the KV cache that overflows GPU memory in host RAM and external flash, then reads it back instead of recomputing. Structure, gains and limits.
SourceR2R3R8
How to Cut LLM Time to First Token Without More GPUs
With long contexts, high time to first token usually means evicted sessions are being recomputed. Three fixes that need no new GPUs, with results from a 480B model.
SourceR1R2R3
How to Accept an Inference Storage Project: 4 Gates
Split acceptance of an inference storage project into four gates on a weekly schedule: delivery, single-node baseline, main joint-test gate and stability sign-off.
SourceR1R2R3R8