Question
After KV cache is evicted from GPU memory, is it faster to recompute or to read it back from external storage?
Short answer
Reading back is far faster. For Qwen3-Coder-480B-FP8 under a long-context cold-recovery load at concurrency 16, full recompute without external storage gives a TTFT of 149.48 s at p50 and 4.1 tok/s; keeping the KV cache on FX100 and reading it back gives 11.85 s and 74.9 tok/sR2.
How the test was run
The model is Qwen3-Coder-480B-FP8 deployed as one 8-GPU instance (TP=8), the production topology, on 8 × AMD MI308X. The load is long-context cold recovery: a session's KV cache is no longer in GPU memory, so an incoming request must either recompute it or read it back.
Three setups were compared under the same load: the FX100 all-flash array, a single local NVMe drive, and full recompute without external storage. FX100 and the local drive were measured at concurrency 8, 16 and 32; full recompute was measured at concurrency 16 onlyR2.
Three readings at concurrency 16
Two baselines that must not be mixed
A speed-up depends on what you compare with. This test has two denominators.
Against full recompute without external storage, the speed-up is 8.6–20×R2. That figure answers whether the KV cache should be kept at all.
Against a single local NVMe drive, throughput rises by 29–40% and TTFT at p50 falls by 26–32%R2R3. That figure answers where to keep it. Folding the two into one sentence would suggest that an external array is an order of magnitude faster than a local drive, which is not what was measured.
Whether the figures can be reproduced
Two independent runs at the same load level deviate by no more than 5%R3. The conclusions for both deployment topologies were compiled into an official report by a third-party testing bodyR4.
The LMCache parallel-read patch, the load client, the orchestration scripts and the raw data are packaged as R8 and can be rerun on your own clusterR8. Your model, context length and concurrency differ, and so will your readings; this data shows the order of magnitude and the direction.
Related test reports
- R2FX100 KV Cache Benchmark (480B, TP8, Long Context)Qwen3-Coder-480B-FP8 deployed as one 8-GPU instance (TP=8), the standard production topology, under a long-context cold-recovery load. A three-way comparison of FX100, a single local NVMe drive and full recompute without external storage, stepped across concurrency 8 / 16 / 32. Most throughput and TTFT figures quoted on this site come from this report.
8 × AMD Instinct MI308X · ROCm 7.2 - R3FX100 KV Cache Benchmark Summary (480B, TP4×2, All Metrics)The same 8-GPU server redeployed as two 4-GPU instances (TP4×2), with every metric recorded across nine load levels: TTFT p50 / p90 / p99, TPOT, throughput and disk bandwidth. The report also verifies two things: the fs:// shared pool lets both instances hot-share one cache, and two independent runs at the same level deviate by no more than 5%.
8 × AMD Instinct MI308X · ROCm 7.2 - R4FX100 KV Cache Benchmark (480B, Multi-Instance, No. -006)The official compilation of R2 and R3, issued by a third-party testing organization as report No. -006. If your procurement or compliance process needs a document from an independent body, this is the one to cite.
8 × AMD Instinct MI308X · ROCm 7.2 - R8FX100 KV Cache Test Code Export and Raw Benchmark DataEverything a third party needs to rerun the tests: the LMCache parallel-read patch (git patch plus full before/after sources), the load client, orchestration scripts, forensic probe scripts and the raw benchmark data.
AMD Instinct MI308X · ROCm 7.2
External references
- 01
- 02vLLM Documentation · vLLM Project
Official documentation of vLLM deployment, prefix caching and tensor-parallel settings
Further reading
What Is KV Cache Tiering and When Does It Help?
KV cache tiering keeps the KV cache that overflows GPU memory in host RAM and external flash, then reads it back instead of recomputing. Structure, gains and limits.
SourceR2R3R8
Local NVMe or an NVMe-oF Array for KV Cache?
Both avoid full recompute. Across three concurrency levels on a 480B model, an external FX100 array gives more throughput and lower TTFT than one local NVMe drive.
SourceR1R2R3
How to Verify a Vendor's LLM Inference Benchmarks
Five checks for vendor inference benchmarks: baseline, platform and topology, run-to-run deviation, third-party report, reproduction material. Plus common traps.
SourceR2R3R4R5R8R9