Question
Should KV cache be kept on a local NVMe drive in the server or on an external NVMe-oF all-flash array?
Short answer
Both avoid full recompute, but under heavy concurrent read-back the external array is faster: measured on a 480B model at TP=8, FX100 delivers 29–40% more throughput and 26–32% lower TTFT at p50 than a single local NVMe driveR2. An external array also lets several inference instances share one cacheR3.
How the comparison was set up
The local setup is one NVMe drive (PCIe Gen4, 2 TB). The external setup is the FX100 all-flash NVMe-oF array: a 4-drive RAID0 pool (14 TB, XFS) reached over RoCEv2 on one 100 GbE portR2.
Note that this is not a like-for-like drive count; the baseline is a single drive. If your servers already have a local multi-drive array, the gap will differ from the one here and has to be measured on your own configuration.
Readings at each concurrency level
The model is Qwen3-Coder-480B-FP8 on 8 × AMD MI308X under a long-context cold-recovery load. The value left of each arrow is the single local NVMe drive; the value on the right is FX100.
Why going over the network is faster
Intuition says a local drive sits closer to the GPU. But time to first token is not decided by the small extra protocol latency; it is decided by whether data can be returned to GPU memory fast enough under concurrent read-back. A multi-drive all-flash pool on an RDMA network offers more concurrent read bandwidth than one local drive.
Software matters too: the LMCache parallel-read patch raises single-GPU cold-read bandwidth from 0.98 GB/s to 5.23 GB/sR1. Without it, a serial read path holds back even a fast drive.
Two things an external array adds
One is sharing. Cache on a local drive is usable only by that server; cache in an external pool can be read by a session scheduled onto any instance. Cross-instance hot sharing through the fs:// shared pool has been verifiedR3.
The other is sizing. The production basis is one FX100 per 8 inference nodes, grounded in the measured anchor that 100 GbE sustains KV read-back for 8 nodesR1R2R3.
When a local drive is enough
For a single server with modest concurrency and sessions that never migrate between instances, a local NVMe drive already captures the largest gain, which is not recomputing. The advantage of an external array appears when concurrent read-back pressure is high or several instances need to share a cache.
Related test reports
- R1FX100 Comprehensive LLM Inference and Training Benchmark (8 × AMD MI308X)One report, five questions: how much the LMCache parallel-read patch helps, whether KV tiering scales across GPUs, how much faster 8 GPUs load models concurrently, how effective utilization changes under frequent model switching, and how much faster concurrent training checkpoint writes become. Test models are Qwen2.5-32B and 7B.
8 × AMD Instinct MI308X · ROCm 7.2 - R2FX100 KV Cache Benchmark (480B, TP8, Long Context)Qwen3-Coder-480B-FP8 deployed as one 8-GPU instance (TP=8), the standard production topology, under a long-context cold-recovery load. A three-way comparison of FX100, a single local NVMe drive and full recompute without external storage, stepped across concurrency 8 / 16 / 32. Most throughput and TTFT figures quoted on this site come from this report.
8 × AMD Instinct MI308X · ROCm 7.2 - R3FX100 KV Cache Benchmark Summary (480B, TP4×2, All Metrics)The same 8-GPU server redeployed as two 4-GPU instances (TP4×2), with every metric recorded across nine load levels: TTFT p50 / p90 / p99, TPOT, throughput and disk bandwidth. The report also verifies two things: the fs:// shared pool lets both instances hot-share one cache, and two independent runs at the same level deviate by no more than 5%.
8 × AMD Instinct MI308X · ROCm 7.2
External references
- 01NVM Express Specifications · NVM Express, Inc.
The official specifications of NVMe and NVMe over Fabrics
- 02RFC 5040: A Remote Direct Memory Access Protocol Specification · IETF
The normative definition of the RDMA protocol
- 03
Further reading
Recompute or Read Back? KV Cache on a 480B Model
Full recompute versus reading KV cache back from FX100 on a 480B model at concurrency 16: time to first token, throughput, test conditions and how to reproduce.
SourceR2R3R4R8
What Is KV Cache Tiering and When Does It Help?
KV cache tiering keeps the KV cache that overflows GPU memory in host RAM and external flash, then reads it back instead of recomputing. Structure, gains and limits.
SourceR2R3R8
How to Accept an Inference Storage Project: 4 Gates
Split acceptance of an inference storage project into four gates on a weekly schedule: delivery, single-node baseline, main joint-test gate and stability sign-off.
SourceR1R2R3R8