# Local NVMe or an NVMe-oF Array for KV Cache?

> Both avoid full recompute. Across three concurrency levels on a 480B model, an external FX100 array gives more throughput and lower TTFT than one local NVMe drive.

URL: https://vuke.top/en/insights/nvme-of-vs-local-nvme-for-kv-cache · Updated November 12, 2026

Published: 2026-11-12

**Question**: Should KV cache be kept on a local NVMe drive in the server or on an external NVMe-oF all-flash array?

**Short answer**: Both avoid full recompute, but under heavy concurrent read-back the external array is faster: measured on a 480B model at TP=8, FX100 delivers 29–40% more throughput and 26–32% lower TTFT at p50 than a single local NVMe drive [R2](https://vuke.top/en/evidence/R2). An external array also lets several inference instances share one cache [R3](https://vuke.top/en/evidence/R3).

## How the comparison was set up

The local setup is one NVMe drive (PCIe Gen4, 2 TB). The external setup is the FX100 all-flash NVMe-oF array: a 4-drive RAID0 pool (14 TB, XFS) reached over RoCEv2 on one 100 GbE port [R2](https://vuke.top/en/evidence/R2).

Note that this is not a like-for-like drive count; the baseline is a single drive. If your servers already have a local multi-drive array, the gap will differ from the one here and has to be measured on your own configuration.

## Readings at each concurrency level

The model is Qwen3-Coder-480B-FP8 on 8 × AMD MI308X under a long-context cold-recovery load. The value left of each arrow is the single local NVMe drive; the value on the right is FX100.

- Concurrency 8: TTFT at p50 10.17 s → 7.53 s; throughput 43.9 → 56.6 tok/s (+29%) [R2](https://vuke.top/en/evidence/R2).
- Concurrency 16: TTFT at p50 17.31 s → 11.85 s; throughput 53.6 → 74.9 tok/s (+40%) [R2](https://vuke.top/en/evidence/R2).
- Concurrency 32: TTFT at p50 35.73 s → 26.35 s; throughput 53.9 → 71.6 tok/s (+33%) [R2](https://vuke.top/en/evidence/R2).

## Why going over the network is faster

Intuition says a local drive sits closer to the GPU. But time to first token is not decided by the small extra protocol latency; it is decided by whether data can be returned to GPU memory fast enough under concurrent read-back. A multi-drive all-flash pool on an RDMA network offers more concurrent read bandwidth than one local drive.

Software matters too: the LMCache parallel-read patch raises single-GPU cold-read bandwidth from 0.98 GB/s to 5.23 GB/s [R1](https://vuke.top/en/evidence/R1). Without it, a serial read path holds back even a fast drive.

## Two things an external array adds

One is sharing. Cache on a local drive is usable only by that server; cache in an external pool can be read by a session scheduled onto any instance. Cross-instance hot sharing through the fs:// shared pool has been verified [R3](https://vuke.top/en/evidence/R3).

The other is sizing. The production basis is one FX100 per 8 inference nodes, grounded in the measured anchor that 100 GbE sustains KV read-back for 8 nodes [R1](https://vuke.top/en/evidence/R1) [R2](https://vuke.top/en/evidence/R2) [R3](https://vuke.top/en/evidence/R3).

## When a local drive is enough

For a single server with modest concurrency and sessions that never migrate between instances, a local NVMe drive already captures the largest gain, which is not recomputing. The advantage of an external array appears when concurrent read-back pressure is high or several instances need to share a cache.

## Related test reports

- [R1](https://vuke.top/en/evidence/R1) · FX100 Comprehensive LLM Inference and Training Benchmark (8 × AMD MI308X) (8 × AMD Instinct MI308X · ROCm 7.2)
- [R2](https://vuke.top/en/evidence/R2) · FX100 KV Cache Benchmark (480B, TP8, Long Context) (8 × AMD Instinct MI308X · ROCm 7.2)
- [R3](https://vuke.top/en/evidence/R3) · FX100 KV Cache Benchmark Summary (480B, TP4×2, All Metrics) (8 × AMD Instinct MI308X · ROCm 7.2)

## External references

- [NVM Express Specifications](https://nvmexpress.org/specifications/) · NVM Express, Inc.. The official specifications of NVMe and NVMe over Fabrics
- [RFC 5040: A Remote Direct Memory Access Protocol Specification](https://datatracker.ietf.org/doc/html/rfc5040) · IETF. The normative definition of the RDMA protocol
- [LMCache (GitHub repository)](https://github.com/LMCache/LMCache) · LMCache Project. LMCache source code, architecture and storage backends