Storage acceleration for AI inference
FX all-flash NVMe-oF arrays catch the KV cache that overflows GPU memory. On a 480B model in production topology that means +29–40% inference throughput and −26–32% time to first tokenR2R3. Every figure maps to a test report you can request and verify.
Fig. 1 · Time to first token · p50
R2§ 01
Each item states its baseline and its report ID. Results from different platforms are listed separately and never merged.
+29–40%
vs. a single local NVMe drive
Qwen3-Coder-480B-FP8 in TP=8 production topology under a long-context cold-recovery load: +29% at concurrency 8 and +40% at concurrency 16, the operating point with the largest gain; +35–36% for the whole machine in the dual-instance TP4×2 layout.
−26–32%
vs. a single local NVMe drive
Across three concurrency levels of the same 480B TP8 load, TTFT p50 falls from 10.17–35.73 s to 7.53–26.35 s.
MeasuredSourceR2
8.6–20×
vs. full recompute with no external storage
At concurrency 16, TTFT p50 is 149.5 s when recomputing and 11.85 s with FX100; throughput is 4.1 tok/s versus 74.9 tok/s.
MeasuredSourceR2
6.2–9.3×
vs. NFS, on Huawei Ascend 910B
On Huawei Atlas 910B: DeepSeek-32B serving load 691 s → 112 s (6.2×); DeepSeek-70B 1399 s → 150 s (9.3×).
MeasuredSourceR9
| Metric | Value | Baseline | Source |
|---|---|---|---|
| TTFT gain from the LMCache parallel-read patch | 4.1× | vs. before the patch | R1 |
| Faster training checkpoint saves | 1.9× | vs. the baseline in report R1 | R1 |
| Effective compute utilization under model switching | 46.7% → 62.8% | at 20 model switches per hour | R1 |
| Concurrent model loading on 8 GPUs | 16% / 30% | vs. the baseline in report R1 | R1 |
| Deviation between two independent runs | ≤5% | same load level, measured twice | R3 |
| HBM residency an external KV tier can stand in for | 128 GB | MetaX N260 single GPU, 14B model, seven-way comparison | R5 |
| ComfyUI + LTX-Video 2.3 models running end to end | 7 / 7 | AMD MI308X, ROCm 7.2 | R6R7 |
| Arrays to inference nodes | 1 : 8 | production sizing basis | R1R2R3 |
§ 02
Long-context sessions keep pushing KV cache out of GPU memory. Without an external tier, evicted sessions must be recomputed in full; with FX they are simply read back.
GPU memory (HBM)
Sessions that are active right now
Managed by vLLM
Host RAM
Sessions used recently
Managed by LMCache
FX all-flash array
All historical sessions, shared across instances
NVMe-oF · RoCEv2
§ 03
Raw readings from report R2: FX100 against a single local NVMe drive at concurrency 8, 16 and 32.
↓ lower is better
Concurrency8
−26%
Concurrency16
−32%
Concurrency32
−26%
↑ higher is better
Concurrency8
+29%
Concurrency16
+40%
Concurrency32
+33%
| Concurrency | FX100 TTFT | Local NVMe drive TTFT | Full recompute TTFT | FX100 tok/s | Local NVMe drive tok/s | Full recompute tok/s |
|---|---|---|---|---|---|---|
| 8 | 7.53 s | 10.17 s | — | 56.6 | 43.9 | — |
| 16 | 11.85 s | 17.31 s | 149.48 s | 74.9 | 53.6 | 4.1 |
| 32 | 26.35 s | 35.73 s | — | 71.6 | 53.9 | — |
§ 04
Four models with per-port rates from 100 Gb to 400 Gb. Specifications are vendor figures; FX100 has a full set of measured reports.
| Model | Interface generation | Rate per port | IOPS | Flash form factor | Status | Measured reports |
|---|---|---|---|---|---|---|
| FX100The workhorse with the most complete benchmark record | PCIe 3.0 | 100 Gb | 16M | U.2 | In production | R1R2R3R4R5R9 |
| FX200The PCIe 4.0 model with twice the port speed and IOPS | PCIe 4.0 | 200 Gb | 32M | U.2 | In production | — |
| FX300The PCIe 5.0 performance tier with 6 DPUs | PCIe 5.0 | 400 Gb | 60M | U.2 | In production | — |
| FX400The next flagship: PCIe 6.0 with E1.S flash | PCIe 6.0 | 400 Gb | 140M | E1.S | Test units; volume production planned for late 2026 | — |
§ 05
Five lines of work share the same measured reports and the same reproducible costing models.
§ 06
Every engagement starts with a gated joint test of about 10 weeks. Four gates are passed in sequence; if one is missed, the project stops there, so you never pay for a benefit that was not demonstrated.
G1Week 2
All materials delivered and inspected; every array passes its power-on self-test with all 24 drives recognized.
G2Week 4
The single-node 8-GPU inference baseline is reproduced within ±10% of the R1 figures; a local stress test of one array reaches 83% line efficiency against the measured anchor.
G3Week 7
KV tiering must deliver both a TTFT reduction of ≥25% and a throughput gain inside the measured +29–40% band; with 8 nodes sharing one array, read-back bandwidth approaches the NIC ceiling.
G4Week 10
72 hours of continuous load without interruption; single-drive and single-link fault injection stays invisible to the workload; a signed joint-test report is issued.
The full costing model is provided as Python source after an NDA, so you can rerun it with your own parameters.
§ 07
Each article gives the answer first, then the data and the sources.
Each model switch leaves GPUs waiting for weights to load. At 20 switches per hour, shorter load times raise effective compute utilization from 46.7% to 62.8%.
SourceR1
Five checks for vendor inference benchmarks: baseline, platform and topology, run-to-run deviation, third-party report, reproduction material. Plus common traps.
SourceR2R3R4R5R8R9
Split acceptance of an inference storage project into four gates on a weekly schedule: delivery, single-node baseline, main joint-test gate and stability sign-off.
SourceR1R2R3R8
§ 08
With Qwen3-Coder-480B-FP8 on 8 × AMD MI308X in TP=8 production topology, the measured results under a long-context cold-recovery load are: 29–40% more inference throughput and 26–32% lower TTFT p50 than a single local NVMe driveR2R3, and 8.6–20× faster than full recompute without external storageR2. These are application-level throughput and time to first token, not microsecond latencies of the storage device.
The gain comes from cache hits. Long contexts, multi-turn conversations and high-concurrency session recovery (agents, coding assistants, customer service) hit the cache often and benefit the most. Short, low-concurrency conversations hit it rarely and benefit little. That is why we recommend testing before buying: measure with your own session-length distribution and concurrency, then decide.
Every key figure carries a report ID (R1–R9), and the reports are available on request. R4 is an official report issued by a third-party testing organization. R8 is the test code export and raw data: the LMCache parallel-read patch, the load client, orchestration scripts and forensic probe scripts, so a third party can rerun the tests on its own cluster. Two independent runs at the same load level deviate by no more than 5%R3.
No rewrite of the inference engine is needed. The integration point is the open-source vLLM + LMCache combination: LMCache moves KV cache that does not fit in GPU memory down to host RAM and external storage, and the FX array is that external tier. We adapted the stack at source level on ROCm, and the parallel-read patch is delivered as part of R8R1R8.
The reference price of a fully populated FX100 is about ¥371,200: ¥160,000 for the platform plus 24 enterprise NVMe SSDs of 7.68 TB each (reference price ¥8,800 per drive), 184.3 TB raw, or about ¥2,014 per TB. SSD prices follow the market, and the purchase contract sets the final price. FX200 and FX300 are quoted by configuration; production pricing for FX400 has not been announced.
Test jointly first, decide afterwards
Our numbers were measured on our platform. Your session lengths and concurrency are different, and so your results will be. A joint test of about 10 weeks with four gates gives you an answer of your own.
The inbox is monitored and triaged by AI around the clock