4.1×
TTFT gain from the LMCache parallel-read patch
vs. before the patch
One GPU, concurrency 16, cold read from disk (Qwen2.5-32B): TTFT 37.97 s → 9.30 s; read bandwidth 0.98 → 5.23 GB/s, about 5.3×.
MeasuredSourceR1
Report R1 · Benchmark report
FX100 Comprehensive LLM Inference and Training Benchmark (8 × AMD MI308X)
One report, five questions: how much the LMCache parallel-read patch helps, whether KV tiering scales across GPUs, how much faster 8 GPUs load models concurrently, how effective utilization changes under frequent model switching, and how much faster concurrent training checkpoint writes become. Test models are Qwen2.5-32B and 7B.
- Date
- Type
- Benchmark report
- Test platform
- 8 × AMD Instinct MI308X · ROCm 7.2
Key findings
- LMCache parallel-read patch: one GPU, concurrency 16, cold read from disk; TTFT 37.97 s → 9.30 s (4.1×), read bandwidth 0.98 → 5.23 GB/s
- Training checkpoints: 65.6 GB full-model snapshots save in 94 s instead of 178 s (1.9×); sustained write bandwidth 3.26 → 6.40 GB/s
- Concurrent model loading: 8 GPUs cold-reading at once, 16% faster wall-clock and 30% faster per model
- Model switching: at 20 switches per hour, effective compute utilization rises from 46.7% to 62.8%
Figures this report supports
1.9×
Faster training checkpoint saves
vs. the baseline in report R1
8-GPU 32B LoRA with 65.6 GB full-model snapshots: 178 s → 94 s; sustained write bandwidth 3.26 → 6.40 GB/s (+96%).
MeasuredSourceR1
46.7% → 62.8%
Effective compute utilization under model switching
at 20 model switches per hour
Modeled from measured load times: at 20 model switches per hour, effective compute utilization rises from 46.7% to 62.8%.
MeasuredSourceR1
16% / 30%
Concurrent model loading on 8 GPUs
vs. the baseline in report R1
Eight GPUs cold-reading models at once: 16% faster wall-clock overall and 30% faster per model.
MeasuredSourceR1
1 : 8
Arrays to inference nodes
production sizing basis
One FX100 serves 8 inference nodes; 100 GbE sustaining KV read-back for 8 nodes is a measured anchor. On the joint-test basis, storage spend stays within roughly 10% of the GPU budget.
Test platform
| Accelerators | 8 × AMD Instinct MI308X (192 GB HBM each, gfx942) |
|---|---|
| Host | 2 × AMD EPYC 9654 (384 threads), about 1.5 TB RAM |
| Software stack | ROCm 7.2 · vLLM 0.20.1+rocm721 · LMCache (built from upstream mainline, 2026-06-29) |
| Device under test | FX100 all-flash NVMe-oF array: 4-drive RAID0 (14 TB, XFS), RoCEv2, one 100 GbE port |
| Baselines | A single local NVMe drive (PCIe Gen4, 2 TB); no external storage, full recompute |
| 480B model | Qwen3-Coder-480B-FP8 (MoE, about 450 GB of weights) |
Pages that cite this report
- Product: FX100
The workhorse with the most complete benchmark record
- Solution: Storage Acceleration: KV Cache Tiering
Storage acceleration
- Solution: Enablement for Domestic and Non-NVIDIA Accelerators
Accelerator enablement
- Solution: AI Datacenter Efficiency Optimization
Efficiency optimization
- Solution: AI Datacenter Construction
Datacenter construction
- Solution: Inference-Stack Software Development
Software development
Request the original reports
Tell us which report IDs you need and what for, and we will reply with how to get them. The R8 code export and raw data are provided on request in the same way.
Request the original reportQuote this ID when you askR1
Test jointly first, decide afterwards
Test jointly first, decide afterwards
Every engagement starts with a gated joint test of about 10 weeks. Four gates are passed in sequence; if one is missed, the project stops there, so you never pay for a benefit that was not demonstrated.
- Contact
- Wang Limin
- Phone
- +86 131 6181 8898
- mingxin@agentmail.to
The inbox is monitored and triaged by AI around the clock