Question
How should an inference storage acceleration project be accepted so that you do not pay for performance that never materializes?
Short answer
Split acceptance into four gates on a weekly schedule, each with a measurable pass condition, and stop at any gate that is missed: G1 delivery acceptance (week 2), G2 single-node baseline (week 4), G3 main joint-test gate (week 7) and G4 stability and sign-off (week 10).
Why use gates
The most common dispute in performance projects goes like this: the equipment has arrived, the money has been paid, the numbers after go-live do not match the proposal, and each side has its own view of what counts as meeting the target.
Gating moves that argument to the front. Before the engagement starts, both sides write down what is tested at each stage and what reading counts as a pass; a pass opens the next stage and a miss stops the loss. The buyer's exposure is limited to the gate currently in progress.
Pass conditions of the four gates
These are the four gates we use in joint tests, about 10 weeks end to end.
- G1 delivery acceptance (week 2): all materials delivered and inspected; every array passes its power-on self-test with all 24 drives recognized.
- G2 single-node baseline (week 4): the single-node 8-GPU inference baseline is reproduced within ±10% of the R1 figuresR1; a local stress test of one array reaches 83% line efficiency against the measured anchor.
- G3 main joint-test gate (week 7): a TTFT reduction of ≥25% and a throughput gain inside the measured +29–40% bandR2R3; with 8 nodes sharing one array, read-back bandwidth approaches the NIC ceiling.
- G4 stability and sign-off (week 10): 72 hours of continuous load without interruption; single-drive and single-link fault injection stays invisible to the workload; a signed joint-test report is issued.
Where the gate thresholds come from
The thresholds are not negotiated; they are taken from existing measured reports. The throughput band required at G3 is the +29–40% measured on the 480B model in two deployment topologiesR2R3.
The benefit is that it binds both sides: the vendor cannot lower the bar afterwards, and the buyer is never asked to accept a target that has not been measured before.
What else a buyer should ask for
Beyond the gates themselves, three things are worth writing into the agreement.
- Signed reports: the result of every gate is on paper rather than left in a demo.
- Reproduction material: the patch, load client, orchestration scripts and raw data, so your own team can rerun the testsR8.
- A recomputable costing model: the return calculation is delivered as source code so that you can rerun it with your own parameters.
Related test reports
- R1FX100 Comprehensive LLM Inference and Training Benchmark (8 × AMD MI308X)One report, five questions: how much the LMCache parallel-read patch helps, whether KV tiering scales across GPUs, how much faster 8 GPUs load models concurrently, how effective utilization changes under frequent model switching, and how much faster concurrent training checkpoint writes become. Test models are Qwen2.5-32B and 7B.
8 × AMD Instinct MI308X · ROCm 7.2 - R2FX100 KV Cache Benchmark (480B, TP8, Long Context)Qwen3-Coder-480B-FP8 deployed as one 8-GPU instance (TP=8), the standard production topology, under a long-context cold-recovery load. A three-way comparison of FX100, a single local NVMe drive and full recompute without external storage, stepped across concurrency 8 / 16 / 32. Most throughput and TTFT figures quoted on this site come from this report.
8 × AMD Instinct MI308X · ROCm 7.2 - R3FX100 KV Cache Benchmark Summary (480B, TP4×2, All Metrics)The same 8-GPU server redeployed as two 4-GPU instances (TP4×2), with every metric recorded across nine load levels: TTFT p50 / p90 / p99, TPOT, throughput and disk bandwidth. The report also verifies two things: the fs:// shared pool lets both instances hot-share one cache, and two independent runs at the same level deviate by no more than 5%.
8 × AMD Instinct MI308X · ROCm 7.2 - R8FX100 KV Cache Test Code Export and Raw Benchmark DataEverything a third party needs to rerun the tests: the LMCache parallel-read patch (git patch plus full before/after sources), the load client, orchestration scripts, forensic probe scripts and the raw benchmark data.
AMD Instinct MI308X · ROCm 7.2
External references
- 01MLPerf Inference: Datacenter · MLCommons
The methodology of a public inference benchmark, a neutral reference when discussing comparable and reproducible results
Further reading
How to Verify a Vendor's LLM Inference Benchmarks
Five checks for vendor inference benchmarks: baseline, platform and topology, run-to-run deviation, third-party report, reproduction material. Plus common traps.
SourceR2R3R4R5R8R9
When Does KV Cache Tiering Not Help?
The gain from KV cache tiering comes from cache hits. Short chats, low concurrency, never-repeated contexts and output-bound workloads benefit little. Here is why.
SourceR2
Recompute or Read Back? KV Cache on a 480B Model
Full recompute versus reading KV cache back from FX100 on a 480B model at concurrency 16: time to first token, throughput, test conditions and how to reproduce.
SourceR2R3R4R8