# How to Verify a Vendor's LLM Inference Benchmarks

> Five checks for vendor inference benchmarks: baseline, platform and topology, run-to-run deviation, third-party report, reproduction material. Plus common traps.

URL: https://vuke.top/en/insights/how-to-verify-vendor-benchmarks · Updated November 12, 2026

Published: 2026-11-12

**Question**: How should a buyer verify the LLM inference performance figures a vendor provides?

**Short answer**: Check five things: whether the baseline is stated, whether the test platform and deployment topology are stated, whether run-to-run deviation is given, whether a third-party body has issued a report, and whether you can get the reproduction material and rerun it yourself. If any one is missing, treat the figure as a reference, not as a basis for purchase.

## Five checks

Each item below uses our own reports as the example of what verifiable data should look like.

- Baseline: for the same equipment, the result is 8.6–20× against full recompute without external storage, and +29–40% throughput against a single local NVMe drive [R2](https://vuke.top/en/evidence/R2) [R3](https://vuke.top/en/evidence/R3). A gain quoted without its baseline means nothing.
- Platform and topology: results on AMD MI308X, Huawei Ascend 910B and MetaX N260 are documented separately and never merged [R2](https://vuke.top/en/evidence/R2) [R9](https://vuke.top/en/evidence/R9) [R5](https://vuke.top/en/evidence/R5). A different accelerator or a different parallelism layout changes the numbers.
- Run-to-run deviation: two independent runs at the same load level deviate by no more than 5% [R3](https://vuke.top/en/evidence/R3). With a single run there is no way to tell whether the reading was a fluke.
- Third-party report: the conclusions for both deployment topologies were compiled into an official report by a third-party testing body [R4](https://vuke.top/en/evidence/R4).
- Reproduction material: the patch, load client, orchestration scripts and raw data are handed over so the buyer can rerun the tests [R8](https://vuke.top/en/evidence/R8).

## Three common traps

Even when the figures are genuine, some ways of presenting them mislead.

- Mixed denominators: a large multiple measured against doing nothing is presented as the gain over the existing setup.
- Reporting only the most favorable level: the complete practice is to report the gradient. On the 480B model, the throughput gain is +29% at concurrency 8, +40% at concurrency 16 and +33% at concurrency 32 [R2](https://vuke.top/en/evidence/R2).
- Modeled results presented as measured: figures derived from measurements should be labeled as estimates, and the model used to derive them should be handed over.

## Questions to send the vendor

Verification does not require deep technical background. Send the questions below and see whether each gets a written answer.

- What was this gain measured against? What is the hardware configuration of the baseline?
- Which model, how many accelerators, which deployment topology and what load were used?
- How many times was the same test repeated, and how far apart were the readings?
- Is there a report from a third-party body, and what is its report number?
- Can we have the test scripts and raw data to rerun on our own cluster?

## What to do after verifying

Verification only shows that the vendor's figures hold under the vendor's conditions. Your model, context length and concurrency differ, and so will your results. The last step is therefore always to measure on your own workload, with the pass conditions written down in advance.

## Related test reports

- [R2](https://vuke.top/en/evidence/R2) · FX100 KV Cache Benchmark (480B, TP8, Long Context) (8 × AMD Instinct MI308X · ROCm 7.2)
- [R3](https://vuke.top/en/evidence/R3) · FX100 KV Cache Benchmark Summary (480B, TP4×2, All Metrics) (8 × AMD Instinct MI308X · ROCm 7.2)
- [R4](https://vuke.top/en/evidence/R4) · FX100 KV Cache Benchmark (480B, Multi-Instance, No. -006) (8 × AMD Instinct MI308X · ROCm 7.2)
- [R5](https://vuke.top/en/evidence/R5) · FX100 KV Cache Benchmark (14B, HBM Efficiency) (Single MetaX N260 GPU)
- [R8](https://vuke.top/en/evidence/R8) · FX100 KV Cache Test Code Export and Raw Benchmark Data (AMD Instinct MI308X · ROCm 7.2)
- [R9](https://vuke.top/en/evidence/R9) · FX100 on Huawei Ascend 910B: Inference and Training Benchmark (vs. NFS Baseline) (8 × Huawei Atlas 910B (Kunpeng-920))

## External references

- [MLPerf Inference: Datacenter](https://mlcommons.org/benchmarks/inference-datacenter/) · MLCommons. The methodology of a public inference benchmark, a neutral reference when discussing comparable and reproducible results