How to Verify a Vendor's LLM Inference Benchmarks

Each item below uses our own reports as the example of what verifiable data should look like.

• Baseline: for the same equipment, the result is 8.6–20× against full recompute without external storage, and +29–40% throughput against a single local NVMe driveR2R3. A gain quoted without its baseline means nothing.
• Platform and topology: results on AMD MI308X, Huawei Ascend 910B and MetaX N260 are documented separately and never mergedR2R9R5. A different accelerator or a different parallelism layout changes the numbers.
• Run-to-run deviation: two independent runs at the same load level deviate by no more than 5%R3. With a single run there is no way to tell whether the reading was a fluke.
• Third-party report: the conclusions for both deployment topologies were compiled into an official report by a third-party testing bodyR4.
• Reproduction material: the patch, load client, orchestration scripts and raw data are handed over so the buyer can rerun the testsR8.

Even when the figures are genuine, some ways of presenting them mislead.

• Mixed denominators: a large multiple measured against doing nothing is presented as the gain over the existing setup.
• Reporting only the most favorable level: the complete practice is to report the gradient. On the 480B model, the throughput gain is +29% at concurrency 8, +40% at concurrency 16 and +33% at concurrency 32R2.
• Modeled results presented as measured: figures derived from measurements should be labeled as estimates, and the model used to derive them should be handed over.

Verification does not require deep technical background. Send the questions below and see whether each gets a written answer.

• What was this gain measured against? What is the hardware configuration of the baseline?
• Which model, how many accelerators, which deployment topology and what load were used?
• How many times was the same test repeated, and how far apart were the readings?
• Is there a report from a third-party body, and what is its report number?
• Can we have the test scripts and raw data to rerun on our own cluster?

Verification only shows that the vendor's figures hold under the vendor's conditions. Your model, context length and concurrency differ, and so will your results. The last step is therefore always to measure on your own workload, with the pass conditions written down in advance.