What Is TPOT (time per output token)?

• R3FX100 KV Cache Benchmark Summary (480B, TP4×2, All Metrics)The same 8-GPU server redeployed as two 4-GPU instances (TP4×2), with every metric recorded across nine load levels: TTFT p50 / p90 / p99, TPOT, throughput and disk bandwidth. The report also verifies two things: the fs:// shared pool lets both instances hot-share one cache, and two independent runs at the same level deviate by no more than 5%.2026-07-06
  8 × AMD Instinct MI308X · ROCm 7.2

• TTFT (time to first token)→

  TTFT (time to first token) is the time from sending a request to receiving the model's first token, and it decides how responsive a service feels. It is dominated by the prefill phase: the model must process the entire input before it can start producing output.

• Prefill→

  Prefill is the first phase of inference: the model processes all input tokens at once and builds their KV cache. The longer the input, the longer prefill takes, which is the direct reason long-context requests have a high time to first token. The following phase, which generates output tokens one at a time, is called decode.

All terms→