What Is TTFT (time to first token)?

• R2FX100 KV Cache Benchmark (480B, TP8, Long Context)Qwen3-Coder-480B-FP8 deployed as one 8-GPU instance (TP=8), the standard production topology, under a long-context cold-recovery load. A three-way comparison of FX100, a single local NVMe drive and full recompute without external storage, stepped across concurrency 8 / 16 / 32. Most throughput and TTFT figures quoted on this site come from this report.2026-07-05
  8 × AMD Instinct MI308X · ROCm 7.2

• Local NVMe or an NVMe-oF Array for KV Cache?→

  Both avoid full recompute. Across three concurrency levels on a 480B model, an external FX100 array gives more throughput and lower TTFT than one local NVMe drive.

• Recompute or Read Back? KV Cache on a 480B Model→

  Full recompute versus reading KV cache back from FX100 on a 480B model at concurrency 16: time to first token, throughput, test conditions and how to reproduce.

• How to Cut LLM Time to First Token Without More GPUs→

  With long contexts, high time to first token usually means evicted sessions are being recomputed. Three fixes that need no new GPUs, with results from a 480B model.

• What Is KV Cache Tiering and When Does It Help?→

  KV cache tiering keeps the KV cache that overflows GPU memory in host RAM and external flash, then reads it back instead of recomputing. Structure, gains and limits.

• KV cache tiering→

  KV cache tiering places KV cache on different media according to how hot it is: GPU memory holds active sessions, host RAM holds recent ones, and external all-flash storage holds all historical sessions and shares them across instances. The first two tiers are managed by the inference framework; the third needs a dedicated storage device.

• TPOT (time per output token)→

  TPOT (time per output token) is the average time taken to generate each token once output has started. TTFT measures how long before the model starts speaking, TPOT how fast it speaks; together they describe the full latency of a request.

All terms→