Question
How do you reduce LLM time to first token (TTFT) without adding GPUs?
Short answer
With long contexts, high time to first token is usually caused by sessions that were evicted from GPU memory and are being recomputed in full, not by a shortage of compute. Add an external all-flash tier that keeps the KV cache and read it back when the session returns: measured on a 480B model against a single local NVMe drive, TTFT at p50 falls by 26–32%R2.
First find where the latency goes
Time to first token is dominated by prefill: the model must process the entire input before it can emit a token. The longer the input, the longer prefill takes.
If a session's KV cache is still in GPU memory, a new turn only has to process the newly added input. If it has been evicted, the whole history has to be computed again. In measurements on a 480B model at concurrency 16, TTFT at p50 for such a full recompute is 149.5 sR2.
So the first step is to look at monitoring: are the slow requests concentrated in long sessions that came back after a pause?
Three fixes that need no new GPUs
All three work on storage and software and require no additional accelerators.
- Keep the evicted KV cache: use vLLM + LMCache to write it to external all-flash storage and read it back on a hit, skipping that part of prefill.
- Make read-back fast enough: the LMCache parallel-read patch cuts TTFT for a single GPU cold-reading from disk at concurrency 16 from 37.97 s to 9.30 s, and raises read bandwidth from 0.98 GB/s to 5.23 GB/sR1.
- Let several instances share one cache: cross-instance hot sharing through the fs:// shared pool has been verified, so a session scheduled onto another instance does not need a recomputeR3.
How much it drops in measurements
When you still need more GPUs
An external cache saves repeated prefill. If the context of a request has never been seen before, there is no cache to read and this tier cannot help.
If the bottleneck is in the output phase, meaning tokens are not generated fast enough, that is a compute problem and calls for more or faster accelerators. Work out which kind of latency you have before deciding where to spend.
Related test reports
- R1FX100 Comprehensive LLM Inference and Training Benchmark (8 × AMD MI308X)One report, five questions: how much the LMCache parallel-read patch helps, whether KV tiering scales across GPUs, how much faster 8 GPUs load models concurrently, how effective utilization changes under frequent model switching, and how much faster concurrent training checkpoint writes become. Test models are Qwen2.5-32B and 7B.
8 × AMD Instinct MI308X · ROCm 7.2 - R2FX100 KV Cache Benchmark (480B, TP8, Long Context)Qwen3-Coder-480B-FP8 deployed as one 8-GPU instance (TP=8), the standard production topology, under a long-context cold-recovery load. A three-way comparison of FX100, a single local NVMe drive and full recompute without external storage, stepped across concurrency 8 / 16 / 32. Most throughput and TTFT figures quoted on this site come from this report.
8 × AMD Instinct MI308X · ROCm 7.2 - R3FX100 KV Cache Benchmark Summary (480B, TP4×2, All Metrics)The same 8-GPU server redeployed as two 4-GPU instances (TP4×2), with every metric recorded across nine load levels: TTFT p50 / p90 / p99, TPOT, throughput and disk bandwidth. The report also verifies two things: the fs:// shared pool lets both instances hot-share one cache, and two independent runs at the same level deviate by no more than 5%.
8 × AMD Instinct MI308X · ROCm 7.2
External references
- 01Efficient Memory Management for Large Language Model Serving with PagedAttention · arXiv (SOSP '23)
vLLM's PagedAttention memory management and the motivation for paging the KV cache
- 02
Further reading
What Is KV Cache Tiering and When Does It Help?
KV cache tiering keeps the KV cache that overflows GPU memory in host RAM and external flash, then reads it back instead of recomputing. Structure, gains and limits.
SourceR2R3R8
Recompute or Read Back? KV Cache on a 480B Model
Full recompute versus reading KV cache back from FX100 on a 480B model at concurrency 16: time to first token, throughput, test conditions and how to reproduce.
SourceR2R3R4R8
Low GPU Utilization From Frequent Model Switching
Each model switch leaves GPUs waiting for weights to load. At 20 switches per hour, shorter load times raise effective compute utilization from 46.7% to 62.8%.
SourceR1