# How to Cut LLM Time to First Token Without More GPUs

> With long contexts, high time to first token usually means evicted sessions are being recomputed. Three fixes that need no new GPUs, with results from a 480B model.

URL: https://vuke.top/en/insights/reduce-ttft-without-adding-gpus · Updated November 12, 2026

Published: 2026-11-12

**Question**: How do you reduce LLM time to first token (TTFT) without adding GPUs?

**Short answer**: With long contexts, high time to first token is usually caused by sessions that were evicted from GPU memory and are being recomputed in full, not by a shortage of compute. Add an external all-flash tier that keeps the KV cache and read it back when the session returns: measured on a 480B model against a single local NVMe drive, TTFT at p50 falls by 26–32% [R2](https://vuke.top/en/evidence/R2).

## First find where the latency goes

Time to first token is dominated by prefill: the model must process the entire input before it can emit a token. The longer the input, the longer prefill takes.

If a session's KV cache is still in GPU memory, a new turn only has to process the newly added input. If it has been evicted, the whole history has to be computed again. In measurements on a 480B model at concurrency 16, TTFT at p50 for such a full recompute is 149.5 s [R2](https://vuke.top/en/evidence/R2).

So the first step is to look at monitoring: are the slow requests concentrated in long sessions that came back after a pause?

## Three fixes that need no new GPUs

All three work on storage and software and require no additional accelerators.

- Keep the evicted KV cache: use vLLM + LMCache to write it to external all-flash storage and read it back on a hit, skipping that part of prefill.
- Make read-back fast enough: the LMCache parallel-read patch cuts TTFT for a single GPU cold-reading from disk at concurrency 16 from 37.97 s to 9.30 s, and raises read bandwidth from 0.98 GB/s to 5.23 GB/s [R1](https://vuke.top/en/evidence/R1).
- Let several instances share one cache: cross-instance hot sharing through the fs:// shared pool has been verified, so a session scheduled onto another instance does not need a recompute [R3](https://vuke.top/en/evidence/R3).

## How much it drops in measurements

These are readings for Qwen3-Coder-480B-FP8 at TP=8 on 8 × AMD MI308X, with a single local NVMe drive as the baseline.

- Concurrency 8: TTFT at p50 falls from 10.17 s to 7.53 s [R2](https://vuke.top/en/evidence/R2).
- Concurrency 16: TTFT at p50 falls from 17.31 s to 11.85 s [R2](https://vuke.top/en/evidence/R2).
- Concurrency 32: TTFT at p50 falls from 35.73 s to 26.35 s [R2](https://vuke.top/en/evidence/R2).

## When you still need more GPUs

An external cache saves repeated prefill. If the context of a request has never been seen before, there is no cache to read and this tier cannot help.

If the bottleneck is in the output phase, meaning tokens are not generated fast enough, that is a compute problem and calls for more or faster accelerators. Work out which kind of latency you have before deciding where to spend.

## Related test reports

- [R1](https://vuke.top/en/evidence/R1) · FX100 Comprehensive LLM Inference and Training Benchmark (8 × AMD MI308X) (8 × AMD Instinct MI308X · ROCm 7.2)
- [R2](https://vuke.top/en/evidence/R2) · FX100 KV Cache Benchmark (480B, TP8, Long Context) (8 × AMD Instinct MI308X · ROCm 7.2)
- [R3](https://vuke.top/en/evidence/R3) · FX100 KV Cache Benchmark Summary (480B, TP4×2, All Metrics) (8 × AMD Instinct MI308X · ROCm 7.2)

## External references

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180) · arXiv (SOSP '23). vLLM's PagedAttention memory management and the motivation for paging the KV cache
- [LMCache (GitHub repository)](https://github.com/LMCache/LMCache) · LMCache Project. LMCache source code, architecture and storage backends