Glossary
The dozen or so terms you need to read about inference storage. Each entry gives a definition first, then what it corresponds to in the measurements.
Updated
- KV cache
- The KV cache is the intermediate state an LLM keeps during inference: the key and value tensors computed at each attention layer for every token already processed. With it, the model does not have to recompute the preceding context when generating the next token. The longer the context, the more GPU memory the KV cache occupies.
- KV cache tiering
- KV cache tiering places KV cache on different media according to how hot it is: GPU memory holds active sessions, host RAM holds recent ones, and external all-flash storage holds all historical sessions and shares them across instances. The first two tiers are managed by the inference framework; the third needs a dedicated storage device.
- TTFT (time to first token)
- TTFT (time to first token) is the time from sending a request to receiving the model's first token, and it decides how responsive a service feels. It is dominated by the prefill phase: the model must process the entire input before it can start producing output.
- TPOT (time per output token)
- TPOT (time per output token) is the average time taken to generate each token once output has started. TTFT measures how long before the model starts speaking, TPOT how fast it speaks; together they describe the full latency of a request.
- Prefill
- Prefill is the first phase of inference: the model processes all input tokens at once and builds their KV cache. The longer the input, the longer prefill takes, which is the direct reason long-context requests have a high time to first token. The following phase, which generates output tokens one at a time, is called decode.
- Cold recovery
- Cold recovery is the case where a session's KV cache is no longer in GPU memory and its context state must be restored when the next request arrives. There are two ways to restore it: read the KV cache back from external storage, or have the GPU recompute the whole context. Many sessions recovering at once is often called a session-recovery storm.
- Prefix caching
- Prefix caching is the mechanism by which an inference engine reuses the KV cache of an identical input prefix: when two requests start the same way (the same system prompt or the same document), the second can skip computing that part. The prefix caching built into vLLM works only within GPU memory.
- LMCache
- LMCache is an open-source KV cache library that plugs into vLLM. It moves KV cache that does not fit in GPU memory down to host RAM and external storage and reads it back on a hit. It is a mainstream open-source implementation of KV cache tiering.
- vLLM
- vLLM is an open-source LLM inference and serving engine known for its PagedAttention memory management and high throughput. It is a common choice for serving open-source models in production.
- NVMe-oF
- NVMe-oF (NVMe over Fabrics) extends the NVMe protocol from the PCIe bus inside a server onto the network, so that many servers can share remote NVMe storage over Ethernet, InfiniBand and other fabrics while keeping performance close to a local drive.
- RoCEv2
- RoCEv2 (RDMA over Converged Ethernet v2) is a protocol that implements RDMA on Ethernet. Its packets are encapsulated in UDP/IP, so they can be routed across layer-3 networks. RDMA moves data directly between the memory of two machines, bypassing the operating-system kernel, which keeps latency and CPU usage low.
- HBM (high-bandwidth memory)
- HBM (high-bandwidth memory) is the stacked, high-bandwidth memory used on AI accelerators, in other words GPU memory. It is extremely fast but limited in capacity and expensive, and model weights and KV cache compete for it.
- TP (tensor parallelism)
- Tensor parallelism (TP) splits every layer of a model across several GPUs that compute it together. TP=8 means one model instance spans 8 GPUs, a common way to deploy models too large for a single card; TP4×2 divides 8 GPUs into two instances of 4 GPUs each.
- MoE (mixture of experts)
- MoE (mixture of experts) is a model architecture in which each layer contains several expert sub-networks and each token activates only a few of them. It lets a model grow very large without a proportional increase in compute per token, at the cost of a very large set of weights.
- ROCm
- ROCm is AMD's open-source software stack for GPU computing, covering drivers, compilers, math libraries and deep-learning framework support. On AMD Instinct accelerators it plays the role CUDA plays on NVIDIA platforms.
- Checkpoint
- A checkpoint is a snapshot of model state saved to storage at intervals during training, used for failure recovery and for keeping results. Training usually pauses while it is written, so the slower the write, the longer the GPUs sit idle.
Test jointly first, decide afterwards
Test jointly first, decide afterwards
Every engagement starts with a gated joint test of about 10 weeks. Four gates are passed in sequence; if one is missed, the project stops there, so you never pay for a benefit that was not demonstrated.
- Contact
- Wang Limin
- Phone
- +86 131 6181 8898
- mingxin@agentmail.to
The inbox is monitored and triaged by AI around the clock