Glossary
What Is LMCache?
LMCache is an open-source KV cache library that plugs into vLLM. It moves KV cache that does not fit in GPU memory down to host RAM and external storage and reads it back on a hit. It is a mainstream open-source implementation of KV cache tiering.
Updated
In the measurements
Related test reports
- R1FX100 Comprehensive LLM Inference and Training Benchmark (8 × AMD MI308X)One report, five questions: how much the LMCache parallel-read patch helps, whether KV tiering scales across GPUs, how much faster 8 GPUs load models concurrently, how effective utilization changes under frequent model switching, and how much faster concurrent training checkpoint writes become. Test models are Qwen2.5-32B and 7B.
8 × AMD Instinct MI308X · ROCm 7.2 - R8FX100 KV Cache Test Code Export and Raw Benchmark DataEverything a third party needs to rerun the tests: the LMCache parallel-read patch (git patch plus full before/after sources), the load client, orchestration scripts, forensic probe scripts and the raw benchmark data.
AMD Instinct MI308X · ROCm 7.2
Nearby terms
- Prefix caching
Prefix caching is the mechanism by which an inference engine reuses the KV cache of an identical input prefix: when two requests start the same way (the same system prompt or the same document), the second can skip computing that part. The prefix caching built into vLLM works only within GPU memory.
- vLLM
vLLM is an open-source LLM inference and serving engine known for its PagedAttention memory management and high throughput. It is a common choice for serving open-source models in production.
Test jointly first, decide afterwards
Test jointly first, decide afterwards
Every engagement starts with a gated joint test of about 10 weeks. Four gates are passed in sequence; if one is missed, the project stops there, so you never pay for a benefit that was not demonstrated.
- Contact
- Wang Limin
- Phone
- +86 131 6181 8898
- mingxin@agentmail.to
The inbox is monitored and triaged by AI around the clock