Glossary
What Is vLLM?
vLLM is an open-source LLM inference and serving engine known for its PagedAttention memory management and high throughput. It is a common choice for serving open-source models in production.
Updated
In the measurements
The MI308X benchmarks on this site use vLLM 0.20.1+rocm721, combined with LMCache to form the KV cache tiering stackR2.
Related test reports
Nearby terms
- LMCache
LMCache is an open-source KV cache library that plugs into vLLM. It moves KV cache that does not fit in GPU memory down to host RAM and external storage and reads it back on a hit. It is a mainstream open-source implementation of KV cache tiering.
- NVMe-oF
NVMe-oF (NVMe over Fabrics) extends the NVMe protocol from the PCIe bus inside a server onto the network, so that many servers can share remote NVMe storage over Ethernet, InfiniBand and other fabrics while keeping performance close to a local drive.
Test jointly first, decide afterwards
Test jointly first, decide afterwards
Every engagement starts with a gated joint test of about 10 weeks. Four gates are passed in sequence; if one is missed, the project stops there, so you never pay for a benefit that was not demonstrated.
- Contact
- Wang Limin
- Phone
- +86 131 6181 8898
- mingxin@agentmail.to
The inbox is monitored and triaged by AI around the clock