Solutions
Storage Acceleration: KV Cache Tiering
Storage acceleration
Updated
The problem
Long contexts, multi-turn chat and agent workloads keep pushing KV cache out of GPU memory. When an evicted session returns, the GPU has to recompute the entire context, which can push time to first token past a hundred seconds and drag throughput down.
Our approach
Key points and sources
- More capacity before more GPUs: on the joint-test basis storage spend stays within roughly 10% of the GPU budget and buys a 29–40% throughput gainR2R3
- Session-recovery storms stop taking the service down: 8.6–20× faster than full recompute without external storageR2
- Several instances share one cache: cross-instance hot sharing through the fs:// shared pool is verified, with two independent runs within ≤5%R3
- Existing clusters benefit too: on an Ascend 910B platform bottlenecked by NFS, model loading is 6.2–9.3× fasterR9
Related test reports
- R1FX100 Comprehensive LLM Inference and Training Benchmark (8 × AMD MI308X)One report, five questions: how much the LMCache parallel-read patch helps, whether KV tiering scales across GPUs, how much faster 8 GPUs load models concurrently, how effective utilization changes under frequent model switching, and how much faster concurrent training checkpoint writes become. Test models are Qwen2.5-32B and 7B.
8 × AMD Instinct MI308X · ROCm 7.2 - R2FX100 KV Cache Benchmark (480B, TP8, Long Context)Qwen3-Coder-480B-FP8 deployed as one 8-GPU instance (TP=8), the standard production topology, under a long-context cold-recovery load. A three-way comparison of FX100, a single local NVMe drive and full recompute without external storage, stepped across concurrency 8 / 16 / 32. Most throughput and TTFT figures quoted on this site come from this report.
8 × AMD Instinct MI308X · ROCm 7.2 - R3FX100 KV Cache Benchmark Summary (480B, TP4×2, All Metrics)The same 8-GPU server redeployed as two 4-GPU instances (TP4×2), with every metric recorded across nine load levels: TTFT p50 / p90 / p99, TPOT, throughput and disk bandwidth. The report also verifies two things: the fs:// shared pool lets both instances hot-share one cache, and two independent runs at the same level deviate by no more than 5%.
8 × AMD Instinct MI308X · ROCm 7.2 - R4FX100 KV Cache Benchmark (480B, Multi-Instance, No. -006)The official compilation of R2 and R3, issued by a third-party testing organization as report No. -006. If your procurement or compliance process needs a document from an independent body, this is the one to cite.
8 × AMD Instinct MI308X · ROCm 7.2 - R9FX100 on Huawei Ascend 910B: Inference and Training Benchmark (vs. NFS Baseline)Three comparison groups against an NFS baseline on an Ascend platform with 8 × Huawei Atlas 910B: model loading for inference (DeepSeek-32B / 70B), training weight and checkpoint I/O (Qwen-7B), and training-data acceleration (YOLOv8 / COCO). The platform is Ascend 910B, not MI308X, and is labeled as such wherever this site quotes it.
8 × Huawei Atlas 910B (Kunpeng-920)
Related products
Test jointly first, decide afterwards
Every engagement starts with a gated joint test of about 10 weeks. Four gates are passed in sequence; if one is missed, the project stops there, so you never pay for a benefit that was not demonstrated.
G1Week 2
Delivery acceptance
All materials delivered and inspected; every array passes its power-on self-test with all 24 drives recognized.
G2Week 4
Single-node baseline
The single-node 8-GPU inference baseline is reproduced within ±10% of the R1 figures; a local stress test of one array reaches 83% line efficiency against the measured anchor.
G3Week 7
Main joint-test gate
KV tiering must deliver both a TTFT reduction of ≥25% and a throughput gain inside the measured +29–40% band; with 8 nodes sharing one array, read-back bandwidth approaches the NIC ceiling.
G4Week 10
Stability and sign-off
72 hours of continuous load without interruption; single-drive and single-link fault injection stays invisible to the workload; a signed joint-test report is issued.
Other solutions
- Enablement for Domestic and Non-NVIDIA Accelerators
Accelerator enablement
- AI Datacenter Efficiency Optimization
Efficiency optimization
- AI Datacenter Construction
Datacenter construction
- Inference-Stack Software Development
Software development
Test jointly first, decide afterwards
Test jointly first, decide afterwards
Every engagement starts with a gated joint test of about 10 weeks. Four gates are passed in sequence; if one is missed, the project stops there, so you never pay for a benefit that was not demonstrated.
- Contact
- Wang Limin
- Phone
- +86 131 6181 8898
- mingxin@agentmail.to
The inbox is monitored and triaged by AI around the clock