Solutions
AI Datacenter Efficiency Optimization
Efficiency optimization
Updated
The problem
Many clusters are not short of GPUs; their GPUs are waiting: for models to load, for models to switch, for checkpoints to finish writing. During those waits the GPUs draw power and produce nothing.
Our approach
Cut the waiting first, then talk about adding GPUs. At 20 model switches per hour, effective compute utilization rises from 46.7% to 62.8%R1.
Key points and sources
- Model switching: modeled from measured load times, effective utilization goes from 46.7% to 62.8% at 20 switches per hourR1
- Concurrent loading: 8 GPUs cold-reading at once, 16% faster wall-clock and 30% faster per modelR1
- Training checkpoints: concurrent writes are 1.9× faster, shrinking idle bubbles during trainingR1
- HBM equivalence: an external KV tier can stand in for 128 GB-class HBM residency (seven-way comparison on a 14B model, MetaX N260)R5
Related test reports
- R1FX100 Comprehensive LLM Inference and Training Benchmark (8 × AMD MI308X)One report, five questions: how much the LMCache parallel-read patch helps, whether KV tiering scales across GPUs, how much faster 8 GPUs load models concurrently, how effective utilization changes under frequent model switching, and how much faster concurrent training checkpoint writes become. Test models are Qwen2.5-32B and 7B.
8 × AMD Instinct MI308X · ROCm 7.2 - R5FX100 KV Cache Benchmark (14B, HBM Efficiency)This report answers whether an external KV tier can stand in for GPU memory. A seven-way comparison on a 14B model (GDS direct read, recompute at two HBM sizes, direct read with low HBM) builds a three-step argument for HBM equivalence. The platform is a single MetaX N260 GPU; the method carries across platforms, but the numbers are never mixed with MI308X results.
Single MetaX N260 GPU
Related products
Test jointly first, decide afterwards
Every engagement starts with a gated joint test of about 10 weeks. Four gates are passed in sequence; if one is missed, the project stops there, so you never pay for a benefit that was not demonstrated.
G1Week 2
Delivery acceptance
All materials delivered and inspected; every array passes its power-on self-test with all 24 drives recognized.
G2Week 4
Single-node baseline
The single-node 8-GPU inference baseline is reproduced within ±10% of the R1 figures; a local stress test of one array reaches 83% line efficiency against the measured anchor.
G3Week 7
Main joint-test gate
KV tiering must deliver both a TTFT reduction of ≥25% and a throughput gain inside the measured +29–40% band; with 8 nodes sharing one array, read-back bandwidth approaches the NIC ceiling.
G4Week 10
Stability and sign-off
72 hours of continuous load without interruption; single-drive and single-link fault injection stays invisible to the workload; a signed joint-test report is issued.
Other solutions
- Storage Acceleration: KV Cache Tiering
Storage acceleration
- Enablement for Domestic and Non-NVIDIA Accelerators
Accelerator enablement
- AI Datacenter Construction
Datacenter construction
- Inference-Stack Software Development
Software development
Test jointly first, decide afterwards
Test jointly first, decide afterwards
Every engagement starts with a gated joint test of about 10 weeks. Four gates are passed in sequence; if one is missed, the project stops there, so you never pay for a benefit that was not demonstrated.
- Contact
- Wang Limin
- Phone
- +86 131 6181 8898
- mingxin@agentmail.to
The inbox is monitored and triaged by AI around the clock