Low GPU Utilization From Frequent Model Switching
Each model switch leaves GPUs waiting for weights to load. At 20 switches per hour, shorter load times raise effective compute utilization from 46.7% to 62.8%.
SourceR1
Insights
The questions buyers actually ask, one per article. Each gives the answer first, then the data and the sources.
Updated
How to Verify a Vendor's LLM Inference Benchmarks
Five checks for vendor inference benchmarks: baseline, platform and topology, run-to-run deviation, third-party report, reproduction material. Plus common traps.
SourceR2R3R4R5R8R9
How to Accept an Inference Storage Project: 4 Gates
Split acceptance of an inference storage project into four gates on a weekly schedule: delivery, single-node baseline, main joint-test gate and stability sign-off.
SourceR1R2R3R8
Slow Model Loading From NFS: Ascend 910B Results
On Huawei Ascend 910B × 8, moving model storage from NFS to an FX100 all-flash array: measured serving load times for DeepSeek-32B and 70B, and where they apply.
SourceR1R9
Local NVMe or an NVMe-oF Array for KV Cache?
Both avoid full recompute. Across three concurrency levels on a 480B model, an external FX100 array gives more throughput and lower TTFT than one local NVMe drive.
SourceR1R2R3
Recompute or Read Back? KV Cache on a 480B Model
Full recompute versus reading KV cache back from FX100 on a 480B model at concurrency 16: time to first token, throughput, test conditions and how to reproduce.
SourceR2R3R4R8
How to Cut LLM Time to First Token Without More GPUs
With long contexts, high time to first token usually means evicted sessions are being recomputed. Three fixes that need no new GPUs, with results from a 480B model.
SourceR1R2R3
What Is KV Cache Tiering and When Does It Help?
KV cache tiering keeps the KV cache that overflows GPU memory in host RAM and external flash, then reads it back instead of recomputing. Structure, gains and limits.
SourceR2R3R8
When Does KV Cache Tiering Not Help?
The gain from KV cache tiering comes from cache hits. Short chats, low concurrency, never-repeated contexts and output-bound workloads benefit little. Here is why.
SourceR2
Test jointly first, decide afterwards
Test jointly first, decide afterwards
Every engagement starts with a gated joint test of about 10 weeks. Four gates are passed in sequence; if one is missed, the project stops there, so you never pay for a benefit that was not demonstrated.
- Contact
- Wang Limin
- Phone
- +86 131 6181 8898
- mingxin@agentmail.to
The inbox is monitored and triaged by AI around the clock