Question
An inference cluster has to switch models often and effective GPU utilization is low. How can it be raised?
Short answer
Shorten model loading first. On every switch the GPU waits for weights to be read from storage into GPU memory and produces no tokens meanwhile. Modeled from measured load times at 20 model switches per hour, effective compute utilization rises from 46.7% to 62.8% with FX100R1.
Where the utilization is lost
On a platform where many models share a pool of accelerators, models are swapped in and out on demand. Swapping one in means reading tens to hundreds of gigabytes of weights into GPU memory, and during that time the accelerator is powered but serving nothing.
The more often you switch, the larger the share of each hour spent waiting. The GPU usage shown in monitoring may look healthy while the time actually spent on inference is eaten by loading.
Measured data
The following comes from a combined test on 8 × AMD MI308X, using Qwen2.5-32B and a 7B model.
- Model switching: at 20 switches per hour, effective compute utilization goes from 46.7% to 62.8%R1.
- Concurrent loading: with eight GPUs cold-reading models at once, wall-clock time is 16% shorter overall and each model loads 30% fasterR1.
- Training checkpoints: for 65.6 GB full-model snapshots, save time goes from 178 s to 94 s and sustained write bandwidth from 3.26 to 6.40 GB/sR1.
Where the utilization figure comes from
62.8% is not a utilization measured under production traffic. It is modeled from measured load times: given the number of switches per hour and the time each load takes, it computes how much of the hour is actually available for inferenceR1.
That means you can redo the calculation with your own inputs. Fewer than 20 switches per hour gives a smaller gain; larger models with longer loads give a bigger one.
How to tell whether it is worth doing
Start with two counts: how many times per hour each accelerator switches models, and how long a switch takes from request to ready. Their product is the time lost every hour.
If your models are resident and almost never switch, this optimization is irrelevant to you, and attention is better spent on KV cache and time to first token.
Related test reports
External references
- 01vLLM Documentation · vLLM Project
Official documentation of vLLM deployment, prefix caching and tensor-parallel settings
- 02
Further reading
Slow Model Loading From NFS: Ascend 910B Results
On Huawei Ascend 910B × 8, moving model storage from NFS to an FX100 all-flash array: measured serving load times for DeepSeek-32B and 70B, and where they apply.
SourceR1R9
How to Cut LLM Time to First Token Without More GPUs
With long contexts, high time to first token usually means evicted sessions are being recomputed. Three fixes that need no new GPUs, with results from a 480B model.
SourceR1R2R3
How to Accept an Inference Storage Project: 4 Gates
Split acceptance of an inference storage project into four gates on a weekly schedule: delivery, single-node baseline, main joint-test gate and stability sign-off.
SourceR1R2R3R8