# Low GPU Utilization From Frequent Model Switching

> Each model switch leaves GPUs waiting for weights to load. At 20 switches per hour, shorter load times raise effective compute utilization from 46.7% to 62.8%.

URL: https://vuke.top/en/insights/gpu-utilization-model-switching · Updated November 12, 2026

Published: 2026-11-12

**Question**: An inference cluster has to switch models often and effective GPU utilization is low. How can it be raised?

**Short answer**: Shorten model loading first. On every switch the GPU waits for weights to be read from storage into GPU memory and produces no tokens meanwhile. Modeled from measured load times at 20 model switches per hour, effective compute utilization rises from 46.7% to 62.8% with FX100 [R1](https://vuke.top/en/evidence/R1).

## Where the utilization is lost

On a platform where many models share a pool of accelerators, models are swapped in and out on demand. Swapping one in means reading tens to hundreds of gigabytes of weights into GPU memory, and during that time the accelerator is powered but serving nothing.

The more often you switch, the larger the share of each hour spent waiting. The GPU usage shown in monitoring may look healthy while the time actually spent on inference is eaten by loading.

## Measured data

The following comes from a combined test on 8 × AMD MI308X, using Qwen2.5-32B and a 7B model.

- Model switching: at 20 switches per hour, effective compute utilization goes from 46.7% to 62.8% [R1](https://vuke.top/en/evidence/R1).
- Concurrent loading: with eight GPUs cold-reading models at once, wall-clock time is 16% shorter overall and each model loads 30% faster [R1](https://vuke.top/en/evidence/R1).
- Training checkpoints: for 65.6 GB full-model snapshots, save time goes from 178 s to 94 s and sustained write bandwidth from 3.26 to 6.40 GB/s [R1](https://vuke.top/en/evidence/R1).

## Where the utilization figure comes from

62.8% is not a utilization measured under production traffic. It is modeled from measured load times: given the number of switches per hour and the time each load takes, it computes how much of the hour is actually available for inference [R1](https://vuke.top/en/evidence/R1).

That means you can redo the calculation with your own inputs. Fewer than 20 switches per hour gives a smaller gain; larger models with longer loads give a bigger one.

## How to tell whether it is worth doing

Start with two counts: how many times per hour each accelerator switches models, and how long a switch takes from request to ready. Their product is the time lost every hour.

If your models are resident and almost never switch, this optimization is irrelevant to you, and attention is better spent on KV cache and time to first token.

## Related test reports

- [R1](https://vuke.top/en/evidence/R1) · FX100 Comprehensive LLM Inference and Training Benchmark (8 × AMD MI308X) (8 × AMD Instinct MI308X · ROCm 7.2)

## External references

- [vLLM Documentation](https://docs.vllm.ai/) · vLLM Project. Official documentation of vLLM deployment, prefix caching and tensor-parallel settings
- [AMD ROCm Documentation](https://rocm.docs.amd.com/) · AMD. The components of the ROCm software stack and its supported hardware