My laptop fan spins during a local model run: GPU offload or notSolved

QuestionsKacey3 repliesThis topic is about ollama/ollama

3

Jump to the accepted answer

KaceyMember#1

I run a local model on a laptop with 16 GB of memory and an integrated GPU. When I ask it anything long, the fan goes up and the whole machine slows down for the rest of the session.

The model is a 7B at Q4, and ollama ps shows the model is split between the GPU and the CPU. I assumed "on the GPU" would mean "the GPU is doing the work". What is actually happening, and what should I do about it?

3 replies

MarsMember#2

Accepted answer

What you are seeing is the split, and on an integrated GPU the CPU half is doing most of the work.

What the split means. A model that does not fit in the available memory has its layers divided: some run on the GPU, the rest on the CPU. Every token has to travel between the two, over a bus that is much slower than either device's own memory. The generation slows down, and because the CPU is doing the work, the laptop heats and the fan spins.

How to see the real split. ollama ps prints a PROCESSOR column, and the split is written as a percentage per device — 100% GPU means everything fits, 50%/50% CPU/GPU means half of it is on the CPU. The number is the diagnosis.

What to do about it, in order of preference.

  1. Use a smaller model or a smaller quant. The most effective change by a distance. A 7B at Q4_K_M and a 7B at Q2 are different machines. /tools/llm-vram-calculator tells you the sizes before you download anything.
  2. Shorten the context. The KV cache grows with the context you allocate, not with what you use. Cutting the context from 32k to 8k can bring a model back to 100% GPU.
  3. Give the server less memory to waste. If the GPU is shared with the display, a unified-memory laptop is giving some of it to the framebuffer. Running the model with a smaller context often fixes the split on its own.

What not to do. Do not assume the answer is the model size. A 7B at Q4 with 32k of context does not fit in 16 GB of shared memory even though the weights alone do, and the context is usually the thing that tipped it over.

MookMember#3

In reply to Mars (#2)

Worth adding the Apple silicon case, because it is the one where the answer is different.

With unified memory, the same 16 GB runs models that need a workstation GPU elsewhere, and the Metal backend is well maintained. The fan still spins, because the whole machine is doing the work, but the throughput is far better than the same machine on a discrete GPU with less memory.

The general rule on a laptop: the limit is memory, not the GPU, and the fan is a sign you are at the limit.

KaceyAuthorMember#4

In reply to Mook (#3)

It was the context, not the model. ollama ps showed 64%/36% CPU/GPU and cutting the context from 32k to 8k moved it to 100% GPU, and the machine is quiet again.

I had assumed the 7B at Q4 was the problem, and it was not. Accepting this, and the calculator is the tool I should have opened first.