Running models on your own computer: Ollama and quantisation

Part 1 called DeepSeek's API throughout. That's convenient, but in some situations you'll want the model running on your own machine: data that mustn't leave the company, no network, usage so high the API gets too expensive, or simply wanting to try it.

This lesson covers running open-source models on your own computer. The key question is: how big a model can my computer hold? The answer depends on two things: how many parameters the model has, and how many bytes each parameter is stored in.

python memory_estimate.py
python quantize.py

Estimating memory

A model's weights are a huge pile of numbers. Stored as 32-bit floats (FP32), each takes 4 bytes; as 16-bit (BF16 or FP16), 2 bytes. So the memory the weights take is roughly:

参数量 × 每个参数的字节数

Take Lesson 3's Qwen2.5-0.5B-Instruct (it has 494 million parameters) as an example:

== 1. Qwen2.5-0.5B-Instruct 的权重，在不同精度下占多少
  FP32        1.84 GB
  BF16/FP16   0.92 GB
  INT8        0.46 GB
  4 比特        0.23 GB

The file it downloads, model.safetensors, is 988 MB, exactly the BF16 size (988 MB is about 0.92 GB, the difference being 1000 versus 1024).

At run time, besides the weights, there's the KV cache from Module 09, Lesson 6. Its size can be computed from the model's configuration: for each token and each layer, one K and one V are stored:

def kv_cache_gb(n_layers, n_kv_heads, head_dim, n_tokens, bytes_per_value=2):
    # 每个词元、每一层，要存一个 K 和一个 V，各是 n_kv_heads × head_dim 个数
    return 2 * n_layers * n_kv_heads * head_dim * n_tokens * bytes_per_value / 1024**3

== 2. 它的 KV 缓存（每层 2 组 K/V，每组 64 维，BF16）
  每个词元 12 KB
    1000 个词元：0.01 GB
   32000 个词元：0.37 GB
  如果不用分组查询注意力（14 个头各存一份 K/V），32000 个词元要 2.56 GB，是现在的 7 倍

The last line is grouped-query attention at work: Qwen2.5-0.5B has 14 attention heads but only 2 groups of K and V, making the cache 7 times smaller. With long contexts, that difference is crucial.

For practical estimates, a simple rule: parameter count × bytes per parameter, plus 20% headroom for the cache and other overheads:

== 3. 粗略估算：参数量 × 每个参数的字节数，再留两成余量给缓存和其他开销
   0.5 B 参数：FP32    2.2 GB  BF16/FP16    1.1 GB  INT8    0.6 GB  4 比特    0.3 GB
     7 B 参数：FP32   31.3 GB  BF16/FP16   15.6 GB  INT8    7.8 GB  4 比特    3.9 GB
    14 B 参数：FP32   62.6 GB  BF16/FP16   31.3 GB  INT8   15.6 GB  4 比特    7.8 GB
    32 B 参数：FP32  143.1 GB  BF16/FP16   71.5 GB  INT8   35.8 GB  4 比特   17.9 GB
    70 B 参数：FP32  312.9 GB  BF16/FP16  156.5 GB  INT8   78.2 GB  4 比特   39.1 GB

This table can be used as is. A laptop with 16 GB of memory can only just run a 7B model in BF16; at 4 bits, even a 14B model runs. Long contexts need extra for the KV cache.

The table also shows why everyone talks about "quantisation": the same model at 4 bits needs only a quarter of what BF16 does.

Writing quantisation by hand

Quantisation means storing each parameter in fewer bits. The simplest approach is called symmetric quantisation: find the largest absolute value in a group of numbers, divide the range [-max, max] into evenly spaced steps, snap each number to the nearest step, and store only the step's index (a small integer) plus the group's scaling factor.

def quantize(w, bits, group=None):
    """对称量化：每组数用一个缩放系数，把 [-最大绝对值, 最大绝对值] 映射到整数 [-qmax, qmax]。
    返回"量化后再还原"的权重，以及实际要存的整数和缩放系数。"""
    qmax = 2 ** (bits - 1) - 1  # 8 比特是 127，4 比特是 7
    shape = w.shape
    w = w.reshape(-1, group) if group else w.reshape(shape[0], -1)  # 按组，或者按行
    scale = w.abs().amax(dim=1, keepdim=True) / qmax
    q = torch.round(w / scale).clamp(-qmax, qmax)  # 这就是要存下来的整数
    return (q * scale).reshape(shape), q, scale

An example: 8 numbers quantised to 4 bits (integers from -7 to 7):

== 1. 一个例子：把 8 个小数量化成 4 比特整数
  原来：   [0.077, -0.0147, -0.1089, 0.0284, -0.0542, -0.0699, 0.0202, 0.0419]
  整数：   [5, -1, -7, 2, -3, -4, 1, 3]（缩放系数 0.01556）
  还原后： [0.0778, -0.0156, -0.1089, 0.0311, -0.0467, -0.0623, 0.0156, 0.0467]

The largest, -0.1089, maps to -7, so the scaling factor is 0.1089 / 7 = 0.01556. The other numbers are divided by it and rounded. Multiplying back to restore them, most numbers are off by about 0.003; -0.0542 became -0.0467, a larger error.

Each number went from 32 bits to 4, at the cost of storing one extra scaling factor.

Quantising our small GPT

Quantise every matrix of the small GPT trained in Module 09 and see how much smaller and how much worse the model gets (LayerNorm and biases are tiny and stay as they are):

== 2. 整个模型量化之后
                                大小      验证损失
  32 位小数（原模型）             6.16 MB     4.476   白露起春光，知君得舞衣。雪时疑是静，山意势悠扬。
  8 比特，每行一个系数             1.58 MB     4.476   白露起春光，知君得舞衣。雪时疑是静，山意势悠扬。
  4 比特，每行一个系数             0.81 MB     4.542   云泉四百古，一树两三五。雪路之山在，山闾势悠扬。
  4 比特，每 32 个数一个系数        0.89 MB     4.512   白露起春光，知君得舞衣。雪时疑报晓，山下势悠扬。
  2 比特，每 32 个数一个系数        0.51 MB     6.933   百日起春光津倚郭对舞儿年，一之州在子。闾里里里里

Each row writes a poem with the same random seed, for easy comparison.

8 bits: a quarter of the original size, with a validation loss of 4.476, exactly the same as the original model, and even the poem is word for word identical. 8-bit quantisation is almost "free".

4 bits, one factor per row: half the size again, with the loss rising from 4.476 to 4.542, and the poem changes too. A row has hundreds of numbers, and if even one is especially large, it stretches the scaling factor, squeezing all the others into a few steps, so the error grows.

4 bits, one factor per 32 numbers: storing some extra scaling factors (the size goes from 0.81 MB to 0.89 MB) brings the loss back down to 4.512, and the poem mostly returns to the original, with just two changes. The smaller the groups, the better each group's scaling factor fits its numbers, and the smaller the error. The 4-bit quantisation used in practice is almost always grouped.

2 bits: only three values, -1, 0 and 1, are available, and the model is completely broken, with a loss of 6.93; what it writes has no format at all and ends up repeating "里里里里".

The conclusion matches practical experience: 8 bits is almost lossless; 4 bits loses a little but gives a quarter of the size, and is the most common choice for running models on your own computer; below that, quality falls off fast.

Real quantisation methods are more complex than this: they deal with especially large "outliers", use some data to decide how to quantise with the least error, and have data formats designed specifically for 4 bits (the QLoRA mentioned in Lesson 2 uses one called NF4). But the basic principle is these few lines of code in this section.

Running models with Ollama

Writing your own code to quantise and run LLMs is laborious. Ollama is a tool that packages all of it: one command downloads an already quantised model and runs it.

Installation (as of September 2026; follow the official instructions):

# macOS 和 Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows（PowerShell）
irm https://ollama.com/install.ps1 | iex

On macOS and Windows you can also download an installer from ollama.com.

Run Lesson 3's Qwen2.5-0.5B:

ollama run qwen2.5:0.5b

The first run downloads the model automatically, then drops you into a chat. As of September 2026, qwen2.5:0.5b in Ollama's model library defaults to a Q4_K_M quantised version of 398 MB, less than half the original BF16's 988 MB. Q4_K_M is a 4-bit grouped quantisation format defined by llama.cpp, the same idea as the hand-written "4 bits, grouped" above, just more refined.

The same model usually has several quantised versions to choose from, such as qwen2.5:0.5b-instruct-q8_0 (8 bits) and qwen2.5:0.5b-instruct-fp16 (16 bits, unquantised). Choose the right version using the memory estimates above.

Calling a local model through the API

Once running, Ollama serves an OpenAI-compatible API on port 11434 of your machine. That means all the code written in Part 1 can switch to the local model by changing three environment variables:

export LLM_BASE_URL=http://localhost:11434/v1
export LLM_API_KEY=ollama        # 本地服务不检查密钥，但 openai 库要求有一个值
export LLM_MODEL=qwen2.5:0.5b

Module 00, Lesson 2 put the model's address, key and name in environment variables precisely for this moment.

But keep your expectations reasonable: we saw in Lesson 3 that the 0.5B model gets Beijing's geography wrong. The small models that run smoothly locally fall well short of big models like DeepSeek on complex tasks. Module 06's evaluation set comes in handy again here: switch to the local model, run it, and see which tasks it can still handle.

Using your own fine-tuned model

Ollama can also run your own models. It can import models in Hugging Face format (safetensors) and in GGUF format, by writing a configuration file called a Modelfile whose FROM points at the model files, then creating it with ollama create.

For the RepoBot fine-tuned in Lesson 3, you can first merge the LoRA back into the original model (merge_and_unload) and save it, then import it following the official docs. The official docs specifically note that Ollama won't quantise an imported GGUF model for you; you need to quantise it first with llama.cpp's tools. Follow Ollama's import documentation for the exact steps, as this part changes quickly.

Exercises

1. Check your computer's memory (or your graphics card's VRAM). Using this lesson's estimate table, what's the largest model you can run? At 4 bits and at 8 bits?
2. In quantize.py, try 3 bits with groups of 32, and 4 bits with groups of 8 and 128, and tabulate size and validation loss.
3. Install Ollama, run a small model, and switch Module 03's RepoBot v1 to call it (changing only environment variables). See whether it works properly.

Self-check

1. How do you estimate how much memory a 7B-parameter model takes in BF16?

Parameter count times bytes per parameter: 7 billion × 2 bytes = 14 billion bytes, about 13 GB. Add the KV cache and other overheads with 20% headroom, and it's about 16 GB.

2. In 4-bit quantisation, why does "one scaling factor per 32 numbers" work better than "one scaling factor per row"?

The scaling factor is set by the largest absolute value in the group. A row has hundreds of numbers, and if even one is especially large, the whole row's scaling factor is stretched, so the others are squeezed onto very few integers and the error is large. Split into small groups, each group's scaling factor fits its numbers better and the error is much smaller, at the cost of storing a few more scaling factors.

3. Why does Part 1's code need almost no changes to switch to a local Ollama model?

Ollama provides an OpenAI-compatible API, and Part 1's code uses the openai library with the API address, key and model name in environment variables. Change those three variables to Ollama's address and model name, and the code calls the local model directly.