AI Tools Write

Which GGUF quant for a 24 GB card: Q4_K_M, Q5_K_M or Q6_K?Solved

QuestionsKacey3 repliesThis topic is about ggml-org/llama.cpp

3

Jump to the accepted answer

KaceyMember#1

Every llama.cpp model page on Hugging Face lists a dozen GGUF files and the names tell you nothing unless you already know. With one 24 GB card, what is the sensible way to choose?

What I think I understand:

  • The letter and number say bits per weight, roughly: Q4 around 4.5–4.8 bits, Q5 around 5.5, Q6 around 6.6, Q8 at 8.5. _K_M and _K_S are variants of the "k-quant" scheme with different mixes of precision across layers.
  • Lower bits means a smaller file and more room for context, at the cost of some quality.

What I do not understand is where the cliff is. Is Q4_K_M "fine" and Q6_K "almost lossless", or is that folklore? And for a 24 GB card specifically, am I better off with a bigger model at Q4 or a smaller one at Q6?

If it matters: this is for a coding assistant, mostly long files in context, not for chat.

3 replies

MarsMember#2

Accepted answer

Rule of thumb that has held up:

  • Q4_K_M is the default for a reason. Measured perplexity loss against the full-precision model is small; most people cannot tell in use.
  • Q5_K_M buys back part of the remaining difference for roughly 15–20% more memory.
  • Q6_K is close to indistinguishable from Q8, and Q8 is close to indistinguishable from FP16. Above Q5 you are paying memory for differences you will struggle to measure.
  • Below Q4 (Q3_K, IQ2) the degradation becomes visible, especially on small models. A 7B at Q2 is worse than a 3B at Q6.

So on a 24 GB card the question is really "which parameter count fits at Q4_K_M with the context I need". Approximate weights only: a 14B model at Q4_K_M is about 9 GB, Q6_K about 12 GB; a 32B at Q4_K_M is about 19–20 GB. The KV cache comes on top and scales with context, so a 32B at Q4 on 24 GB leaves you a few GB for context, which is tight for "long files". A 14B at Q6_K with 32k of context is the more comfortable fit for your use.

/tools/llm-vram-calculator does the weights-plus-KV sum for the exact model, and /learn/quantization and /learn/gguf explain the file naming. In llama.cpp, -c sets context, -ngl the layers on GPU, -fa turns on flash attention, and --cache-type-k q8_0 quantises the KV cache itself if you need to squeeze.

EvieMember#3

In reply to Mars (#2)

One nuance for the coding case: quality losses from quantisation show up first in exact-token tasks, and code is exact-token. A hallucinated bracket is a syntax error, where a slightly worse word choice in prose is invisible. So for code I would lean to Q5_K_M or Q6_K on the model size that still leaves room for the context you actually paste, rather than maximum parameters at Q4. The context being too short hurts more than any quant level.

KaceyAuthorMember#4

In reply to Mars (#2)

That is the mental model I was missing: pick the parameter count that fits at Q4, then spend the leftover on context or on a step up in quant, in that order. Accepting this one. Thanks both.

The project behind this thread

More in Questions

Open-source projects

Concepts

Free browser tools

From the AI course

Topic hubs