Ollama answers as if the context were 4k even though the model card says 128k. Where is the limit set?Solved

Help & QuestionsEden3 repliesThis topic is about ollama/ollama

Jump to the accepted answer

EdenMember#1

While checking the Ollama project page I ran into something that confused several people in the project's own issue tracker as well, so it belongs here.

You pull a model whose card says 128k context, you send it a long document, and it clearly forgets the beginning. ollama show <model> prints a context length in the tens of thousands or more, so it looks like the model supports it. Yet the answers behave as if only the last few thousand tokens were there.

My understanding so far:

  • Ollama does not use the model's maximum by default. The server applies its own default context (num_ctx), which was 2048 for a long time and is 4096 on recent builds.
  • The model card number is what the model can do, not what the server allocates.

What I am unsure about is the recommended place to raise it. Options I have seen mentioned: a PARAMETER num_ctx line in a Modelfile, options.num_ctx on each API request, or an environment variable on the server. Which one should a self-hoster reach for first, and does the OpenAI-compatible /v1 endpoint honour any of them?

3 replies

TerryRegular#2

Accepted answer

All three exist and they apply at different layers:

  1. Per request on Ollama's native API (/api/generate, /api/chat): pass "options": {"num_ctx": 32768}. Good for testing, useless if the client is something else talking OpenAI-style.
  2. Per model: a Modelfile with
FROM llama3.1
PARAMETER num_ctx 32768

then ollama create llama3.1-32k -f Modelfile. Every client that uses that model name gets the context. This is the one to reach for first. /tools/ollama-modelfile-generator on this site writes the file for you.
3. Server-wide: set OLLAMA_CONTEXT_LENGTH in the environment of the ollama serve process. It changes the default for every model that does not set its own.

The OpenAI-compatible /v1 endpoint does not take num_ctx, so for clients like Open WebUI or an SDK pointed at /v1, use option 2 or 3.

One warning: the KV cache grows with the context you allocate, not with what you use. Going from 4k to 32k on a 8B model can add several gigabytes. Check /tools/llm-vram-calculator before you set 128k on a 12 GB card, and see /learn/kv-cache for why.

IvanMember#3

In reply to Terry (#2)

Adding to the warning: if the allocated context does not fit on the GPU, Ollama will quietly split layers to the CPU rather than fail. Throughput falls off a cliff and it looks like "the model got slow" rather than "the context is too big". ollama ps shows the processor split (100% GPU versus 48%/52% CPU/GPU), which is the quickest way to notice.

EdenAuthorMember#4

In reply to Terry (#2)

Thanks, that settles it. Modelfile per model, environment variable only on a box that serves one kind of workload. Marking this as the answer. I will also add a line about this to the project page's notes, since the "128k" on the card is exactly what sends people here.

The project behind this thread

More in Help & Questions

Open-source projects

Concepts

Free browser tools

From the AI course

Topic hubs