Questions3 replies
I have a system prompt that says "answer in British English, never use bullet points, and cite the source paragraph". The model does all three things it was told not to do, and it does them more often the longer the conversation gets.
The prompt is about 200 words. It is the first thing in the request, in the system role, and I am sending it on every call.
What am I missing?
Accepted answer
Three separate things, and they are worth separating because the fixes are different.
1. The prompt is fighting the model's training. A rule the model was trained to ignore — a formatting rule that contradicts the default — is the first thing to lose in a long context. The system prompt is not a setting; it is the first paragraph of a conversation, and it competes with everything after it. "Never use bullet points" is much weaker than "reply in one paragraph".
2. The rules are competing with each other. Three rules in one prompt means the model has to trade them off when they conflict, and the one that loses is usually the one with the fewest words. Split a long prompt into named sections — task, format, constraints — and put the format rule last, next to the question, where the model's attention is.
3. The rules are in the wrong place for a long conversation. A system prompt that is 200 words and then 4,000 words of conversation is the least attended part of the request. The fix is to restate the expected format on every turn, near the end, or to put the format into the tool or the schema rather than into prose.
The test. Take a conversation that produced a bad answer and re-send it with the instruction moved to the end. If the output changes, the problem was position. If it does not, the problem is that the instruction contradicts the model's prior. That is one experiment and it tells you which of the two you are dealing with.
Worth adding the case where the rule is technically obeyed and practically ignored: "cite the source paragraph" when the model has no paragraph numbers. If there is nothing to cite, the model will produce something that looks like a citation. Give it a format it can fill in, like the chunk id, and it will do it correctly.
The end-of-request test settled it. Moving the format rules to the end of the last user message changed the output immediately, and the British English instruction started holding once I stopped asking for three things in one paragraph of prose.
I also took the citation rule out of prose entirely: the tool now returns a chunk_id field and the model has to repeat it, and it cannot invent one because it does not have the field. Accepted, thank you.