The most useful thing we changed in our pipeline was not a better model. It was sending a smaller problem to a smaller model.
The split
Our request pipeline had one model doing four jobs: classify the request, extract the fields, call the right tool, and write the answer. Replacing that with a bigger model made it slower and more expensive and did not make the classification better.
So we split it. A small model classifies and extracts. A large model answers, but only for the requests that need it. The classifier is a 1B-class model, it runs on whatever is available, and it does the job well enough that we never had to think about it again.
Why a small model is good at this
Classification is a narrow, well-specified task with a fixed output space. The model is not reasoning; it is choosing between a small number of labels, and a small model is excellent at that. The capabilities that make a large model valuable — long context, multi-step reasoning, writing — are not the capabilities a classifier needs.
The second reason is latency. A 1B model on a small GPU answers in a fraction of the time, and if the classifier is on the critical path, that fraction is the whole user-visible latency of the first step.
The third is that the small model can be run locally. A classifier that decides whether a request touches billing, health data or anything else sensitive is exactly the kind of thing you do not want to send to a third party, and at that size you do not have to.
The number that made it stick
The cost of the pipeline is not the classifier. Even at a thousand times the price of a large model, a classifier that handles 90% of requests is noise in the total.
The thing that dominated the bill was always the answering step, and it always will be. Which is the argument for counting before you change anything: /tools/token-counter and /tools/llm-cost-calculator will tell you which part of your pipeline is actually costing money, and it is almost never the part you think.