Token counting is not billing: what a token counter can and cannot tell you

AI Tools & ModelsNico3 replies

NicoMember#1

A token counter tells you how many tokens a piece of text is. It does not tell you what you will be charged. The gap between those two is where most cost surprises live.

What the counter knows

The exact token count for the tokeniser of a given model family, and where each token starts and ends. That is genuinely useful information, and it is more useful than the number: seeing that one word in your prompt is twelve tokens tells you what to cut.

The limit of that knowledge is that the count is a count of one thing. The bill is a function of several.

What the counter does not know

Which tokens are cached. Providers charge less for a repeated prefix. A counter on the prompt tells you nothing about whether the prefix that came before it was already cached, and that difference can be larger than the whole prompt.

What the model actually returns. The input is known before the call; the output is not. Output tokens are often priced higher than input tokens, and the length of a response is a property of the model and the task, not of your prompt. A prompt of 200 tokens can produce 20 tokens or 2,000, and the counter cannot tell you which.

What else is in the request. Tool definitions, the chat template's markers, a system prompt that is added by a library you do not control, a schema. All of them are billed and none of them are in the text you handed to the counter.

The per-minute and per-day limits. Rate limits are a separate constraint from cost, and a workload that fits the budget can still be throttled.

What to do with it

Use the counter for what it is good at: finding the expensive parts of your text, and comparing two versions of a prompt. Then use the cost calculator to turn a token count and an assumed output length into a monthly figure, and treat that figure as an estimate with a stated assumption rather than a number.

The honest summary: the counter is exact about the input and silent about everything else, and the cost is mostly in the parts it is silent about.

3 replies

RonnyMember#2

The cached-prefix point is the one that makes the gap concrete, and it is the reason I would not trust a single number from a cost model.

A prompt of 4,000 tokens that hits a cached prefix costs a fraction of one that misses, and the two are indistinguishable from the text alone.

EdenMember#3

In reply to Ronny (#2)

The output-length point deserves to be repeated because it is the biggest single term for most agent workloads.

A tool-using agent makes five calls, each with a large prompt, and each call produces a small output. The input dominates for an agent and the output dominates for a chat, and a cost model that assumes one ratio will be wrong for half the traffic.

NicoAuthorMember#4

In reply to Eden (#3)

That is the right way to use the calculator: two numbers, one for the input side and one for the output side, rather than one blended rate.

The blended rate is what a price page gives you, and it is fine for a rough monthly figure, but it hides the two levers that actually differ between workloads. Measure them separately and the estimate gets much better for very little work.