We write a lot of text with models. Before each piece goes out, three lines of code run, and they have caught more mistakes than any amount of proofreading.
The three lines
The script counts the tokens of the text, of the system prompt, and of the example that will be attached to it. Then it prints the three numbers and the cost.
That is the whole thing. The reason it exists is not that the model fails, it is that the model succeeds at a length nobody noticed, and the length is what you pay for.
What it caught
A prompt that had grown to nine hundred tokens and was being sent on every request, in a system that was doing 400 requests a day. Not wrong, just expensive, and invisible because the prompt was in a file that everybody had stopped reading.
An example that was longer than the text it was an example of. A 300-token few-shot example attached to a 40-token answer. It had been copied from a longer draft and never trimmed, and it was being paid for on every call.
A page whose conclusion was longer than its body. The conclusion is the part people read, and it was fine; the body underneath was three times as long, and the body was the part that had to be generated.
None of these is a mistake a careful reader would catch, because a careful reader reads the text and the text is good. The number is the thing that is not obvious.
Why a script and not a tool
There is a token counter on this site, and it does the same thing for a person. The script exists because the cost of running it is zero and the cost of remembering to run it is not.
That is the real lesson from the whole exercise: the number has to arrive on its own, at the moment the thing is about to happen. A tool somebody has to remember to open is a tool that gets opened once and then forgotten.