Pricing a RAG pipeline before building it: token counter, cost calculator and chunker together

ShowDevn Nie3 replies

Devn NieMember#1

A workflow we use when reviewing "we built a RAG system" write-ups, and which works for planning your own. Three tools on this site, in order:

  1. /tools/rag-text-chunker: paste a representative document, pick a chunk size and overlap, see the chunks and how many there are. This is where you discover that your 400-token chunks split tables in half, or that 50% overlap doubles your index for little gain.
  2. /tools/token-counter: count the tokens of a chunk, of the system prompt, and of a typical question. Tokenisers differ between model families; a count from one is a rough guide for another, which is why the tool lets you pick.
  3. /tools/llm-cost-calculator: input tokens per request (system prompt + retrieved chunks × top-k + question), output tokens, requests per day, and the model's price. It gives the monthly figure you can put in front of whoever pays.

Two things the numbers usually reveal. The embedding cost of indexing is small compared with the generation cost of answering, so "we re-index nightly" is cheap and "we retrieve 20 chunks" is expensive. And a system prompt that is 1,500 tokens long is in every request; trimming it is often the single biggest saving.

Caveat: prices change, and the calculator uses the price you type in. Check the vendor's page that day rather than a figure from a blog post. /learn/embeddings and /learn/vector-database cover the vocabulary if any of this is new.

3 replies

RonnyMember#2

Suggestion for the calculator: a field for cached input pricing. Several providers charge much less for the part of the prompt that repeats, which for RAG is the system prompt and often the instructions around the chunks. The gap between "all input at full price" and "repeated prefix cached" can change the monthly figure a lot.

Devn NieAuthorMember#3

In reply to Ronny (#2)

Good call and not hard: a "cached share of input" percentage and a cached price. Will add it; the models that offer it differ in how they define the cacheable prefix, so the tool will say "if your provider caches this" rather than promise it.

EricMember#4

For the latency side of the same plan, /tools/llm-speed-estimator takes tokens per second and output length and gives you time to first token and total time; worth doing alongside the cost, because a cheap model that makes the user wait eight seconds has its own cost.