PromptCache: a self-hosted semantic cache in front of your LLM provider
Cut LLM costs by up to 80% and unlock sub-millisecond responses with intelligent semantic caching.A drop-in, provider-agnostic LLM proxy written in Go with sub-millisecond response
At a glance
- What is it?
- PromptCache is a Go proxy that embeds prompts, compares them against a BadgerDB-backed store, and answers near-duplicates locally. It is useful when your traffic repeats, and risky when your traffic is user-specific.
- Who is it for?
- Adopt PromptCache if you run a RAG service, support bot or agent loop where the same questions recur and you can tolerate probabilistic matching. Do not adopt it if prompts carry per-user or per-tenant data, or if you need cache hits to respect authorization boundaries, because the README states plainly that semantic similarity is not an authorization mechanism.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 29 days ago.
- What is it written in?
- Mainly Go, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The repetition problem PromptCache targets
Production LLM traffic is not as varied as it looks. The README names three patterns: RAG applications with recurring internal questions, agents that repeat reasoning or tool-use steps, and support bots fielding similar customer questions. In each case the same generation is bought from a provider over and over, and the bill scales with request count rather than with the number of distinct answers.
PromptCache is aimed at the engineer who owns that bill. It is a self-hosted proxy, written in Go, that sits between the application and the provider. The application keeps speaking the OpenAI chat completions shape; PromptCache decides whether the request needs an upstream call at all. The project description claims cuts of up to 80% in LLM cost and sub-millisecond responses, but the README is more careful: it presents a table of typical effects (lower provider usage, lower latency, more rate-limit headroom) and says actual results depend on provider, model, prompt distribution, hit rate, hardware and configuration. The 80% figure belongs to the marketing line, not to the documented benchmark output.
That distinction matters when you are deciding whether to put this in a request path. The honest framing is that PromptCache converts some fraction of your traffic into local reads, and the size of that fraction is a property of your workload, not of the software.
Two thresholds and a gray zone: the matching mechanism
Matching is not exact-string. PromptCache embeds the incoming prompt and compares it against stored embeddings. The README describes a three-band strategy controlled by two environment variables. A similarity score at or above CACHE_HIGH_THRESHOLD is a direct hit and the cached response is returned. A score below CACHE_LOW_THRESHOLD is a clear miss and the request goes upstream. Scores in between fall into a gray zone, where an optional smaller model verifies intent before the cache is trusted. The documented defaults are 0.70 and 0.30, and the README states the invariant that must hold: CACHE_HIGH_THRESHOLD must stay above CACHE_LOW_THRESHOLD.
ENABLE_GRAY_ZONE_VERIFIER toggles that third band. Turning it off removes provider calls and latency from the gray zone, and the README says it may also reduce matching accuracy. That trade is the core tuning decision in the product: the verifier is what stops a merely similar prompt from receiving someone else's answer, and it costs a model call each time it fires.
The README is explicit that this is probabilistic. It states that semantic matching does not guarantee two prompts are interchangeable. There is no exact-match fast path documented, no per-key namespacing described, and no way to pin a request to a specific cached entry. If you need deterministic reuse, hashing the prompt yourself and storing responses in Redis is a smaller and more predictable system.
Storage, providers and the request path
Cached prompts, responses, embeddings and metadata live in BadgerDB, an embedded key-value store, so there is no separate cache server to run. The README says the TTL is configurable and defaults to 24 hours. Persistence is on disk, which means the BadgerDB directory and its backups hold prompt and response text and should be protected like any other store of application data.
Provider selection is a single variable, EMBEDDING_PROVIDER, with three documented values. openai uses text-embedding-3-small for embeddings and gpt-4o-mini for verification. mistral uses mistral-embed and mistral-small-latest. claude is the odd one out: it embeds through Voyage AI with voyage-3 and verifies with claude-3-haiku-20240307, so it needs both ANTHROPIC_API_KEY and VOYAGE_API_KEY set. Provider names indicate API compatibility only, and the README states the project is not affiliated with OpenAI, Anthropic, Mistral AI or Voyage AI.
Streaming is handled asymmetrically. On a miss, PromptCache forwards the provider stream and buffers the assembled response for caching. On a hit, it synthesizes OpenAI-compatible SSE chunks from the stored response. The buffering on the miss path is worth noting: the full response is held in memory to be written to BadgerDB, which is a different memory profile from a pass-through proxy.
Getting it running: Docker, source, and the variables that matter
The README gives two paths. With Docker: clone the repository, export EMBEDDING_PROVIDER and the matching API key, export API_AUTH_TOKEN, then run docker-compose up -d. From source: the same exports followed by ./scripts/run.sh. The README also lists make run, or a manual build with go build -o prompt-cache cmd/api/main.go followed by ./prompt-cache. The Go badge indicates Go 1.24 or newer.
Client-side integration is the selling point. Point an OpenAI-compatible SDK at http://localhost:8080/v1 and pass any API key value; the README's Python example uses the openai package with base_url set to that address. First request goes upstream, later sufficiently similar requests may be served locally.
Management endpoints are protected by Bearer-token auth when API_AUTH_TOKEN is set. The README lists /metrics, /v1/stats, /v1/config, /v1/config/provider, /v1/cache and /v1/cache/warm as protected, and shows the curl pattern with an Authorization: Bearer header against /v1/stats. If the variable is unset, management authentication is disabled and PromptCache logs a warning. The README recommends setting it for every non-local deployment, and that recommendation is not optional in practice: an open /v1/config endpoint exposes your runtime configuration.
Note the boundary the README draws. /v1/chat/completions is not an application-level authorization system. PromptCache should sit behind whatever authentication and authorization your application already enforces.
Where PromptCache is the wrong tool
The most serious limitation is stated by the project itself: semantic similarity is not an authorization mechanism, and cache matching must not be used as a security boundary between users, tenants or authorization scopes. If two users ask similar questions about different accounts, a high similarity score can hand one user a response generated for the other. The README points to RESPONSIBLE_USE.md for sensitive or multi-user deployments, and that document is the thing to read before deciding, not the feature list.
The second limitation is the gray-zone verifier. With it enabled, ambiguous prompts still cost a small model call, so the saving is smaller than the hit rate suggests. With it disabled, you save the call and accept weaker matching. There is no documented third option that removes the ambiguity.
Third, the benchmark numbers in the README are micro-benchmarks of the matching primitives, not end-to-end figures. BenchmarkCosineSimilarity reports 441.0 ns/op with zero allocations, and BenchmarkFindSimilar reports 32000 ns/op with 2048 B/op and 45 allocs/op. The README says to treat these as examples and to run the included suite against your own workload. Neither number includes embedding generation, which requires a network call to the embedding provider on every incoming request unless the prompt is an exact repeat. That embedding call is the hidden cost of the whole design, and the README does not quantify it.
Finally, the version history shows three releases in roughly eight months, with v0.4.0 in April 2026 adding authentication, SSE streaming, runtime threshold configuration and cache warming. Those are recent additions, and the README's own warning about unset API_AUTH_TOKEN suggests the auth layer is young.
Alternatives and the actual difference in approach
The closest comparison is LangChain's caching layer, which the repository's topic list references. LangChain's caches (in-memory, SQLite, Redis, and others) are exact-match by default: the same prompt string returns the stored response. That is deterministic and cheap to reason about, and it needs no embedding provider and no similarity thresholds. PromptCache's difference is that it embeds the prompt and matches on vector similarity, so paraphrases can hit. The cost of that capability is the embedding call on every request, a probabilistic hit decision, and the gray-zone verifier as a correctness backstop.
A second option is a plain Redis or Memcached layer in front of your provider client, keyed on a hash of the normalized prompt. This is the right choice when your traffic is genuinely repetitive with identical strings, when you cannot send prompts to a third-party embedding API, or when you need cache keys to encode tenant identity explicitly. PromptCache does not document a way to include tenant identity in the cache key, which is precisely why the README frames matching as unsuitable for authorization boundaries.
A third option is provider-side prompt caching, where the provider itself discounts repeated prefixes. That keeps data inside the provider relationship you already have and requires no new service, but it only helps with shared prefixes, not with semantically equivalent questions phrased differently. PromptCache's value proposition is the paraphrase case specifically.
Maintenance cost, licence and what to verify
The licence is MIT, which permits commercial use, modification and redistribution with the copyright notice and licence text retained. That is a permissive arrangement, but it says nothing about the terms of the embedding and verification providers you configure. Sending prompts to OpenAI, Mistral, Voyage AI or Anthropic for embedding is governed by those providers' data-handling terms, and PromptCache's own README directs you to RESPONSIBLE_USE.md for prompts containing personal, confidential, regulated, user-specific or tenant-specific information. This is not legal advice; the point is that the MIT grant covers the proxy code only.
Operationally, you are adding a stateful service to the request path. BadgerDB data must be backed up and protected. The TTL defaults to 24 hours and is configurable, but the README warns that a TTL is not a substitute for a data-retention policy. Upgrades come as tagged releases (v0.2.0, v0.3.0, v0.4.0), and the jump to v0.4.0 changed authentication and streaming behaviour, so read the release notes before moving.
The first thing to verify is your hit rate under your own thresholds. Run the repository's benchmark suite, then instrument /v1/stats against a shadow deployment before putting PromptCache in front of real traffic. The second is whether your prompts carry anything that should not be shared across a cache. If they do, the answer is a tenant-scoped key store you control, not a similarity threshold.
Editorial conclusion
Adopt PromptCache if you run a RAG service, support bot or agent loop where the same questions recur and you can tolerate probabilistic matching. Do not adopt it if prompts carry per-user or per-tenant data, or if you need cache hits to respect authorization boundaries, because the README states plainly that semantic similarity is not an authorization mechanism. Before rollout, set API_AUTH_TOKEN, keep CACHE_HIGH_THRESHOLD above CACHE_LOW_THRESHOLD, and run the repository benchmark suite against your own prompt distribution to see what your hit rate actually is.
Community notes