Simba: A Self-Hosted Customer Service Assistant With Evals Built Into the Pipeline
OpenSource Production ready Customer service with built in Evals and monitoring
At a glance
- What is it?
- Simba is an Apache-2.0 TypeScript and Python project that combines a RAG customer service stack with an evaluation framework and an npm chat widget. The interesting part is the eval layer; the uncertain part is how much of the production claim you can verify before deploying.
- Who is it for?
- Adopt Simba if you want a self-hosted RAG assistant where retrieval and generation metrics are part of the product rather than an afterthought, and if you are comfortable reading source to confirm behaviour the README does not specify. Do not adopt it if you need multi-tenant isolation today (the roadmap lists it as unchecked) or if you want a managed service with a support contract.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 90 days ago.
- What is it written in?
- Mainly TypeScript, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The Problem Simba Targets: Unmeasurable RAG Assistants
Most open source customer service bots ship as a retrieval pipeline plus a chat UI. You point them at documents, they answer, and you have almost no way to tell whether a change to the chunking strategy made answers better or worse. Simba's README frames this directly: the first row of its problem table is "Can't measure AI quality," answered with a built-in evaluation framework covering retrieval and generation metrics. The stated audience is teams who want "full control" over their AI rather than a black-box vendor product, and who are willing to run the infrastructure themselves. That framing matters because it sets the adoption bar. This is not a tool for a support team that wants to paste a script tag and be done. It is for an engineering team that already has opinions about embeddings, rerankers and chunk sizes, and wants a harness to test those opinions against real traffic.
Architecture: FastAPI, Celery, Redis and a Swappable Vector Store
The README's architecture diagram shows four moving parts. The npm widget talks to a FastAPI backend. The backend queries a vector store, listed as Qdrant or FAISS in the diagram and expanded to Qdrant, FAISS or Chroma in the customization table. Generation goes to an LLM that can be OpenAI or a local model. Document ingestion runs separately: a Celery worker pulls tasks from a Redis queue and writes into the same vector store the API reads from. That split is the design decision worth noting. Ingestion is asynchronous and decoupled from query serving, so a large PDF upload does not block chat requests. It also means the system has a queue to operate. If Redis is down, ingestion stops while chat keeps working, which is a reasonable failure mode but one you have to monitor. The customization table lists five swappable component classes: vector store, embeddings, LLM, reranker and parser, with parser options including Docling, Unstructured and PyMuPDF. Nothing in the supplied material describes how these are selected at runtime, whether through environment variables, a config file or code changes, so treat the swap claim as a claim about interfaces rather than a documented config surface.
Getting It Running: Docker, pip, or Claude Code
The README gives three paths. The Docker path is marked recommended: clone the repository, create a .env file containing OPENAI_API_KEY, then run DEVICE=cpu make build && make up, or DEVICE=cuda make build && make up for an NVIDIA GPU. The dashboard is served at http://localhost:3000. The manual path is pip install simba-core followed by simba server and simba front as two separate commands, which implies the API and the Next.js frontend run as distinct processes. The third path is a Claude Code slash command, /setup --all, with narrower variants for --backend, --frontend and --services. The existence of a --services flag confirms that Docker infrastructure is separate from the application processes even in the manual workflow. Website integration is a single npm package: npm install simba-chat-widget, then import SimbaChat and render it with an apiUrl prop pointing at your instance and a theme prop. Note what the README does not give you: no example of the .env beyond the OpenAI key, no port list for the API, no description of how documents get into the knowledge base on first run. You will be reading the repository to fill those gaps.
What the Evaluation Framework Actually Claims to Measure
Simba's evaluation section lists three groups. Retrieval metrics: precision, recall and relevance scores. Generation metrics: faithfulness, answer relevancy and latency. Conversation analytics: user satisfaction and resolution rates. The first two groups map onto standard RAG evaluation vocabulary, and the third is the one to interrogate. Satisfaction and resolution rate are only meaningful if something captures them, either an explicit user signal or an inference from conversation outcome. The README does not say which. There is also no description of where evaluation datasets come from, how a run is triggered, or whether results are stored per pipeline configuration so you can compare a reranker change against a baseline. The roadmap marks "Core evaluation framework" as complete, so the capability is claimed as shipped, but the documentation stops at naming the metrics. For a project whose main differentiator is evaluation, that is the thinnest part of the README and the first thing to inspect in the source.
Limitations and Cases Where Simba Is the Wrong Choice
The roadmap is the clearest statement of what is missing. Multi-tenant support is unchecked. If you are building a single assistant that serves many customers with isolated knowledge bases, the current version does not claim to do that, and retrofitting tenant isolation into a vector store schema is not a small change. Advanced analytics, webhooks and fine-tuning are also unchecked. Beyond the roadmap, the production claim in the feature list is asserted rather than evidenced in the supplied material: streaming responses and async processing are named, but there are no published latency figures, no load test description and no guidance on concurrent user capacity. The default .env example sets an OpenAI key, so the out-of-the-box path sends your customers' questions to a hosted model; local model support is listed as an option but not documented as a default. Finally, the project is young. Releases v0.2.0 through v0.4.0 landed within five days in March 2025, which suggests fast early iteration and a correspondingly unsettled API surface. If you need a stable integration contract today, that release cadence is a risk, not a feature.
How Simba Differs From a General RAG Framework
The obvious comparison is a general-purpose RAG framework such as LangChain or LlamaIndex, where you assemble the pipeline yourself and evaluation is a separate library you bolt on afterward. Simba inverts that: the assistant, the ingestion queue, the dashboard and the widget come as one deployable unit, and the eval metrics are described as part of the product rather than an add-on. The trade is flexibility at the edges. With a general framework you can reshape the control flow arbitrarily; with Simba you work inside its FastAPI and Celery structure and swap the components its customization table names. The other comparison is a hosted customer service platform, where you get multi-tenancy, analytics and support on day one and give up the self-hosted requirement and the ability to inspect the retrieval logic. Simba sits between those two: more opinionated than a framework, less complete than a hosted product. The README's own framing, "swap any component," is the honest summary of where it lands.
Licence, Maintenance and Upgrade Considerations
Simba is Apache-2.0, which permits commercial use, modification and redistribution provided you preserve the licence and notices, and it includes an explicit patent grant. That is a permissive licence with no copyleft obligation on your own code, but it is not legal advice and the terms should be read in full if you plan to redistribute a modified version. Maintenance cost is the harder question. The repository was pushed in June 2026, after the March 2025 releases, so the project is not abandoned, but the supplied material does not show a release between v0.4.0 and that push, which means changes may be landing on main without tagged versions. If you pin to a tag, you get reproducibility and you forgo fixes. If you track main, you inherit whatever the current state is. The dependency surface is the other ongoing cost: a Python package, a Next.js frontend, an npm widget, plus Docker services for the vector store and Redis. Upgrading the vector store or the LLM client is your responsibility, not the project's, because you are the operator.
Editorial conclusion
Adopt Simba if you want a self-hosted RAG assistant where retrieval and generation metrics are part of the product rather than an afterthought, and if you are comfortable reading source to confirm behaviour the README does not specify. Do not adopt it if you need multi-tenant isolation today (the roadmap lists it as unchecked) or if you want a managed service with a support contract. Before deploying, verify three things in the repository: how evaluation datasets are created and stored, whether the Celery and Redis ingestion path survives a worker restart mid-job, and which vector store backend the Docker Compose file actually configures by default.
Community notes