rag-chatbot: a local Markdown RAG stack built on llama.cpp and Chroma
RAG (Retrieval-augmented generation) ChatBot that provides answers based on contextual information extracted from a collection of Markdown files.
At a glance
- What is it?
- This project wires a local llama.cpp server to a Chroma vector store so a chatbot answers questions from a folder of Markdown files. Its distinguishing feature is incremental indexing with SQLite-backed deletion tracking, and its main constraint is that swapping embedding models forces a full rebuild.
- Who is it for?
- Adopt this if you want a local, Apache-2.0 RAG pipeline over Markdown with no LangChain dependency, and you are on Ubuntu with CUDA 12.4+ or Apple Silicon. Do not adopt it if you need a managed vector store, a Windows path, or a corpus whose embedding model might change later without a rebuild.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 95 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem: answering questions from a Markdown corpus you already own
Most documentation teams already have the knowledge base. It sits in a docs folder as Markdown files. The gap is retrieval: a general-purpose chatbot has no access to those files, and pasting them into a prompt does not scale past a handful of pages. This project closes that gap with a retrieval-augmented generation pipeline that reads Markdown, embeds it, and answers questions using the retrieved passages as context.
The target user is an engineer or small team who wants to run the whole thing locally. The README lists Python 3.12+, Poetry 2.3.0+, Docker, and either a CUDA 12.4+ GPU or Apple Silicon. There is no hosted API in the loop. The language model runs through llama.cpp, and the embeddings run through Sentence Transformers. That choice buys privacy and predictable cost, and it costs you hardware. The README is explicit that the code was tested on Ubuntu 22.04.2 LTS with an NVIDIA GeForce RTX 3060 and on macOS Sonoma 14.3.1 with an M1, and it points anyone on different hardware to llama.cpp's own issue tracker rather than claiming broad support.
How retrieval works: question rewriting before the vector lookup
The pipeline has two phases. The Memory Builder loads Markdown pages from the docs folder, splits them into smaller sections, computes embeddings with Sentence Transformers, and stores them in Chroma. The README notes that the chunking uses a RecursiveCharacterTextSplitter class taken from LangChain and refactored into the project so that LangChain is not a dependency. That is a deliberate trade: you get a known chunking behaviour without pulling in a large framework.
At query time the flow is not a straight embed-and-search. The README states that because the original question is not always a good retrieval query, the system first prompts an LLM to rewrite the question, then performs retrieval-augmented reading. The rewritten query goes to Chroma, the most relevant sections come back, and those sections become the context for the final answer generated by the local model. Chat history is persisted as well, and prior turns are considered when producing the answer.
The part worth studying is context overflow handling. Two strategies are implemented. Create And Refine synthesizes a response sequentially through all retrieved contents, which means one generation pass per retrieved chunk in sequence. Hierarchical Summarization generates an answer for each relevant section independently and then combines those answers hierarchically. The first trades latency for a single coherent thread; the second parallelizes the per-section work but adds a merge step that can lose detail. The README presents both without declaring a winner, and the choice is exposed as a configuration value.
Incremental indexing: version hashes, metadata filters and a SQLite map
The most concrete engineering in this repository is the incremental index. Rebuilding a vector store on every document edit is the usual failure point as a corpus grows, and this project avoids it in three ways described in the README.
First, document-level metadata tracking. Every chunk is tagged with a source document ID and a version hash. When a document changes, only that document's chunks are regenerated: the old ones are deleted by metadata filter and the new ones inserted.
Second, an incremental ingestion pipeline. The pipeline diffs source documents against what is already indexed using those version hashes, so only changed or new documents get processed.
Third, deletion handling. A separate mapping table in a SQLite database records doc_id to chunk_ids, so removals can target specific chunks without scanning the whole store. That is the piece most RAG tutorials skip, and it is the reason this project can claim incremental behaviour rather than just describing it.
The design has one sharp edge, and the README flags it with an important note: if you swap embedding models, you must rebuild the index from scratch because the vector spaces are not compatible. Version hashes track document changes, not model changes. Plan the embedding model before the first ingestion, or accept a full reindex later.
Getting it running: make targets, Docker and the llama.cpp server
The README routes almost everything through a Makefile. Run make check first to confirm that which pip3 and which python3 resolve to the intended paths. Then pick one of two setup targets. make setup_cuda creates the environment and installs dependencies with NVIDIA CUDA acceleration. make setup_metal does the same for Metal on macOS. The README states that both also start the llama.cpp server locally via Docker Compose.
For day-to-day use, make start brings up the backend and frontend, and the README notes it waits for the backend to be ready before launching the frontend. If you want the model server on its own, make start_llama_server_cuda or make start_llama_server_metal starts it through Docker Compose, and make stop_llama_server shuts it down. The server listens at http://0.0.0.0:8080 and serves the llama-ui interface.
Configuration is split across several concerns the README names as distinct steps: setting the open-source LLM model, setting the embedding model, and setting the response synthesis strategy (the Create And Refine versus Hierarchical Summarization choice). The README does not reproduce the exact key names or file paths for these values in the material available here, so check the environment section of the README and the notes directory before editing configuration.
One prerequisite is easy to miss because it is not Python: the UI needs Node 22.12+ and Yarn 1.22+. Installing the Python side alone will not give you a working interface.
Limitations: hallucinations, hardware lock-in and the reindex cliff
The README carries its own warning that the large language model sometimes generates hallucinations or false information. That is not a defect unique to this project, but it matters more here because the answers are meant to be grounded in your documents. Retrieval reduces hallucination; it does not remove it. If your corpus contains conflicting or outdated pages, the retriever will surface them and the model will synthesize across them.
Hardware is the second constraint. The tested configurations are an RTX 3060 machine and an M1 MacBook Pro. Anyone else is directed to llama.cpp's issue tracker, which is an honest way of saying the project does not guarantee your setup. Windows is not in the tested list.
The third constraint is the embedding model swap. Because version hashes track documents rather than the embedding space, changing models invalidates the entire index. On a small docs folder this is a rebuild you barely notice. On a large corpus it is the difference between a background job and an outage window.
Finally, the retrieval quality depends on chunking. The project refactored LangChain's RecursiveCharacterTextSplitter rather than writing a Markdown-aware splitter. For prose documentation this is usually fine. For Markdown with heavy code blocks, tables or deeply nested lists, chunk boundaries can split structures that should stay together, and the README does not describe any special handling for those cases.
Alternatives: LangChain plus a managed vector store
The obvious alternative is assembling the same pipeline with LangChain and a hosted vector database such as Pinecone or Weaviate. The difference is not features; it is where the work happens. LangChain gives you a large library of loaders, splitters, retrievers and chain abstractions, and a managed vector store removes the operational burden of running Chroma and SQLite yourself. In exchange you take on a framework whose abstractions change between releases and, if you use a hosted store, a network dependency and per-vector pricing.
This project makes the opposite bet. It refactors the one LangChain class it needs (RecursiveCharacterTextSplitter) and drops the dependency, runs Chroma locally, and keeps the language model on your own machine through llama.cpp. The result is a smaller surface area and no external calls, at the cost of doing your own indexing logic. The incremental ingestion design, with version hashes and a SQLite mapping table, is exactly the work a managed store would otherwise absorb.
If your corpus is small and static, that trade is poor. Rebuilding a few hundred chunks takes little time, and the incremental machinery adds code you will not exercise. If your corpus changes frequently or is large, the trade flips: the diffing pipeline is the reason to pick this project over a naive script.
Maintenance, licence and what to verify before adopting
The repository is Apache-2.0, which permits commercial use, modification and redistribution provided you retain the licence and notices and state significant changes. That is a permissive licence with an explicit patent grant. It is not legal advice; read the LICENSE file and the NOTICE requirements yourself if you plan to redistribute.
Maintenance cost concentrates in three places. The llama.cpp server runs in Docker and is pinned by your Compose file, so model or image updates are a deliberate action rather than something that happens underneath you. The embedding model choice is effectively permanent for a given index, per the README's rebuild note. And the Makefile is the entry point for every operation, so any change to environment layout has to be reflected in make setup_cuda, make setup_metal, make start and the llama server targets together.
Before adopting, verify three things. Confirm your hardware appears in the tested list or that llama.cpp supports it, because the README defers to that project's issue tracker otherwise. Confirm you can install Node 22.12+ and Yarn 1.22+ if you want the bundled UI. And read notes/todo.md, which the README links as the project's own list of next steps, so you know which parts of the pipeline the maintainers still consider unfinished.
Editorial conclusion
Adopt this if you want a local, Apache-2.0 RAG pipeline over Markdown with no LangChain dependency, and you are on Ubuntu with CUDA 12.4+ or Apple Silicon. Do not adopt it if you need a managed vector store, a Windows path, or a corpus whose embedding model might change later without a rebuild. Before committing, verify that your hardware matches the two tested configurations and decide your embedding model up front, because the README states that swapping it requires rebuilding the index from scratch.
Community notes