haiku.rag: agentic RAG on an embedded LanceDB, with Docling parsing and an MCP server

The problem haiku.rag targets, and who it is actually for

Most RAG tutorials assume you can stand up a vector service and keep it running. haiku.rag takes the opposite position: the database is embedded LanceDB, so a single Python process owns the index, and the README states plainly that it runs locally with no server required. The stated output is answers about your own documents with citations to page numbers and section headings, which is a narrower promise than general-purpose chat over a corpus. The citation target matters. A page number and a heading are things a reader can go and check, and the project also offers visual grounding, described as viewing chunks highlighted on original page images. That combination points at a specific user: someone doing technical or regulatory document work where an unsourced answer is worthless. It is not aimed at people who want a hosted API with an SLA, and it is not aimed at teams who want to avoid running any model endpoint at all. The README's Quick Start carries an explicit note that an embedding provider such as Ollama or OpenAI is required, so the local-first claim is about the database and the orchestration, not about the models.

Hybrid retrieval, RRF and the multimodal embedding space

The retrieval stack is the part worth reading closely. Search is described as vector plus full-text, combined with Reciprocal Rank Fusion. RRF is a rank-fusion method: you take the ordered result list from each retriever and merge by rank rather than by raw score, which avoids the problem of comparing a cosine distance against a BM25 score on incompatible scales. That is a sensible default and it is why the README can offer one `search` command instead of making you choose a mode. On top of that sits reranking, with local cross-encoders, Cohere, Zero Entropy, or vLLM listed as options. A cross-encoder rescores a small candidate set by reading query and chunk together, which is slower per candidate but usually sharper than the first-stage retrieval. The multimodal claim is the more unusual one. Multimodal embedders (vLLM, VoyageAI, Cohere) are said to place picture vectors in the same space as text, so a text query can return figure hits, and an image can be used as a query. That is a property of the embedder, not of haiku.rag, and the README is clear that multimodal support is switched on with `multimodal: true` on vLLM, VoyageAI or Cohere. If your embedder does not support it, cross-modal search is not available to you.

What DoclingDocument storage buys you, and what it costs

haiku.rag stores the full DoclingDocument rather than only flattened text. The README frames the benefit as structure-aware context expansion, meaning the retriever can pull in neighbouring content using the document's own hierarchy instead of guessing from character offsets. That is a real architectural choice with a real cost: you are persisting a richer representation, and the fidelity of everything downstream depends on Docling's parse. If Docling misreads a table or drops a heading, the citation that haiku.rag produces will point at the wrong place, and no amount of reranking fixes it. The README also lists an Inspector, a TUI for browsing documents, chunks, and search results. That tool exists precisely because ingestion is where these pipelines fail, and being able to look at the chunks before you trust the answers is the correct workflow. I would treat parsing quality, not retrieval quality, as the first thing to evaluate on your own corpus.

Getting it running: packages, commands and config keys

Python 3.12 or newer is required. Two distributions exist. `pip install haiku.rag` is described as the full package including document processing, all embedding providers, and rerankers. `pip install haiku.rag-slim` installs minimal dependencies and you add only the extras you need, with the Installation page in the docs listing the available options. The CLI surface is small and readable: `haiku-rag add-src paper.pdf` to index, `haiku-rag search "attention mechanism"` to retrieve, `haiku-rag ask "What datasets were used for evaluation?"` for cited answers, `haiku-rag analyze "How many documents mention transformers?"` for tasks routed through sandboxed Python execution, `haiku-rag chat` for a multi-turn TUI, and `haiku-rag mcp --stdio` to expose the whole thing as tools. Vision QA is reachable from the same command with `haiku-rag ask "..." --image figure.png`. The Python API is async and context-managed: `async with HaikuRAG("knowledge.lancedb", create=True) as rag`, then `await rag.create_document_from_source("paper.pdf")` or a URL, `await rag.search("self-attention")` returning objects with `score`, `page_numbers` and `content`, and `await rag.ask(...)` returning an answer plus citation objects exposing `chunk_id`, `page_numbers` and `content`. The one config key the README names explicitly is `multimodal: true` on vLLM, VoyageAI or Cohere embedders. Everything else about provider configuration lives in the Configuration docs page, which this material does not reproduce, so I cannot list the remaining keys.

The ingester service is a separate deployment decision

Buried near the end of the feature list is the most operationally significant item: a long-lived `haiku-ingester` service with a persistent SQLite queue, an async worker pool with retries and a dead-letter queue, FS, HTTP, S3 and WebDAV source adapters, a FastAPI control plane, and a browser dashboard for operators. The command is `haiku-ingester serve`. This changes the shape of the project. The library path is a script you run when you want to index something. The ingester path is a daemon you supervise, with a queue that survives restarts and a dead-letter queue you have to actually look at. Those are two different operating models, and the README's Docker example is built around the second one, pairing continuous ingestion with the MCP server. If you only need to index a folder occasionally, running a daemon with a SQLite queue and a FastAPI control plane is more machinery than the job requires. If you have documents arriving continuously from a WebDAV share or an S3 bucket, the retry and dead-letter behaviour is the reason to use it, and you should read docs/ingester.md before deciding.

Where haiku.rag is the wrong tool

The clearest failure mode is provider dependency. haiku.rag does not ship models. Embeddings come from Ollama, OpenAI, VoyageAI, Cohere, LM Studio or vLLM, and QA uses any model supported by Pydantic AI. If your environment forbids outbound calls and you cannot run Ollama or vLLM locally, the project has nothing to offer you. A second boundary is scale and concurrency. Embedded LanceDB means the index lives with the process, and the README does mention S3, GCS, Azure and LanceDB Cloud as storage options, but the default posture is single-application access, not a shared multi-tenant retrieval service. Teams that need many independent writers hitting one index, or per-request isolation, should look elsewhere. A third issue is the release cadence. The supplied release history shows 0.84.0, 0.83.0 and 0.82.1 within roughly a week in September 2026, and the version is still below 1.0. Frequent minor releases below 1.0 mean the Python API surface can move, so pinning a version is the reasonable default rather than tracking main. Finally, the optional capabilities are worth understanding before you enable them. Evidence compaction replaces earlier questions' search results on the request with the evidence they cited, which is a token-saving measure with a correctness implication: if the cited evidence was incomplete, compaction propagates that gap. The citation policy capability requires every answer to declare what grounds it, including declaring that nothing does, which is a hallucination-prevention measure that will also make your model refuse more often.

Alternatives and the actual difference in approach

The obvious comparison is LlamaIndex, which also does document ingestion, chunking, retrieval and question answering in Python. The difference is where the state lives. LlamaIndex is a framework whose storage and retrieval are assembled from pluggable components, and you typically wire it to an external vector store such as a hosted service or a separately run database. haiku.rag commits to one embedded store, LanceDB, and builds the rest of the pipeline around that commitment, which is why `HaikuRAG("knowledge.lancedb", create=True)` is the whole setup. A second comparison is a plain search stack: BM25 over extracted text plus a reranker, no embeddings at all. That gets you keyword and phrase matching with zero model dependencies and no vector index to maintain, and for corpora with stable terminology it is often enough. What it cannot do is match a paraphrase, which is exactly what the vector half of haiku.rag's hybrid search is for. The choice between them is a choice about whether your users' queries use the same words as your documents. If they do, the simpler stack wins on operational cost. If they do not, hybrid retrieval with RRF is the more defensible design, and reranking on top is what keeps the fused result list honest.

Licence, maintenance and what to check before you commit

haiku.rag is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. MIT says nothing about the models you connect it to, and that is where the licence question actually bites: Ollama models, OpenAI, VoyageAI and Cohere each carry their own terms, and a self-hosted vLLM endpoint inherits the licence of whatever weights you serve. The MIT grant covers the library, not your inference stack. On maintenance, the repository is not archived, the last push shown is 2026-09-10, and releases are frequent. Frequent releases below 1.0 are a signal to pin. The upgrade cost is not in the dependency graph so much as in the stored index: the README offers `haiku-rag tag` to name database states and roll back to them, which exists because re-indexing is expensive and schema or embedder changes force it. If you change embedding provider or model, expect to rebuild the LanceDB table rather than migrate it, and note that switching the `multimodal: true` setting changes what vectors you are writing. Verify that before you index a large corpus, not after.

Editorial conclusion

Adopt haiku.rag if you want retrieval over a document set you control, with citations you can check against page numbers, and you are willing to run an embedding provider such as Ollama or point at OpenAI, VoyageAI, Cohere, LM Studio or vLLM. Skip it if you need a managed hosted retrieval API, or if you cannot operate a provider endpoint, since the library ships no model weights and the README's Quick Start explicitly notes that an embedding provider is required. Before committing, verify two things yourself: that your chosen embedder's output dimension and any multimodal setting match the LanceDB table you create, and that `haiku-rag add-src` on one representative PDF yields the page numbers and section headings you expect in `haiku-rag ask` citations, because citation quality depends on what Docling extracted, not on the retrieval layer.

Official sources

1. ggozad/haiku.rag on GitHub
2. License: MIT
3. Project website
4. README
5. Releases