Knowhere: a dual-track document memory layer for agent retrieval
Knowhere extracts, parses, and outputs structured chunks ready for AI Agents and RAG.
At a glance
- What is it?
- Knowhere parses messy PDFs, slide decks and text-native files into one hierarchy-native chunk schema, then hands agents a corpus tool contract instead of a fixed navigation pipeline. The interesting decision is that it stops treating perfect extraction as a prerequisite for retrieval.
- Who is it for?
- Adopt Knowhere if your corpus is mostly PDFs and slide decks, you need citations that resolve back to pages and assets, and you are willing to run the self-hosted stack rather than call the managed API. Do not adopt it if you only ingest clean Markdown or HTML, where the dual-track machinery buys you nothing, or if you need a retrieval API that returns ranked chunks without an agent in the loop.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem is dirty documents, not missing embeddings
Most RAG stacks fail on the ingestion side. A pipeline that extracts every element before a model sees the document accumulates errors: reading order drifts, tables split, hidden text layers leak into context. Knowhere's README states the position directly, that traditional OCR and document intelligence pipelines try to extract every element before a model can understand the document, and that mistakes in reading order, layout, tables or hidden text layers can accumulate into unreliable model context. The project's answer is to refuse that precondition. Extraction quality becomes a routing decision rather than a gate. The audience is engineers building agentic RAG over document collections that include scanned reports, technical atlases, drawing sets and PowerPoint decks, plus anyone who wants retrieval to stay connected to source pages and assets instead of returning orphaned text. The README also names local and offline document collections as a target, which is why a self-hosted repository exists alongside the managed API.
Vision and Text tracks converge on one chunk contract
Knowhere 2.0 runs two parsing paths. The Text Track preserves native text and structure where the source is text-native and reliable. The Vision Track sends a page or slide to a frontier vision model and lets it understand the page as a whole. According to the README, PDF and .pptx uploads through the V2 Jobs API use the Vision Track, while other supported formats use the Text Track. That routing rule is format-based, not content-based, so a clean digital PDF still goes through the vision path if it arrives via that API generation. The two tracks are not separate products: the README states both produce the same chunk and metadata schema, described as a hierarchy-native schema, so downstream storage, graph construction and retrieval stay format-independent. The practical consequence is that a text section and a vision-understood page become compatible hierarchy nodes, each carrying source evidence, linked assets and cross-document relationships. The trade-off is visible in the naming. The Text Track is described as preserving precise extracted structure, while the Vision Track is described as understanding pages holistically. Precision and recall are being traded against each other per format, and the schema is what keeps that trade invisible to the agent.
Retrieval 2.0 hands the navigation decision to the agent
The September 8, 2026 release note describes MapNav evolving from a fixed navigation workflow into what the project calls a corpus-native foundation for agents. The mechanism, as the README lays it out, is a substrate plus a tool contract. Knowhere exposes one corpus schema and tools for document outlines, structural filters, exact search, fuzzy recall, full reading, assets and cross-document relationships. The agent then decides how to search, traverse, read and cite. This is a real architectural choice with a cost. A fixed pipeline can be tuned and benchmarked as a single unit; a tool contract pushes that work into the agent's planning loop, which means retrieval quality now depends on the calling model's ability to use the tools well. The README claims the same foundation powers built-in retrieval and external agents through MCP, so the substrate is meant to be model-neutral. Whether that holds in practice is not something the supplied material demonstrates.
Getting it running: Cloud first, self-hosted second
The README offers two entry points. The managed route is Knowhere Cloud at knowhereto.ai, which the project advertises with five dollars in free credits on registration, positioned explicitly as a way to avoid self-deployment complexity. The self-hosted route points to the Ontos-AI/knowhere-self-hosted repository, with a companion dashboard at Ontos-AI/knowhere-dashboard and container images published to ghcr.io/ontos-ai/knowhere. The stated runtime floor is Python 3.11 or newer. What the README does not give is a pip install line, a docker run command, or the config keys for the self-hosted stack. Those live in the self-hosted repository, which is not part of the supplied material, so treat any deployment plan as requiring a separate read of that repo. The one concrete API detail available is the routing rule: PDF and .pptx uploads through the V2 Jobs API take the Vision Track, everything else takes the Text Track. If your pipeline depends on controlling which track handles a given file, that API generation boundary is the lever you have.
Long documents and atlases are a stated, narrow strength
The June 1, 2026 release note states that the parsing pipeline handles long-form PDFs with hundreds of pages, naming 300 and 500 as examples, and routes technical atlases or drawing collections through a dedicated layout-aware parser. That is a specific claim about a specific document class. It matters because long PDFs and drawing sets are exactly where element-by-element extraction tends to fall apart, and where a page-level vision pass plus hierarchy reconstruction is a plausible alternative. It is also the kind of claim that needs verification on your own files. Page counts are easy to state and hard to satisfy once tables span pages, drawings carry dense annotation layers, or a scanned page has no text layer at all. Nothing in the supplied material reports accuracy, throughput or cost per page for these cases.
Where Knowhere is the wrong tool
Three cases stand out. First, text-native corpora. If you ingest Markdown, HTML or clean office documents, the dual-track design adds a routing layer and a vision path you will not use, and the hierarchy schema may be more structure than your retrieval needs. Second, pipelines that want ranked chunks back from a single call. Knowhere's Retrieval 2.0 model assumes an agent that plans, traverses and cites; if your application just needs top-k similarity search, a vector database plus your own chunker is a shorter path. Third, teams that cannot run or pay for a vision model. The Vision Track depends on frontier vision models by definition, and the README does not describe a local fallback for that path, so the offline document collections the project targets may still require a hosted model for PDFs and slide decks. The README is also thin on failure modes: it does not discuss what happens when a vision model misreads a page, whether summaries and entities are stored alongside source text for verification, or how errors propagate into citations.
Compared with a plain chunker plus vector store
The default alternative is a text splitter feeding a vector database, with LangChain or a similar framework wiring the two together. The difference is not the embedding step, which both approaches share. It is what gets stored. A splitter produces chunks with offsets; the retrieval unit is a chunk. Knowhere produces hierarchy nodes with source pages, linked assets and cross-document relationships, and the retrieval unit is a position in a navigable tree that an agent can traverse. That richer substrate is what makes structural filters, outline traversal and resolvable citations possible. It is also more to build, more to store and more to keep consistent when documents are re-ingested. Knowhere lists langchain and vector-database among its topics, which suggests it is meant to sit alongside those tools rather than replace the storage layer, but the README does not spell out the integration contract.
Licence, release cadence and what to verify
Knowhere is Apache-2.0, which permits commercial use and modification with the usual notice and patent terms; that is a factual statement about the licence identifier, not legal advice, and you should read the LICENSE file and your own obligations. The release history shows v1.2.4 on September 10, 2026, v1.2.3 the day before, and v1.2.2 on September 8, so the project is shipping patch releases on a daily cadence during this window. Frequent patches are cheap to apply but they also mean the schema and API surface may still be moving, and the README describes a 2.0 architecture that landed in the same month as these tags. Before adopting, verify the V2 Jobs API routing for your actual file types, inspect the chunk and metadata schema for one document you own, and read the knowhere-self-hosted repository for the deployment requirements the README omits. If those three checks pass, the dual-track design is doing real work for you. If they do not, a splitter and a vector store will get you to a working pipeline with fewer moving parts.
Editorial conclusion
Adopt Knowhere if your corpus is mostly PDFs and slide decks, you need citations that resolve back to pages and assets, and you are willing to run the self-hosted stack rather than call the managed API. Do not adopt it if you only ingest clean Markdown or HTML, where the dual-track machinery buys you nothing, or if you need a retrieval API that returns ranked chunks without an agent in the loop. Before committing, verify three things in your own environment: whether your document formats route to the Vision or the Text track under the V2 Jobs API, what the hierarchy schema looks like for a document type you actually own, and what the self-hosted deployment requires beyond the Python 3.11 floor the README states.
Community notes