wdoc: a scriptable RAG pipeline for heterogeneous document corpora
Summarize and query from a lot of heterogeneous documents. Any LLM provider, any filetype, advanced RAG, advanced summaries, scriptable, etc
At a glance
- What is it?
- wdoc is a Python RAG system from thiswillbeyourgithub that loads PDFs, EPUBs, audio, Anki decks and more into one queryable corpus. Its selling point is sourced answers with indented markdown citations, and its cost is a dependency tree you have to assemble yourself.
- Who is it for?
- Adopt wdoc if you already work from the command line, your corpus mixes filetypes, and you want answers that point back to specific passages rather than a chat window that forgets its sources. Do not adopt it if you need a stable supported product with a vendor behind it, or if a single-format pipeline such as a PDF-only extractor already covers your inputs.
- Can I use it commercially?
- Yes, with strict conditions. AGPL-3.0 is a network copyleft licence: if people use a modified version over a network, for example as a hosted service, you must offer them its source code under the same licence.
- Is it still maintained?
- Yes. The repository last received commits 22 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem wdoc was built to solve
The README is explicit about its origin. wdoc was created by a psychiatry resident who wanted a definitive answer pulled from several sources at once: audio recordings, video lectures, Anki flashcards, PDFs, EPUBs. That is a different problem from the one most RAG tutorials address. A single-format pipeline assumes your documents look alike, so the same chunker and the same parser work everywhere. A psychiatry resident's material does not look alike. Lecture audio, a flashcard deck and a textbook chapter have nothing in common at the parsing layer.
So the project targets people whose information is spread across formats and who want one query to reach all of it. The README names researchers, students and professionals dealing with extensive information sources. The stated goal is not a chatbot. It is a sourced answer, where the output points to the exact portion of the source document, plus an opinionated summary feature the README describes as capturing the author's thought process rather than vague takeaways. If you only ever query one PDF, wdoc is heavier than the problem requires.
How the query pipeline is put together
The repository ships three workflow diagrams, one per task, and they name the stages. The query diagram runs from user inputs through Raphael the Rephraser, then a VectorStore, then Eve the Evaluator, then Anna the Answerer, then recursive combining to the final output. The summary diagram is shorter: user inputs, loading and chunking, Sam the Summarizer, concatenation into a wdocSummary. The search diagram stops earlier, at Raphael the Rephraser, VectorStore and Eve the Evaluator.
That split matters. Search returns retrieved material; query returns an answer built from it. The README describes the retrieval strategy as using both an expensive and a cheap LLM, because the system can afford to fetch a lot of documents per query through embeddings. The design intent is high recall first, then gradual aggregation: answers are combined in semantic batches until one answer remains, carrying source references. LangChain and LiteLLM are the backends, which is where provider independence comes from. LiteLLM is the layer that lets the same code path talk to hosted and local models. The README also mentions extra layers of security for sensitive material, without detailing them, so treat that claim as unverified until you read the reference files.
Getting it running, and the extras trap
The README gives a one-line answer for installation: when in doubt, use uvx wdoc[full]. The reason is stated plainly. The plain wdoc package ships only PDF and URL/web loaders; everything else (youtube, audio, anki, office formats, logseq) lives in optional extras. The [full] extra bundles all of them so you do not have to think about missing dependencies. The README says there are 15 or more filetypes implemented in total.
That single sentence is the most operationally important thing in the README, because the failure mode is quiet. Install plain wdoc, point it at a YouTube link, and the loader is simply not there. The README also points to a Docker deployment with a Gradio-based web interface for people who would rather not use the CLI, and describes that interface as experimental. A worked query example in the README starts by assigning a link variable to a PDF URL, which is the shape of the CLI: you hand it sources, then ask questions. For the complete list of CLI arguments, environment variables and filetypes, the repository points to the wdoc-skill directory, which holds SKILL.md, REFERENCE.md and EXAMPLES.md. Those files, not the README, are where the exact flag and env var names live. I have not installed wdoc, so I am not going to invent flag names here.
Where wdoc is the wrong tool
The README's own status section is the most honest part of the project. It reads: usable, tested, still under active development, tens of planned features. The author states directly that they would benefit from testing by users as the quickest way to find minor bugs, and asks people to open an issue before submitting a pull request because improvements may already be in the pipeline. That is a maintainer describing a project that works but has not been hardened by a wide user base.
The practical consequences are worth spelling out. The main branch is described as more stable than the dev branch, so branch choice is a real decision, not a formality. The web search feature is labelled preliminary and depends on the ddgs library for DuckDuckGo results, which means it inherits that dependency's stability. The documentation lives in three places: the readthedocs site, the wdoc-skill files, and the README itself. If your team needs a supported product with a release cadence you can point at, or if your corpus is uniform enough that a single-format extractor handles it, wdoc's breadth is cost without benefit. The AGPL-3.0 licence is also a constraint worth reading carefully: it is a strong copyleft licence, and if you plan to offer wdoc as part of a network service, the licence terms are something to review with your own counsel rather than assume.
How it differs from a general document QA stack
The obvious comparison is to a general-purpose document QA stack built on LangChain directly. The difference is not the backend, since wdoc uses LangChain too. It is the layer above. A hand-rolled LangChain pipeline gives you a retriever and a prompt, and you decide how many documents to fetch, how to combine partial answers, and whether to cite. wdoc ships those decisions already made: a rephrasing step, an evaluator step, an answerer step, and recursive combining, with the README stating the system deliberately fetches many documents per query because embeddings are cheap relative to generation. It also ships the loader layer for 15 or more filetypes, which is the part most teams underestimate when they start building.
The trade is control. A hand-built pipeline lets you swap the aggregation strategy for your domain; wdoc's aggregation is the product. The other comparison worth naming is a hosted document assistant, where you upload files and get answers in a browser. Those tend to be easier to start and harder to script, and they generally do not let you point the pipeline at a local model. wdoc's LiteLLM backend and CLI orientation are aimed at the opposite preference: you keep the corpus on your own machine and you drive it from a shell or from Python as a library.
Maintenance and upgrade cost
The release history in the repository shows 5.2.2 on 2026-08-24, 5.2.0 on 2026-07-03 and 5.1.3 on 2026-06-23. That is a steady cadence rather than a frozen one, and the last push timestamp matches the newest release, so the project is being worked on. For anyone pinning a version, the implication is that minor releases arrive often enough that you should decide deliberately whether to track them or pin.
The dependency surface is the real maintenance cost. wdoc sits on LangChain and LiteLLM, and the optional extras pull in loaders for audio, YouTube, Anki, office formats and logseq. Each of those is a third-party package with its own release cycle. The [full] extra is convenient precisely because it hides that surface, but hiding it does not remove it: a breaking change in an audio or office loader still reaches you through wdoc. The README's request for typo reports and bug reports is a signal about where the project is in its lifecycle, and it is worth reading as a statement about expected polish rather than as modesty. On licensing, AGPL-3.0 is a copyleft licence with network-use provisions; whether that fits your deployment is a question for your own legal review, not something this article can settle.
Who should pick it up, and what to check first
wdoc fits a specific profile: you work from a terminal or from Python, your material is genuinely mixed (a lecture recording next to a PDF next to a flashcard deck), and you care more about an answer that cites its source than about a polished chat interface. The README's framing of sourced answers and an opinionated summary is the whole pitch, and if that is what you need, the breadth of loaders is the reason to choose it over assembling the same thing yourself.
Before you commit, do two concrete checks. First, install with the extras you actually need rather than assuming plain wdoc covers your formats, since the README states plain wdoc ships only PDF and URL/web loaders. Second, run one real document of your most awkward type through the pipeline with your chosen provider. The README says the author needs user testing to find minor bugs, so the first document is where you find out whether your format and your provider combination is one of the smooth paths or one of the rough ones. If both checks pass, the recursive combining and citation behaviour is the part worth evaluating on your own corpus, because that is the claim the project is built around and the part no README can verify for you.
Editorial conclusion
Adopt wdoc if you already work from the command line, your corpus mixes filetypes, and you want answers that point back to specific passages rather than a chat window that forgets its sources. Do not adopt it if you need a stable supported product with a vendor behind it, or if a single-format pipeline such as a PDF-only extractor already covers your inputs. Before committing, verify two things yourself: that the extras you need install cleanly on your Python version, and that your chosen LLM provider works end to end on one real document, because the README states the author needs user testing to surface the minor bugs.
Community notes