llm-wiki-compiler: A Knowledge Compiler That Turns Raw Sources Into a Cited Markdown Wiki
The knowledge compiler. Raw sources in, interlinked wiki out. Inspired by Karpathy's LLM Wiki pattern.
At a glance
- What is it?
- llmwiki compiles papers, notes, transcripts and web pages into typed, citation-traceable wiki pages that agents can query. The interesting part is not the generation step, it is the profile contract that gates every write.
- Who is it for?
- Adopt llmwiki if your source material is worth compiling once and reusing, and if you can accept a two-phase LLM pipeline plus a profile contract you have to author and validate. Skip it if you need ad-hoc search over fast-changing logs, or if you want a static-site generator, since the README explicitly rules those out.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 5 days ago.
- What is it written in?
- Mainly TypeScript, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The Problem llmwiki Compiles Away
The README frames the core problem in one sentence: instead of re-discovering knowledge from raw files at query time, compile it once into durable pages. That is the whole pitch. If you have a directory of papers, transcripts, READMEs or PDFs and you keep pointing an agent at it, every question re-reads the same material, re-derives the same conclusions, and produces an answer you cannot audit afterwards. llmwiki moves that cost to a build step and leaves behind markdown pages with citations attached.
The intended audience is narrow and stated plainly. The README lists four cases: compiling papers, notes, READMEs, transcripts, PDFs, images or web pages into typed wiki pages; giving agents a stable context pack; keeping generated knowledge auditable with review queues and freshness checks; and exchanging compiled knowledge through formats like OKF, JSON-LD, GraphML and llms.txt. It also names three things it is not: a general static-site generator, a heavy ontology database, or a replacement for ad-hoc search over fast-changing raw logs. That last exclusion matters. If your sources change hourly, compiling them is wasted work.
Two-Phase Extraction and the Typed Page Model
The pipeline is described as two-phase. An LLM first extracts concepts from raw sources, then generates typed pages. The default page types are concept, entity, comparison and overview. Citations are not decorative: paragraphs and claims cite source files and line ranges, and llmwiki lint validates those links afterwards. That means a broken citation is a lint failure, not a silent inaccuracy.
Retrieval is hybrid rather than pure vector search. The README describes semantic chunk search, BM25 reranking, and wikilink graph expansion feeding compact evidence packs for queries and agents. The wikilink expansion is the piece that distinguishes this from a chunk store: because pages are interlinked, a query can pull in neighbours that a similarity score alone would miss. The same graph is exposed for inspection. llmwiki view opens a read-only browser UI with search, page metadata, graph exploration, source-freshness badges and citation chips.
On top of page generation sits a review policy. Generated pages can be auto-held when confidence, contradiction, schema or provenance rules trip. This is the mechanism that keeps a compiled wiki from silently accumulating claims nobody checked.
Lifecycle Profiles as a Fail-Closed Contract
The 1.0 release introduced Configurable Lifecycle Profiles, and this is where the project stops being a generic summariser. A validated .llmwiki/profile.json declares typed entities, fields and directed relations; lifecycle states, transition evidence and trust gates; multi-stage workflows and declared actions; hash-pinned artifacts and first-party connector bindings; and content tiers with retrieval behaviour.
The claim worth scrutinising is that these rules are enforced by the runtime rather than left as prompt conventions. The README states that relation, evidence, artifact and human/agent gates are enforced by the write path, and that invalid profiles and writes which bypass a declared gate fail closed. Standing lint then detects drift after the fact. If that holds, it is a meaningful difference from prompt-level instructions, which an LLM can ignore. The trade-off is rigidity: a fail-closed write path means a profile that does not match your domain will reject writes rather than let them through with a warning.
Backward compatibility is handled by absence. A project without .llmwiki/profile.json falls back to the built-in default concepts-and-queries profile and preserves pre-1.0 behaviour. So the profile is opt-in, and the compiler itself is not supposed to grow domain branches. Domain specificity lives in configuration.
Getting It Running: Profiles, Templates and Workflows
The README gives three starting paths. You can author a profile one entity type at a time:
llmwiki profile init research --entity paper
Or install a built-in or local declarative template, inspecting it first:
llmwiki template list llmwiki template inspect autosci llmwiki template init autosci
Then validate and inspect what you have:
llmwiki profile validate llmwiki workflow list
The autosci template is described as a research system with papers, ideas, experiments, manuscripts, evidence artifacts, workflows and Crossref ingestion. newsroom applies the same machinery to articles, desks, bylines and editorial workflows. The README states that templates contain configuration and examples, never executable plugin code, and that publishers can build signed, offline distributions with llmwiki template publish using Ed25519 signatures.
Beyond setup, the operational commands named in the material are llmwiki lint, llmwiki next, llmwiki refresh --stale, llmwiki eval, llmwiki view and llmwiki serve. The refresh flag is scoped deliberately: it repairs changed knowledge without compiling unrelated new sources. The eval harness reports a health score, a per-page health distribution that flags the worst pages, wikilink-graph health, citation coverage and precision, corpus stats and regression deltas, with optional judge-model citation support. llmwiki serve exposes ingest, compile, query, lint, read, status, eval, context-pack and OKF exchange tools over MCP. For TypeScript integration, createWiki({ root }) drives ingest, compile, query, context, status, export, eval and OKF import/export without shelling out.
Where the Design Bites Back
The clearest limitation is stated by the project itself: do not use it as a replacement for ad-hoc search over fast-changing raw logs. Compilation has a cost, and that cost is only repaid when the same knowledge is queried repeatedly. A log stream that changes every few minutes will spend most of its time stale and most of its budget recompiling.
The second constraint is the profile contract. Fail-closed enforcement is a safety property, but it is also a failure mode. A profile that misdeclares an entity type or a transition requirement will block writes, and the README does not describe a dry-run or permissive mode for iterating on a profile against real sources. The commands it does give you are llmwiki profile validate and llmwiki workflow list, which tell you whether the profile is well-formed, not whether it fits your material.
Third, external knowledge entering through OKF is staged through the review queue by default, and the README notes that trusted bundles can be written live explicitly. That default is sensible, but it means an import of any size becomes a review workload. The README does not quantify how much of that queue is typically auto-resolved.
Finally, the project is TypeScript and distributed via npm, with a CI workflow on main. Nothing in the supplied material describes deployment topology, storage backend or how the compiled wiki scales past a single project root.
Compared With Plain RAG Over a Document Folder
The obvious alternative is a retrieval-augmented generation setup: chunk your files, embed them, retrieve at query time, and let the model synthesise an answer. That approach has real advantages here. There is no compile step to keep current, no profile to author, and no review queue. Adding a document is a re-index, not a rebuild.
The difference in approach is where the structure lives. In a RAG pipeline, structure is implicit in embeddings and reconstructed per query. In llmwiki, structure is materialised: typed pages, directed relations, wikilinks, lifecycle states and citations with line ranges. The README's own framing is that compiled pages accumulate structure, provenance, review state and retrieval metadata over time. That is a durable artifact you can lint, export to GraphML or JSON-LD, and hand to another tool. A chunk index is not.
llmwiki also borrows from RAG rather than rejecting it. Semantic chunk search and BM25 reranking are both present, with wikilink graph expansion layered on top. So the honest comparison is not retrieval versus no retrieval. It is whether you want the intermediate representation to persist and be reviewable, or whether per-query synthesis is good enough. If your answers are consumed once and discarded, the compiled wiki is overhead.
Maintenance Cost, Licence and What to Verify
The licence is MIT, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is the standard permissive position and it places no copyleft obligation on your own code. This is a description of the licence identifier, not legal advice; if you are redistributing a modified compiler or embedding it in a product, have counsel read the LICENSE file rather than this paragraph.
Maintenance has three distinct costs. The first is provider configuration: the README lists Anthropic, Claude Agent SDK local login, OpenAI Codex CLI local login, OpenAI-compatible servers, Ollama, GitHub Copilot, Atlas Cloud, OrcaRouter and local OpenAI-compatible runtimes. That breadth is useful but it means model choice is a live variable affecting output quality and cost. The second is the profile itself, which is a schema you own and must evolve as your domain changes; invalid profiles fail closed, so profile edits are production changes. The third is the review queue, since generated pages can be auto-held and external OKF imports are staged by default.
The release cadence visible in the material is three versions across roughly two months (1.0.0 in July, 1.1.0 in mid-July, 1.2.0 in September), which suggests active development but also that APIs may still move. Before adopting, verify three things concretely: that llmwiki profile validate accepts a profile modelling your actual entity types, that llmwiki eval's citation coverage and precision numbers are acceptable on a sample of your own sources, and that the refresh --stale path repairs the pages you expect when a source file changes.
Editorial conclusion
Adopt llmwiki if your source material is worth compiling once and reusing, and if you can accept a two-phase LLM pipeline plus a profile contract you have to author and validate. Skip it if you need ad-hoc search over fast-changing logs, or if you want a static-site generator, since the README explicitly rules those out. Before committing, run llmwiki profile validate against your own .llmwiki/profile.json and check whether the built-in autosci or newsroom template actually matches your entity types, because a profile that does not fit will fail closed rather than degrade quietly.
Community notes