llm-wiki-compiler: A Knowledge Compiler That Turns Raw Sources Into a Cited Markdown Wiki
The knowledge compiler. Raw sources in, interlinked wiki out. Inspired by Karpathy's LLM Wiki pattern.
At a glance
- What is it?
- llmwiki compiles papers, notes, transcripts and web pages into typed, citation-traceable wiki pages that agents can query. The interesting part is not the generation step, it is the profile contract that gates every write.
- Who is it for?
- Adopt llmwiki if your source material is worth compiling once and reusing, and if you can accept a two-phase LLM pipeline plus a profile contract you have to author and validate. Skip it if you need ad-hoc search over fast-changing logs, or if you want a static-site generator, since the README explicitly rules those out.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly TypeScript, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- atomicstrata/llm-wiki-compiler2,260 ★ · +250
| Repository | 2026-09-10 | 2026-11-15 |
|---|---|---|
| atomicstrata/llm-wiki-compiler | 2k | 2.3k |
250 Stars in the last 66 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
The Problem llmwiki Compiles Away
The README frames the core problem in one sentence: instead of re-discovering knowledge from raw files at query time, compile it once into durable pages. That is the whole pitch. If you have a directory of papers, transcripts, READMEs or PDFs and you keep pointing an agent at it, every question re-reads the same material, re-derives the same conclusions, and produces an answer you cannot audit afterwards. llmwiki moves that cost to a build step and leaves behind markdown pages with citations attached.
The intended audience is narrow and stated plainly. The README lists four cases: compiling papers, notes, READMEs, transcripts, PDFs, images or web pages into typed wiki pages; giving agents a stable context pack; keeping generated knowledge auditable with review queues and freshness checks; and exchanging compiled knowledge through formats like OKF, JSON-LD, GraphML and llms.txt. It also names three things it is not: a general static-site generator, a heavy ontology database, or a replacement for ad-hoc search over fast-changing raw logs. That last exclusion matters. If your sources change hourly, compiling them is wasted work.
Two-Phase Extraction and the Typed Page Model
The pipeline is described as two-phase. An LLM first extracts concepts from raw sources, then generates typed pages. The default page types are concept, entity, comparison and overview. Citations are not decorative: paragraphs and claims cite source files and line ranges, and llmwiki lint validates those links afterwards. That means a broken citation is a lint failure, not a silent inaccuracy.
Retrieval is hybrid rather than pure vector search. The README describes semantic chunk search, BM25 reranking, and wikilink graph expansion feeding compact evidence packs for queries and agents. The wikilink expansion is the piece that distinguishes this from a chunk store: because pages are interlinked, a query can pull in neighbours that a similarity score alone would miss. The same graph is exposed for inspection. llmwiki view opens a read-only browser UI with search, page metadata, graph exploration, source-freshness badges and citation chips.
On top of page generation sits a review policy. Generated pages can be auto-held when confidence, contradiction, schema or provenance rules trip. This is the mechanism that keeps a compiled wiki from silently accumulating claims nobody checked.
Lifecycle Profiles as a Fail-Closed Contract
The 1.0 release introduced Configurable Lifecycle Profiles, and this is where the project stops being a generic summariser. A validated .llmwiki/profile.json declares typed entities, fields and directed relations; lifecycle states, transition evidence and trust gates; multi-stage workflows and declared actions; hash-pinned artifacts and first-party connector bindings; and content tiers with retrieval behaviour.
The claim worth scrutinising is that these rules are enforced by the runtime rather than left as prompt conventions. The README states that relation, evidence, artifact and human/agent gates are enforced by the write path, and that invalid profiles and writes which bypass a declared gate fail closed. Standing lint then detects drift after the fact. If that holds, it is a meaningful difference from prompt-level instructions, which an LLM can ignore. The trade-off is rigidity: a fail-closed write path means a profile that does not match your domain will reject writes rather than let them through with a warning.
Backward compatibility is handled by absence. A project without .llmwiki/profile.json falls back to the built-in default concepts-and-queries profile and preserves pre-1.0 behaviour. So the profile is opt-in, and the compiler itself is not supposed to grow domain branches. Domain specificity lives in configuration.
Getting It Running: Profiles, Templates and Workflows
The README gives three starting paths. You can author a profile one entity type at a time:
llmwiki profile init research --entity paper
Or install a built-in or local declarative template, inspecting it first:
llmwiki template list llmwiki template inspect autosci llmwiki template init autosci
Then validate and inspect what you have:
llmwiki profile validate llmwiki workflow list
The autosci template is described as a research system with papers, ideas, experiments, manuscripts, evidence artifacts, workflows and Crossref ingestion. newsroom applies the same machinery to articles, desks, bylines and editorial workflows. The README states that templates contain configuration and examples, never executable plugin code, and that publishers can build signed, offline distributions with llmwiki template publish using Ed25519 signatures.
Beyond setup, the operational commands named in the README are llmwiki lint, llmwiki next, llmwiki refresh --stale, llmwiki eval, llmwiki view and llmwiki serve. The refresh flag is scoped deliberately: it repairs changed knowledge without compiling unrelated new sources. The eval harness reports a health score, a per-page health distribution that flags the worst pages, wikilink-graph health, citation coverage and precision, corpus stats and regression deltas, with optional judge-model citation support. llmwiki serve exposes ingest, compile, query, lint, read, status, eval, context-pack and OKF exchange tools over MCP. For TypeScript integration, createWiki({ root }) drives ingest, compile, query, context, status, export, eval and OKF import/export without shelling out.
Where the Design Bites Back
The clearest limitation is stated by the project itself: do not use it as a replacement for ad-hoc search over fast-changing raw logs. Compilation has a cost, and that cost is only repaid when the same knowledge is queried repeatedly. A log stream that changes every few minutes will spend most of its time stale and most of its budget recompiling.
The second constraint is the profile contract. Fail-closed enforcement is a safety property, but it is also a failure mode. A profile that misdeclares an entity type or a transition requirement will block writes, and the README does not describe a dry-run or permissive mode for iterating on a profile against real sources. The commands it does give you are llmwiki profile validate and llmwiki workflow list, which tell you whether the profile is well-formed, not whether it fits your material.
Third, external knowledge entering through OKF is staged through the review queue by default, and the README notes that trusted bundles can be written live explicitly. That default is sensible, but it means an import of any size becomes a review workload. The README does not quantify how much of that queue is typically auto-resolved.
Finally, the project is TypeScript and distributed via npm, with a CI workflow on main.
Compared With Plain RAG Over a Document Folder
The obvious alternative is a retrieval-augmented generation setup: chunk your files, embed them, retrieve at query time, and let the model synthesise an answer. That approach has real advantages here. There is no compile step to keep current, no profile to author, and no review queue. Adding a document is a re-index, not a rebuild.
The difference in approach is where the structure lives. In a RAG pipeline, structure is implicit in embeddings and reconstructed per query. In llmwiki, structure is materialised: typed pages, directed relations, wikilinks, lifecycle states and citations with line ranges. The README's own framing is that compiled pages accumulate structure, provenance, review state and retrieval metadata over time. That is a durable artifact you can lint, export to GraphML or JSON-LD, and hand to another tool. A chunk index is not.
llmwiki also borrows from RAG rather than rejecting it. Semantic chunk search and BM25 reranking are both present, with wikilink graph expansion layered on top. So the honest comparison is not retrieval versus no retrieval. It is whether you want the intermediate representation to persist and be reviewable, or whether per-query synthesis is good enough. If your answers are consumed once and discarded, the compiled wiki is overhead.
Maintenance Cost, Licence and What to Verify
The licence is MIT, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is the standard permissive position and it places no copyleft obligation on your own code. This is a description of the licence identifier, not legal advice; if you are redistributing a modified compiler or embedding it in a product, have counsel read the LICENSE file rather than this paragraph.
Maintenance has three distinct costs. The first is provider configuration: the README lists Anthropic, Claude Agent SDK local login, OpenAI Codex CLI local login, OpenAI-compatible servers, Ollama, GitHub Copilot, Atlas Cloud, OrcaRouter and local OpenAI-compatible runtimes. That breadth is useful but it means model choice is a live variable affecting output quality and cost. The second is the profile itself, which is a schema you own and must evolve as your domain changes; invalid profiles fail closed, so profile edits are production changes. The third is the review queue, since generated pages can be auto-held and external OKF imports are staged by default.
The release cadence is three versions across roughly two months (1.0.0 in July, 1.1.0 in mid-July, 1.2.0 in September), which suggests active development but also that APIs may still move. Before adopting, verify three things concretely: that llmwiki profile validate accepts a profile modelling your actual entity types, that llmwiki eval's citation coverage and precision numbers are acceptable on a sample of your own sources, and that the refresh --stale path repairs the pages you expect when a source file changes.
Editorial conclusion
Adopt llmwiki if your source material is worth compiling once and reusing, and if you can accept a two-phase LLM pipeline plus a profile contract you have to author and validate. Skip it if you need ad-hoc search over fast-changing logs, or if you want a static-site generator, since the README explicitly rules those out. Before committing, run llmwiki profile validate against your own .llmwiki/profile.json and check whether the built-in autosci or newsroom template actually matches your entity types, because a profile that does not fit will fail closed rather than degrade quietly.
Frequently asked questions
What is the LLM wiki and what does it do?
It is a pattern rather than a product: instead of re-discovering knowledge from raw files at query time, a compiler builds durable pages once, and those pages accumulate structure, provenance, review state and retrieval metadata over time. This repository implements that pattern as a TypeScript CLI, an MCP server and an SDK that compile sources into an interlinked, citation-traceable markdown wiki.
Is LLM wiki a RAG?
The page does not call it that, and it describes the opposite approach: compiling once into durable pages rather than re-discovering knowledge from raw files at query time. Retrieval still exists inside the system, as semantic chunk search reranked with BM25 and expanded through the wikilink graph, but it runs against compiled pages rather than raw sources.
Where does llm-wiki-compiler enforce its quality rules?
On the write path rather than in the prompt. Relation, evidence, artefact and human or agent approval gates are checked when a page is written, the profile file is described as fail-closed, and a standing lint pass detects drift afterwards. Generated pages can be auto-held when confidence, contradiction, schema or provenance rules trip.
What do the citations in an llmwiki page point at?
Source files and line ranges. Paragraphs and claims carry them, a lint command validates the links, and the local viewer shows citation chips alongside source freshness badges. Since the 1.4.0 release you can also follow an answer's citations back to the wiki pages that were used to answer.
When should you not use llmwiki?
The page rules out three uses by name: as a general static site generator, as a heavy ontology database, and as a replacement for ad-hoc search over fast-changing raw logs. Its own stated strength is conditional on source knowledge being worth compiling, reviewing and reusing.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/atomicstrata-llm-wiki-compiler)Star history chart for your README
[](https://hysenlabs.com/projects/atomicstrata-llm-wiki-compiler)Was this useful?