sift-kg: a CLI pipeline that turns a folder of documents into a reviewable knowledge graph
Turn any collection of documents into a knowledge graph. Extract entities and relationships via LLM, deduplicate with your approval. Map domains, find hidden connections, spot patterns across documents — knowledge that persists and compounds, for you and your AI agents. All from the CLI.
At a glance
- What is it?
- sift-kg extracts entities and relations from PDFs, DOCX, HTML and other formats with an LLM, then asks you to approve entity merges before writing the graph. The design is opinionated about human review and silent about scale.
- Who is it for?
- Adopt sift-kg if you have a bounded document collection, an LLM API key or a local Ollama model, and you want a graph you can inspect and export rather than a hosted knowledge base. Skip it if you need incremental extraction from a live stream, if the corpus is large enough that per-document LLM calls and a full re-extract on growth are unacceptable, or if you cannot review merge decisions yourself.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 127 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem sift-kg addresses: relationships that survive the reading session
Reading a folder of PDFs produces notes, and notes do not connect to each other. The README frames the target as a graph that shows how everything connects, built without a database or infrastructure, from a CLI and your documents. The stated audience is broad: research and education, business intelligence, investigative work over FOIA releases and court filings, legal review, and genealogy. The README also pitches a second use case, an AI knowledge base, where the JSON graph becomes persistent structured memory for an agent. Both audiences share one requirement: the entities and the links between them must be extracted once and then persist, rather than being re-derived from raw text on every query. That is the gap sift-kg occupies. It is not a document search tool and not a note-taking app. It is an extraction pipeline whose output is a graph file you own.
The pipeline: Kreuzberg extraction, LLM schema discovery, NetworkX graph
The README shows a linear flow. Documents in 75+ formats (PDF, DOCX, XLSX, PPTX, HTML, EPUB, images) go through Kreuzberg for local text extraction, with optional OCR via Tesseract, EasyOCR, PaddleOCR, or Google Cloud Vision. Then schema discovery: one LLM call samples the corpus and designs entity and relation types, saved as discovered_domain.yaml for reuse and editing. Extraction then runs against that schema, or against a bundled domain such as general, osint or academic, or one you define in YAML. The graph itself is NetworkX serialized to JSON. Entity resolution comes next, and this is the part that shapes the tool: the LLM proposes merges and the human approves them in a terminal UI, after which sift apply-merges writes the decisions. Narrative generation and the browser viewer sit downstream of the resolved graph. Every extraction is claimed to link back to the source document and passage, which is what makes the review step meaningful: you are judging a proposed merge with the originating text available. The schema-free default is the interesting design choice. Rather than forcing your corpus into a fixed ontology, the tool asks the model what types exist in your material, then lets you edit the result. That trades reproducibility for fit, and the saved YAML is the mechanism that recovers reproducibility.
Getting it running: the command sequence and the config surface
Installation is a single pip command. The README gives the full sequence: pip install sift-kg, then sift init to create sift.yaml and .env.example, then sift extract ./documents/, sift build, sift resolve, sift review, sift apply-merges, sift narrate, sift view, and finally sift export graphml. The order matters because resolve and review sit between extraction and the graph you actually explore; skipping them leaves duplicate entities in place. Configuration lives in sift.yaml, and the README shows a domain key set to academic so the flag is not needed on every extract call. Provider selection is not spelled out in the material beyond the list of supported backends (OpenAI, Anthropic, Mistral, Ollama, or any LiteLLM-compatible provider), so the .env.example generated by sift init is where credentials go. Cost control is a flag, --max-cost, applied to extraction. For agent use the README lists four JSON-emitting commands: sift topology, sift query "topic", sift search "X" --json, and sift info --json. Viewer pre-filtering uses --neighborhood, --top, --community, --source-doc and --min-confidence. The bundled agent skill lives at .agents/skills/sift-kg/SKILL.md. Note what is absent: no release tags were retrieved, so there is no version number to pin against and no changelog to read for breaking changes between installs.
Entity resolution with a human in the loop is the real product decision
Most extraction tools either merge silently or leave duplicates for you to clean up later. sift-kg inserts an interactive approval step: sift resolve proposes, sift review presents the proposals in a terminal UI, and sift apply-merges commits only what you accepted. The README states plainly that you control what gets merged. This is a defensible choice for legal review or investigative work, where an incorrect merge of two people with similar names corrupts the graph in a way that is hard to detect downstream. It is also a cost. Review is manual work that scales with the number of proposed merges, and the material gives no indication of how many proposals a given corpus generates or how the terminal UI batches them. If your corpus produces thousands of candidate merges, the review step becomes the bottleneck rather than the LLM calls. The README does not describe a threshold or auto-merge policy for high-confidence cases, which would be the obvious escape hatch. Treat the review step as a genuine time commitment and size your first corpus accordingly.
Where it fits badly: incremental growth, scale, and the re-extract boundary
The README says the graph persists across sessions and grows incrementally: extract new documents into the same output directory and rebuild. That is the documented growth path, and it deserves scrutiny. Adding documents means running extract again over the new material and then build, which implies the graph is reconstructed rather than patched in place. The README does not describe how prior merge decisions are preserved when the graph is rebuilt, nor whether resolve re-proposes pairs you already rejected. If you rejected a merge and a rebuild resurrects it, the review work is not durable in the way the persistence claim suggests. This is the single most important thing to verify before adopting sift-kg for a corpus you intend to keep feeding. The second boundary is scale. Extraction is per-document LLM work, and the only cost control named is --max-cost. There is no discussion of parallel extraction, caching, or incremental diffing. For a few hundred documents with a mid-priced model this is probably fine. For a continuously updated corpus it is the wrong shape: you want a system that ingests deltas, not one that rebuilds. The third boundary is that the graph is only as good as the extraction schema, and schema discovery is a single LLM call over a sample. If your corpus is heterogeneous, one sampled schema may fit the first half of your documents and miss the entity types that matter in the second half.
Alternatives and the actual difference in approach
The nearest comparison is GraphRAG-style pipelines, which also extract entities and relationships from a corpus with an LLM and build a graph for retrieval. The difference is where the graph goes. Those pipelines are built to serve retrieval at query time inside an application, and the graph is an internal index. sift-kg treats the graph as the deliverable: it exports to GraphML, GEXF, CSV, SQLite and native JSON, opens in Gephi, yEd or Cytoscape, and ships a browser viewer with community regions, focus mode, keyboard navigation and a trail breadcrumb. The human-approval merge step has no equivalent in most retrieval-oriented pipelines, which merge automatically because no one inspects the result. The other comparison is manual knowledge bases in Notion or Obsidian, which the README dismisses on time grounds. That dismissal is fair on effort but understates the difference: a hand-built wiki encodes judgement about what matters, while sift-kg encodes whatever the extraction model found. If your goal is a curated concept map for teaching, the manual route still wins on quality. If your goal is coverage of a corpus too large to read twice, the extraction route wins. Pick based on which of those two you actually need.
Licence, maintenance cost, and what to check before you commit
sift-kg is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is the permissive end of the spectrum and imposes no copyleft obligation on your own code. Two caveats follow from the material rather than from the licence text. First, the tool depends on an LLM provider, and your documents leave your machine unless you run Ollama locally; the README lists Ollama as a local/private option, which is the configuration to choose if the corpus is sensitive. Second, OCR backends vary in where processing happens: Tesseract, EasyOCR and PaddleOCR run locally, while Google Cloud Vision sends page images to Google. Choose the backend deliberately rather than by default. On maintenance, the last push recorded is 2026-05-12 and no releases were retrieved, so there is no versioned artifact to track. Upgrades mean re-running pip install and re-reading the README for changed flags. Because discovered_domain.yaml and sift.yaml are plain files in your project, schema and domain settings survive an upgrade, but merge decisions live in whatever the apply-merges step writes and the material does not name that file. Locate it before you accumulate review work you cannot afford to redo. This is not legal advice; if you redistribute sift-kg or bundle it into a product, read the MIT text yourself.
Editorial conclusion
Adopt sift-kg if you have a bounded document collection, an LLM API key or a local Ollama model, and you want a graph you can inspect and export rather than a hosted knowledge base. Skip it if you need incremental extraction from a live stream, if the corpus is large enough that per-document LLM calls and a full re-extract on growth are unacceptable, or if you cannot review merge decisions yourself. Before committing, run sift init, check that the schema discovered from a sample of your own documents names the entity types you actually care about, and confirm your chosen provider and --max-cost ceiling keep a full extract inside budget.
Community notes