SigMap: a deterministic signature map for AI coding sessions

The problem SigMap targets: context an agent cannot prove

An agent working in a large repository usually gets its bearings by grepping, opening files and pasting what it found into the next prompt. That loop has no fixed output. The same question asked twice can surface different files, and nothing in the transcript records which of the referenced paths actually exist. The README frames the gap sharply: the map it produces is what agentic grep is worst at, namely reproducible, auditable context plus a check that an answer is anchored to real signatures and line numbers.

The audience follows from that framing. It is teams that already run Claude, Copilot, Cursor, Aider, OpenCode, OpenHands or Cline against a codebase and want a cheaper, more inspectable way to decide which files belong in the prompt. It is also teams that want to review AI output mechanically rather than by reading it closely. The README lists local runtimes (Ollama, llama.cpp, vLLM) alongside cloud models, so the tool is aimed at people who care about keeping context on their own machine as much as at people watching token spend.

What the map actually contains, and why byte-stability matters

SigMap does not call a model and does not build embeddings. According to the README, it walks the repository and emits a signature-and-evidence map: files and symbols, each tied to a real line anchor. Because the process is deterministic, the same repository produces the same map, which is what makes the output diffable and cacheable. A changed map then means the code changed, not that a sampling temperature moved.

That property is the whole design bet. An index that is stable under repetition can be committed, compared between branches, and used as a gate. An index produced by an LLM summarisation pass cannot be any of those things without extra machinery. The cost of the bet is that the map only knows what a parser can see: names, shapes and positions. It has nothing to say about intent, and the README does not claim otherwise.

Ranking on top of the map is TF-IDF, per the six-step flow in the README (Ask, Rank, Context, Validate, Judge, Learn). TF-IDF is a lexical scorer, so a query that shares vocabulary with your identifiers retrieves well, and a query phrased in business language retrieves poorly. The README does not describe a synonym layer or a stemming configuration, so treat lexical overlap as the retrieval contract.

The verify command is the part worth evaluating first

The README calls sigmap verify the grounding flagship and shows it indexing the repository plus the libraries actually installed in that working tree. It then flags fabricated files, imports, symbols, tests and npm scripts in a written answer. The documented invocation is sigmap verify answer.md, with a --json mode that exits 1 when issues are found and a --report mode that writes a standalone HTML report. verify-ai-output is the same command under its longer name.

The sample output in the README is the clearest statement of what the tool claims to do. It reports a fake file at one line and a fake symbol at another, and for the symbol it prints a suggestion: authorize() with the note "did you mean authenticate()?". That suggestion implies the verifier matches unknown identifiers against the symbol index and offers nearest candidates, which is more useful than a bare boolean.

Two companions are documented. sigmap verify-plan checks a plan before execution, and the verify_suggestion MCP tool verifies AI code against repository, private and installed-library symbols mid-session. The MCP server is what makes the check available inside an agent loop rather than only after the fact.

Getting it running: npx, the ask flow, and the MCP server

The README's quickest path needs no install. Two commands are given verbatim:

npx sigmap npx sigmap ask "Where is auth handled?"

The first produces the map, the second ranks files against a question. The README describes the result of ask as a ranked file list and says the Context step writes compact signatures to your AI's context file, though it does not name that file in the material available here, so the path is something to confirm in the repository rather than assume.

From there the documented loop is sigmap validate to confirm the right files are in scope, sigmap judge to score how grounded an answer is against the context, and sigmap weights to boost files that keep solving your tasks. The weights step is the one stateful piece in the flow: it accumulates preference across sessions, which means the map stops being purely a function of the current tree and starts being a function of your history as well.

For editor and agent integration the project ships an MCP server, which is what exposes verify_suggestion to a client. The README also states there is a homepage at sigmap.io with a benchmark methodology page, and that the benchmark numbers are generated from benchmarks/latest.json via npm run metrics:sync rather than typed by hand. That last detail is worth noting because it means the figures in the README are derived from a checked-in artefact you can inspect.

The benchmark numbers and what they do not measure

The README reports a hit@5 of 81.1% against a grep-agent baseline of 44.0%, which it describes as a 1.73x lift, and a token reduction of 96.8% averaged across 21 repositories. It also reports a prompt reduction from 2.84 to 1.54 prompts per task and a task success figure of 64.8%.

The prompt and task-success figures carry an explicit caveat in the README itself: the task success number is labelled a proxy, "modeled from retrieval tiers, not measured LLM sessions", and the prompt reduction is likewise marked modeled. So two of the four headline numbers are derived rather than observed. That does not make them wrong, but it does mean the only directly measured claims here are retrieval hit rates and token counts. The README also notes the benchmark covers 90 coding tasks across 18 real public repositories while the metrics block says 21 repositories, and the language list in the benchmark block is broader than the languages named elsewhere. If retrieval quality on your language is the deciding factor, resolve that discrepancy against the repository before trusting the aggregate.

One more framing note: the description attached to the repository advertises roughly 97% token reduction, and the README's own position is that token reduction "comes for free" while trust is the point. Those are two different pitches for the same tool, and the second one is the more defensible.

Where SigMap is the wrong tool

The clearest failure mode is semantic retrieval. With TF-IDF ranking and no embeddings, a question that does not share tokens with your code will rank badly. "Where do we charge a customer twice?" will not find a function called applyLedgerAdjustment unless the query happens to contain those words. A vector index handles that case better, at the cost of infrastructure, API spend and non-reproducible output.

The second limitation is verification scope. The README says verify indexes the repository plus the libraries installed in that working tree. That is a local view. An answer that references a symbol from a dependency that is not installed in the tree being checked, or from a service reached over the network, sits outside what the index can confirm. The verifier's silence on such a reference is not evidence that the reference is real.

The third is parser coverage. The README advertises 33 languages, but a signature map is only as good as the parsing behind each one, and the material here does not break the count down by parser maturity. A repository in a language with shallow support would produce a thin map, and a thin map makes the fake-symbol check less useful rather than more.

Finally, the weights step introduces drift. Because sigmap weights boosts files that keep solving your tasks, two developers on the same commit can end up with different rankings. The map is deterministic; the ranking on top of it may not be, once that command has been run.

Alternatives and the actual difference in approach

The obvious comparison is a source-code embedding index such as a vector store built with an embedding model and queried by similarity. The difference is not speed, it is what the artefact is. An embedding index stores vectors that change when the model or its version changes, and similarity scores are not reproducible across runs in any strict sense. SigMap stores signatures and line anchors and makes no model call, so its output is byte-stable and diffable. You trade recall on paraphrased questions for an index you can commit and gate on.

The second comparison is the agentic grep loop the README positions against, meaning an agent that runs ripgrep, reads files and decides for itself what matters. That loop is flexible and needs no setup, and it can follow a hunch across a repository in ways a TF-IDF ranking cannot. What it cannot do is hand you a stable artefact or a mechanical check on whether a cited path exists. SigMap is the narrower tool: it answers fewer kinds of question, but it answers the same way every time and it can say no.

A third point of comparison is a language server. An LSP already resolves symbols and imports precisely, and its resolution is more accurate than a text index for the languages it supports. SigMap's difference is packaging: it produces one cross-language map plus a verification command that runs offline and exits non-zero, which is a CI-shaped output rather than an editor-shaped one.

Maintenance cost, licence and what to check before adopting

The project is MIT licensed, so the code can be used, modified and redistributed commercially provided the copyright notice and permission notice are retained. That is a permissive licence with no copyleft obligation, and it is the same licence most JavaScript tooling ships under. Nothing here constitutes legal advice; if the map is committed into a proprietary repository, the usual notice-retention requirement still applies to the tool itself.

The dependency story is genuinely cheap: the README states zero dependencies and shows npx sigmap as the entry point, so there is no install step and no transitive tree to audit. The cost that remains is runtime and output churn. The tool requires Node on every machine that runs it, including CI runners, and a deterministic map still changes whenever the code changes, so a committed map will produce diffs on every branch that touches indexed files. The release cadence is also fast: v8.28.1, v8.29.0 and v8.30.0 all landed within roughly three weeks of each other, and v8.29.0 is titled "Retrieval Index Split", which suggests the index format itself is still moving. Pin the version in CI rather than tracking latest.

Before adopting, run sigmap verify answer.md --json against a few real AI answers from your own repository and read the reported lines, because a verifier that flags correct references will be ignored within a week. Then confirm your primary language is among the 33 with real parser support, and decide deliberately whether to run sigmap weights at all, since that command is the one place where the deterministic story stops holding.

Editorial conclusion

Adopt SigMap if you already paste file context into an agent and want a repeatable, offline way to catch fabricated paths and symbols before a patch lands, and if a Node runtime on the developer machine is acceptable. Do not adopt it as a semantic search engine, and do not expect it to replace an embedding index for questions phrased in domain language rather than code identifiers. Before rolling it into CI, run sigmap verify against a handful of real AI answers from your own repository and read the false-positive lines yourself, then check whether the language you care about is inside the 33 the README claims, since the benchmark repository list and the advertised language count are not the same set.

Official sources

1. License: MIT
2. manojmallick/sigmap on GitHub
3. Project website
4. README
5. Releases