Leviathan indexes records once so an agent reads a few cards, not a log

The problem is context budget, not search

Leviathan is described as deep memory for agents over large datasets, and the framing is important. The target is not a developer who wants a search engine. It is an agent that currently answers a question by grepping and reading raw history, which means the entire history ends up in the context window, and the answer arrives with a large bill attached. The stated alternative is to index any table, export or log once, so the agent gets the few records that answer the question, with result cards of roughly 450 tokens per answer at any dataset size. The output shape is a citation card rather than a chunk of text, which is the point: a card carries the id, the date, the fields you configured for display and the matched passage, so the agent can answer and point at the source. The audience is engineers wiring an agent to an operational dataset, and the alternative to adopting it is grep.

The published comparison is the project's own claim

The README carries a table of results at 1M records and 678 MB, and it is worth reading with the right amount of scepticism. At that scale the median tokens per question are given as 436 for Leviathan against 107,122 for the best grep strategy, a factor of 245, the relevant record was returned 99.0 percent of the time in the top 5 and 98.5 percent at rank 1 against 96.0 percent within a 30K character output for grep, the worst case across 1,200 questions is 602 tokens against 9.7M, and median latency is 33 ms against 92 ms. The README is explicit about the conditions: it was measured on a synthetic maintenance log, described as one example dataset with nothing in Leviathan specific to it, and the methodology, all six scales and the caveats live in docs/BENCHMARKS.md. So the numbers are a claim by the project about a shape of workload, not a guarantee for your data.

Any dataset reaches the index through a pipe

The input side is the part that makes the tool usable on data you cannot move. The package description says it indexes JSONL, JSON, CSV or TSV, SQLite, or anything a database CLI can export, and the README makes the same point in stronger terms: any database works through its own CLI, and Leviathan never holds credentials.

psql "$DATABASE_URL" -At -c "SELECT row_to_json(t) FROM tickets t" | leviathan index - -c tickets.toml
leviathan index app.db --sql "SELECT * FROM tickets" -c tickets.toml
duckdb -json -c "SELECT * FROM 'events/*.parquet'" | leviathan index - -c events.toml

The dash as the source means standard input, so a database that can emit rows as JSON becomes indexable without a connector and without credentials living in the indexer. A local SQLite file is read directly with an SQL statement, and a parquet directory is read through duckdb. The result is a single SQLite file, with no runtime dependencies, and prebuilt binaries are published on the Releases page.

A field map is the configuration surface

Leviathan does not guess your schema, it is told. Fields are paths such as a.b or items[].name, and only id is required. The mapping table in the README lists what each field buys you: id enables get, upsert, delete and citations, title and text control the card headline, with title weighted at twice the weight, and the searched text, group and group_name give scoped search with the -g flag and name resolution with a labeled fallback, date enables the --since and --until filters and the recent ordering, filters and display drive the --where facets and which fields appear on the card, and empty_values and rank.boost let you treat placeholders as missing and favour records that are complete.

leviathan init ./export                       # infer a commented leviathan.toml
leviathan index tickets.csv --id "Ticket ID" --group customer_id --date created_at   # or flags
leviathan index -c tickets.toml               # or a config your agent wrote from the schema
leviathan describe                            # fields, groups, filter values, example calls

The init command infers a commented configuration file, flags work as a shortcut for the same thing, and describe prints the fields, groups, filter values and example calls, which is the command to run first against a real export.

Search refuses to guess when the scope is ambiguous

Two behaviours distinguish this from a plain grep. The -g flag accepts a key, a name or a partial name, and when the group is ambiguous or unknown the command lists candidate groups and exits with code 3 rather than picking one. If nothing matches inside the requested group, the search falls back to the other groups and labels the result OTHER CUSTOMER, so a result is never silently missing. The query language adds filters, with --where field=value, --since and --until, quoted phrases and -exclusions. The result cards are the human and agent readable form of this, and the example output in the README shows the shape: a header with the customer, the query, how many records were shown and how many were indexed, then a numbered record with its id, timestamp, relevance, the title, the configured display fields, and the matched passage.

leviathan search · customer C-ACME "Acme Corp" (7 tickets) · query "sso login loop after password reset" · shown 3 of 3 · 24 tickets indexed
[1] T-1001 · 2024-01-08 09:12 · rel 16.9
  Login loops back to sign-in page after password reset
  status: closed · priority: high

The build side is designed for a dataset that changes. Builds are atomic and skipped when nothing changed, and upsert and delete keep an index fresh, so the index does not have to be thrown away and rebuilt on every change.

Three ways to reach it, from a shell skill to a remote server

Leviathan exposes the same search surface through three integration paths. The recommended one is the CLI plus a skill: any agent with a shell can call it, and you copy the skills/leviathan skill into ~/.claude/skills/ or paste it into AGENTS.md or .cursor/rules, which the README notes costs zero tokens until used. The MCP path is optional and runs leviathan mcp, which serves four read-only stdio tools, search, resolve_group, get and describe, whose descriptions include a summary of your dataset at about 640 tokens per session. leviathan wrap <agent> prints the configuration for 24 agents, from Claude Code and Codex to Cursor, VS Code with Copilot, Gemini CLI, Windsurf, Cline, Zed and JetBrains, and --apply writes it. The remote path is leviathan serve --http, which serves the same tools as Streamable HTTP MCP, REST and OpenAPI, with a bearer token or OAuth 2.1, so that claude.ai, ChatGPT, the Anthropic and OpenAI APIs and agent frameworks can connect.

The memory store replaces a markdown file, with a key per fact

The optional memory side is a separate feature from the index and lives in its own SQLite file. The agent remembers one claim at a time, and the design point is that anything which can change has a key, so the next write replaces the old value instead of contradicting it. Sessions start with a short briefing and recall returns only what fits a token budget.

leviathan wrap claude --memory --hooks --rules --apply   # tools, session-start briefing, always-on rule
leviathan memory remember "Prefers pnpm over npm" --kind preference -s josh -k package_manager
leviathan memory recall package manager
leviathan memory import ~/.claude/projects/<project>/memory/MEMORY.md   # bring an existing markdown memory

The README quantifies the trade: after 300 sessions, tokens at session start are 736 for Leviathan memory against 33,313 for the whole MEMORY.md file and 2,588 for its first 200 lines, and the stale value in context figure is 0 percent against 73.8 percent and 36.4 percent. The mechanism behind the last number is the key, since a keyed fact is overwritten rather than appended, so the old value stops being carried. The import command exists for people who already have a markdown memory file and want to bring it in.

Where this is the wrong shape of tool

The limitation follows from what the index is. It is a ranked full text index over records, and full text means lexical, so a question whose wording shares no terms with the record is a hard case. The accuracy figure the README publishes, an answer rate against dataset size, is the graph to look at for that, and nothing in the project claims semantic or embedding based retrieval. If your corpus is long prose where the answer is a summary rather than a located passage, a different tool fits better. The second limit is the rebuild problem. The index is derived from a source, so a dataset that changes on every request, a live activity stream, for example, means a pipeline rather than a one off step, and the atomic build and skip when unchanged behaviour only helps when the source has stable chunks. The third is scope of the code itself. The crate is young, at version 0.2.0, with v0.1.0 on 2026-10-06 and v0.2.0 on 2026-10-09, so the interface between the index, the MCP tools and the memory store is still moving, and the Apache 2.0 licence with a NOTICE file and a THIRD_PARTY.md is worth reading before you depend on it.

Editorial conclusion

Leviathan suits a team whose agent currently reads entire logs or ticket dumps into context to answer a question, and it suits anyone who wants one indexing step separated from the agent so the same index can be queried by a script, a skill or an MCP client. It is the wrong tool for unstructured prose that needs summarising rather than retrieving, since the index is a ranked full text index over records and not a semantic store, and it is the wrong tool when the dataset changes constantly, because an index over a moving table is a rebuild problem you have to own. Before adopting it, run leviathan describe against a real export to see how your fields map, keep the raw data in the database and let Leviathan read it through a CLI pipe so it never holds credentials, and read docs/BENCHMARKS.md before quoting any of the published token figures. The last push was on 2026-10-19 and the current release is v0.2.0 from 2026-10-09.

Frequently asked questions

Official sources

1. elstongun/leviathan on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases