Claw Compactor: A 14-Stage Token Pipeline That Routes by Content Type
14-stage Fusion Pipeline for LLM token compression — reversible compression, AST-aware code analysis, intelligent content routing. Zero LLM inference cost. MIT licensed.
At a glance
- What is it?
- Claw Compactor chains fourteen named compression stages behind an auto-detecting router, aiming to cut LLM context cost without spending inference on compression. The design is content-aware rather than perplexity-driven, but the README leaves several operational questions unanswered.
- Who is it for?
- Adopt Claw Compactor if your prompts are dominated by structured material: source files, JSON payloads, logs, diffs or search results, and you want compression that runs locally with no model call and can be undone through RewindStore. Do not adopt it if your context is mostly conversational prose, if you need a published accuracy guarantee before shipping, or if you cannot run the benchmark subcommand against your own workspace first.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 168 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The bill Claw Compactor Is Trying to Cut
Long agent contexts are expensive in a way that is easy to ignore until the invoice arrives. A coding agent that pastes an entire repository into a prompt pays for every import line, every duplicated log entry, and every JSON key it will never reason about. The README's own demo shows a 47-file workspace at 234,891 tokens, priced at $2.35 per run at GPT-4 rates, dropping to 108,250 tokens and $1.08 after the pipeline runs. That arithmetic, not the compression percentage, is the pitch.
The target user is someone building agents or retrieval pipelines who controls the prompt assembly step. If you are calling a hosted API through a framework that hides the message array, there is nothing here for you to hook into. Claw Compactor sits between your content and the model, which means you must own that seam.
The project's framing is that perplexity-based token dropping is the wrong primitive for structured text. The README states plainly that LLMLingua-2 drops tokens by perplexity score, which it says is effective for natural language but destroys code identifiers, JSON keys, and log patterns. Whether or not you accept the comparison table's numbers, the underlying claim about what perplexity scoring does to a function signature is testable on your own files.
Fourteen Stages, One Frozen Context
The architecture is a linear chain, not a tree. Cortex runs first and auto-detects content type and language across sixteen languages, then downstream stages make decisions based on what Cortex decided. The stages are named and each has a stated job: QuantumLock for KV-cache alignment, Photon for base64 stripping, RLE for path shortening, SemanticDedup for simhash-based deduplication, Ionizer for JSON statistical sampling, LogCrunch for log folding, SearchCrunch for result dedup, DiffCrunch for context folding, StructuralCollapse for import merging, Neurosyntax for AST compression via tree-sitter, Nexus for ML token classification, TokenOpt for format optimization, and Abbrev for natural-language shortening.
Two mechanics matter more than the stage list. The first is immutable data flow: FusionContext is a frozen dataclass and every stage returns a new FusionResult rather than mutating in place. That makes stage ordering debuggable, because you can inspect the intermediate result at any boundary. The second is gate-before-compress. Each stage exposes should_apply(), which inspects context type, language and role before doing any work. Skipped stages cost nothing, which is why the demo table shows Photon applied to 3 of 47 files while RLE applied to 41.
The demo output is worth reading carefully because it shows the per-stage reduction is uneven and sometimes large in isolation. Ionizer applied to only 8 of 47 files and reports 71.2 percent reduction on those files. Neurosyntax hit 23 files at 18.4 percent. A stage that applies rarely but cuts hard is doing targeted work, not general-purpose summarization. The aggregate was 53.9 percent in that example, inside the README's stated 15 to 82 percent band.
Reversibility Lives in a Hash-Addressed LRU
Ionizer is the stage that makes the reversibility claim concrete. When it samples a JSON payload statistically, it stores the original in a RewindStore, described in the pipeline diagram as a hash-addressed LRU for reversible retrieval. The README's sentence is cut off mid-word at "The LLM can ca", so what is confirmed is the storage mechanism and the intent, not the retrieval API.
That gap matters. Reversible compression is only useful if the caller can actually invoke the rewind at the right moment, and the material does not show the method signature, the key format, or what happens when an entry has been evicted from the LRU. An LRU is a bounded cache by definition. If your agent compresses a large workspace, then needs an original three turns later after other content has passed through the store, eviction behaviour decides whether you get the original back or an error.
Treat reversibility as a property to verify rather than a property to assume. The mechanism is present in the architecture; the operational contract around it is not documented in the material supplied here.
Install and the First Command You Should Run
The package is published to PyPI as claw-compactor, requires Python 3.9 or higher, and the README advertises zero required dependencies. Installation follows the standard path, and the CLI entry point is claw-compactor.
The command that matters most is the benchmark subcommand, which takes a workspace path and prints the per-stage table:
claw-compactor benchmark ./my-workspace
Run that before integrating anything. The output tells you which stages actually fire on your content and what each one contributes. If your workspace is mostly prose, most of the table will show low apply counts and you will learn that cheaply. If Ionizer and Neurosyntax dominate, you have confirmation that the structured-content thesis applies to your corpus.
The README references an API section and a Quick Start section, and points to a documentation site at open-compress.github.io/claw-compactor plus an ARCHITECTURE.md in the repository. Configuration keys are not enumerated in the material available here, so the honest position is that the CLI surface is documented and the programmatic configuration surface is not, at least not in what was supplied.
The Dependency-Free Claim and What It Costs
Zero required dependencies is a real deployment advantage. LLMLingua-2 requires torch and transformers; SelectiveContext requires torch. Claw Compactor requires neither, which means no GPU, no multi-gigabyte download, and no model weights to pin. The README puts latency under 50ms against roughly 300ms for LLMLingua-2 and 200ms for SelectiveContext, and inference cost at $0 because no model is invoked.
That advantage is also the constraint. A pipeline that does no inference cannot make semantic judgements about what matters in a passage. It can detect that a block is JSON and sample it statistically. It can parse a file with tree-sitter and collapse imports. It cannot decide that a particular paragraph is the one the user actually cares about. The README's own comparison table concedes the tradeoff indirectly: the compression band is 15 to 82 percent, a wide range that reflects how much the result depends on content type. gzip plus base64 gets 60 to 80 percent but the table marks its output as not LLM-readable, which is the reason raw compression is not a substitute.
The tree-sitter dependency deserves scrutiny despite the zero-dependency headline. Neurosyntax is described as AST compression via tree-sitter, and Cortex detects sixteen languages. Grammar availability per language is a practical question the material does not answer. If your repository is primarily a language outside that set, Neurosyntax will not fire and you fall back to the text-oriented stages.
Where This Is the Wrong Tool
Conversational context is the clearest mismatch. Abbrev is described as natural-language shortening and applies to text only, which suggests the project's own authors treat prose as the hardest case. If your prompts are user messages, chat history and free-form instructions, most of the fourteen stages have nothing structural to latch onto. You would be paying the integration cost for the RLE and Abbrev stages alone.
A second failure mode is anything that requires exact reproduction of the compressed region on the first pass. If your downstream consumer needs the original JSON keys, the original log timestamps, or the original import list, and it needs them without a round trip through RewindStore, compression is the wrong move. The reversibility exists precisely because some of that information is lost from the visible payload.
A third case is evaluation. The README cites ROUGE-L figures of 0.653 at compression rate 0.3 and 0.723 at 0.5, and states 1600+ tests. Those are the project's numbers for the project's benchmarks. They are not a guarantee about your task. A pipeline that scores well on ROUGE-L can still drop the one identifier your agent needed to complete a refactor, and ROUGE-L will not tell you that.
Against LLMLingua-2 and SelectiveContext
The comparison in the README is specific enough to be useful. LLMLingua-2 compresses by perplexity, using a trained model to score tokens and drop the low-information ones. That costs roughly $0.02 per call and about 300ms, requires torch and transformers, and produces output the README characterizes as partially LLM-readable. SelectiveContext works from self-information scores, also requires torch, and the README places it at 10 to 40 percent compression with roughly 200ms latency.
The difference in approach is what the stages operate on. Both alternatives score tokens by some measure of surprise and remove the unsurprising ones. Claw Compactor instead classifies the content first and applies a stage written for that content type: statistical sampling for JSON, folding for logs, AST manipulation for code, dedup for search results. The claim is that a JSON key is not low-information just because it is predictable, and perplexity scoring cannot distinguish a predictable key from a disposable one.
A fair reading is that these are different tools for different corpora. If your context is natural-language documents and you already have torch in your stack, perplexity-based compression is a mature approach with published evaluation. If your context is a code repository, a log stream, or an API response payload, the stage-per-content-type design is a better structural fit.
Maintenance, Licence and What to Check First
The licence is MIT, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is a permissive arrangement with no copyleft obligation on your own code. This is a description of the licence text, not legal advice; if you are redistributing the package inside a product, read the LICENSE file in the repository.
The release history shows recent churn around packaging rather than features. v7.0.1 fixed PyPI packaging, v7.0.2 fixed wheel packaging for CI, and v7.1.0 is titled Modular Architecture plus Repo Polish, all within roughly two weeks in March 2026. Two consecutive packaging fixes before a minor release is a signal that the distribution path was unstable at that point. It also means the version in the JSON-LD block, 7.0.0, lags the latest release tag, 7.1.0.
Upgrade cost is hard to estimate from the material. The pipeline is stage-based and the stages are named, so adding or reordering stages is the kind of change that can shift output on the same input without any API break. If you pin claw-compactor, pin it exactly and re-run claw-compactor benchmark on a fixed workspace after each bump, comparing the stage table rather than the summary percentage. A stage that starts firing on content it previously skipped will change your token count and possibly your output quality, and the summary line alone will not show you which stage moved.
Editorial conclusion
Adopt Claw Compactor if your prompts are dominated by structured material: source files, JSON payloads, logs, diffs or search results, and you want compression that runs locally with no model call and can be undone through RewindStore. Do not adopt it if your context is mostly conversational prose, if you need a published accuracy guarantee before shipping, or if you cannot run the benchmark subcommand against your own workspace first. Before wiring it into a pipeline, verify three things yourself: that the stage table produced by claw-compactor benchmark on your files matches the README's claimed 15 to 82 percent band, that Ionizer's RewindStore round-trip actually restores the identifiers and keys your downstream code depends on, and that the tree-sitter grammar coverage includes the languages in your repository rather than only the sixteen Cortex detects.
Community notes