clp: search compressed logs without decompressing them

Compression that stays searchable

The Compressed Log Processor is a free log management tool capable of compressing logs and searching the compressed logs without decompression. It supports both JSON logs and unstructured, meaning free text, logs, and it also supports real-time log compression within several logging libraries. CLP also includes purpose-built web interfaces for searching and viewing the compressed logs.

That last clause, the web interfaces, is what turns this from an archiver into a log management system, and the references for that are two papers, a 2021 one about handling unstructured logs and a 2024 one on extending it to JSON logs. Both are USENIX papers, and the project treats them as the primary documentation alongside the site.

The design decision that makes the whole thing work is the intermediate representation. The logging libraries parse the log lines before compressing them, so the compressed form is not opaque bytes but a structured representation of what was in the line. That is why a filter on log level works, why a search can run without decompressing the archive, and why the compression ratio beats a general purpose compressor.

The project is from YScope, the primary language in the repository listing is C++, the licence is Apache-2.0, and the site is yscope.com. The last push was on 2026-10-15 and the repository is not archived.

The pipeline: compression, search, analytics, viewing

The system overview describes CLP as an end-to-end log management pipeline consisting of compression, search, analytics and viewing, and the component list explains what each part is.

Compression and search is the core. CLP compresses logs into archives which can be searched and analysed in a web interface, and the input can either be raw logs or the compressed intermediate representation produced by CLP's own logging libraries.

Real-time compression is the part that changes how your application behaves. There are logging libraries for Python and for Java, covering Log4j1, Log4j2 and Logback, and they compress logs in real time so that only compressed logs are written to disk or transmitted over the network. The compressed logs use the intermediate representation format, which achieves a higher compression ratio than general purpose compressors like Zstandard.

The log viewer is a separate project, and the README compares it to viewing logs in an editor with specific advantages. It supports advanced features like filtering logs based on log level verbosity, for example showing only logs at ERROR or above, and it can do that because CLP's logging libraries parse the logs before compressing them into the intermediate representation.

The analytics side is exposed as libraries rather than only as a service, with a Python library and a Go library that can analyse compressed intermediate representation directly.

And there is a log parser, built on pushdown automata, which the project claims is three times faster than state-of-the-art regular expression engines like RE2. It is available as a library other applications can use.

IR versus archives, and the memory trade-off

There are two forms of the compressed output, and the difference between them is a design trade-off the documentation spells out.

The intermediate representation is what the logging libraries produce. It uses a format that achieves a higher compression ratio than general purpose compressors like Zstandard, and it is what the log viewer and the analytics libraries consume.

Compressing the intermediate representation into archives can further double the compression ratio and enable global search, but this requires more memory usage, as it needs to buffer enough logs. The more details on IR versus archives are in an Uber Engineering Blog post linked from the README, which is a useful reference because the trade-off is not obvious from the outside.

So the choice is between two things. The intermediate representation alone is fast to produce and already structured enough for a level filter and a targeted search, and the archive adds roughly a factor of two in compression and the ability to search across everything, at the cost of buffering.

That buffering requirement is the practical constraint. An archive that needs enough logs buffered to be worth compressing is a different operational model from a log stream you forward as it happens, and it is why the project ships both the real-time libraries and the archive stage, and why the distributed package exists as a separate download from the core container.

The benchmarks, and how they were run

The benchmark section is unusually careful about methodology, and the details are worth reading before the numbers.

The figures compare CLP's compression and search performance against other tools. The experiments are separated between JSON and unstructured logs, for two reasons: some tools can only handle one type of log, and tools that can handle both often have different designs for each type, such as CLP does.

Compression ratio is measured as the average across a variety of log datasets, and some of those datasets are published so you can reproduce the measurement.

Search performance is measured using queries on the MongoDB logs for JSON and the Hadoop logs for unstructured logs. The note that matters is the next one: CLP uses an index-less design, so for a fair comparison the indexes on MongoDB and PostgreSQL were disabled, because with them left enabled the compression ratio of those two would be worse. The indexes on Elasticsearch and Splunk were not disabled, since these tools are fundamentally index-based, meaning logs cannot be searched without indexes.

That is a fair thing for a competing project to do, and it is also a reminder that the comparison is between different designs rather than different implementations of the same one. CLP is arguing that you do not need an index at all, which is a strong claim, and the way the comparison is set up is part of that argument.

The Rust workspace and the search engine underneath

The Cargo manifest shows a workspace with six members and an edition of 2024:

[workspace]
members = [
    "components/api-server",
    "components/clp-rust-utils",
    "components/clp-tdl-package",
    "components/compression-coordinator",
    "components/log-ingestor",
    "components/query-coordinator",
]
resolver = "3"

The names describe the topology. The log ingestor takes logs in, the compression coordinator turns them into archives, the query coordinator handles searching, the api server exposes the service, clp-rust-utils is the shared library, and clp-tdl-package is the package format for the data.

The dependency list is where the interesting choices are. axum is the web framework, tokio the runtime, clap the command line, sqlx with the MySQL runtime for one of the backends, and mongodb at 3.8 for the other. The AWS SDK covers S3 and SQS, which is the object store and the queue for the distributed deployment. OpenTelemetry and the OTLP exporter are there for the project's own metrics. rmp-serde is MessagePack, which is the serialisation format for the intermediate representation, and secrecy is the crate for keeping credentials out of the log output.

And then there are the spider crates, spider-client, spider-core and spider-tdl, taken from a git repository at a pinned revision rather than from crates.io. Pinning those crates to a commit rather than taking them from crates.io suggests that the Spider crates and the rest of the workspace move in step.

The rest of the repository, and how to try it

The repository root is broader than the Rust workspace, and the extra directories tell you the project spans more than one language.

There is components/ holding the Rust crates, docs/ with the documentation source, integration-tests/, python-wheels/ for the Python packages, tools/ for helper scripts, and a taskfile.yaml with a taskfiles/ directory, which is the Task runner for the build. The Python side has its own requirements file pinning pip at about 23.3 and setuptools at about 75.2.0, and a ruff.toml for linting with a lint-requirements.txt beside it. There is a .cargo/ directory for cargo configuration, a Cargo.lock, a NOTICE file alongside the LICENSE, and .gitmodules for the vendored pieces.

To try it there are two routes. You can download a release package, which includes support for distributed compression and search, or, to quickly try the core compression and search, you can use a prebuilt container. There are also guides for building the package and for building the core from source, and for some logs to test with, the project points at its own open-source datasets.

The documentation is available online or in the source tree at docs/src, and the community is spread across Discord, Slack and Zulip, with the Discord being the busiest by the badge. The project describes itself as an open-source release which will be constantly updated with bug fixes and features, and it asks for issues and feature requests rather than direct contributions, which is the model of a project whose upstream is the YScope organisation's own products.

The release history is regular, with v0.12.0 on 2026-04-22, v0.13.0 on 2026-07-22 and v0.14.0 on 2026-10-01.

Editorial conclusion

Adopt CLP if you run services that emit a lot of logs and you want to keep them compressed while still being able to search them, and you are willing to route your logging through the CLP libraries or the log parser rather than shipping raw text. Do not adopt it if you need arbitrary ad hoc queries over old archives, because the search model is built around structured fields and parsed log levels rather than free-form scanning. Before you start, decide between the full release package with distributed compression and search and the core container for a quick trial, and read the benchmark methodology, because the comparison figures are produced with the other systems' indexes turned off and that context changes how you read them.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. y-scope/clp on GitHub