cmix squeezes files harder by spending gigabytes of memory

Compression ratio is the goal, RAM and CPU are the price

Most general-purpose compressors trade a little ratio for a lot of speed. cmix does the opposite, and the project states the trade plainly: it is aimed at optimizing compression ratio at the cost of high CPU and memory usage. The hardware requirement is spelled out too. At least 32GB of RAM is recommended to run it, which is a floor for a program whose main lever is keeping more model state around.

The platform coverage is wide in the usual way for a command line tool written in C++. It works in Linux, Windows and Mac OS X, and on Windows it can be compiled with MinGW or Cygwin. The homepage is the author's own page, and the source is the canonical location for the project.

What you get back is a single compressed file that decompresses to the original bytes. There is no container format, no index and no partial extraction. That narrowness is part of why the ratio can be pushed so far: the compressor is free to spend everything on modelling the one file it was given.

One file per run, so archive before you compress

The single most important operational detail is that cmix can only compress and decompress single files. There is no directory mode and no multi-file mode. The documented workaround is to create an archive first using tar, or a similar tool, and then point cmix at that archive.

That has a consequence worth planning for. Compression happens on the tar file, so the archive format, its metadata and the order of the entries are all baked into the compressed bytes. If you need to pull a single file out later, you decompress the archive first and then untar it, and the extraction step needs free disk space roughly equal to the uncompressed size.

It also means cmix and tar compose badly with each other in one specific way. Recompressing an already compressed archive with cmix, for instance a tar of gzip files, gives the compressor a harder input than a plain tar of raw data, and the project does not promise any gain there. The practical workflow is to archive first and compress once, with the tar options you would use anyway.

For some kinds of input the project points at a separate step: preprocessing with precomp, from the precomp-cpp project, may improve compression for some files. That is an external tool the author recommends rather than a component of cmix itself.

Building from source, two build systems in the tree

There is no package recipe in the documentation, so the route to a binary is compiling it. The instruction is to use make:

make

The repository root contains both a `makefile` and a `meson.build`, so a Meson based build is available as well, and the sources live in a `src/` directory with a separate `dictionary/` directory. The dictionary directory matters because it is where the preprocessor dictionary and the article ordering file used by the enwik9 workflow live, and the change notes record that in older versions the dictionary used to be compiled into the binary itself before it was moved out.

The `COPYING` file holds the GNU General Public License, and the project describes itself as free software distributed under that licence. For a compressor that you might ship inside a product, that is the constraint to weigh: GPL-3.0 obliges you to offer the corresponding source to anyone you distribute the binary to.

The build is also where the project's own development history shows up in the file layout. The `src/` tree accumulates models and mixers over years, and the change notes in the project documentation are a running list of what was added or removed in each version.

The compiler flags decide whether a file survives the move

The build instructions call out an optimisation detail that is unusual to see stated this plainly. Compiling with `-Ofast -march=native` gives the fastest performance, but those options might lead to incompatibility between different computers due to floating-point precision differences. Compiling with `-O3` fixes the compatibility issues, at the cost of slower performance.

For a compressor whose entire value is the exact bytes it produces, that is not a footnote. Two builds of the same source, one tuned with `-march=native` for the local CPU and one portable, can disagree on the compressed output because the context mixer depends on floating point arithmetic. The compressed file produced by a fast local build may not decompress correctly on a machine with a different instruction set or a different compiler version.

The same caution applies to the CPU target in general. The `-march=native` flag bakes the build machine's instruction set into the binary, which is a reasonable trade for a tool you run once on one machine and a bad one for archives you intend to keep.

The rest of the workflow is one binary per direction: the program compresses a file and decompresses it, and the documentation points at the archive creation step in between when you are working with directories.

Model history shows the compressor growing by accretion

The project keeps a detailed changelog across versions, and reading it explains what the program actually is: a collection of statistical models whose predictions are mixed together, refined over many successive releases.

Some entries show models being removed. The PAQ8HP model was removed in version 20, the PPM model was removed in version 6, and picture models were dropped in version 2. Others show models being added. Version 10 added a PPMD model, version 8 added a recurrent backpropagation neural network to mix byte level models along with some PPM models, and version 12 replaced the byte mixer with an LSTM.

The mixing layer has been the recurring theme. Version 16 brought context mixer improvements, version 15 more of the same plus LSTM tuning and pretraining, version 3 merged paq8pxd code into paq8l and optimised the mixers, and version 2 introduced facade models whose purpose is to copy the paq8 model predictions to the cmix mixer. Version 4 added another mixer context based on longest match length.

The dictionary is part of the same story. Version 5 added dictionary pretraining and removed the dictionary from the cmix binary, version 14 brought a new dictionary from phda, and version 17 rewrote the WRT dictionary preprocessor and added a new dictionary from the phda 2017 November release.

The enwik9 workflow is a separate, manual procedure

The documentation describes a specific procedure for the enwik9 corpus, and it is worth reading because it shows how much of cmix's advantage comes from work that happens outside the compressor.

The steps are: copy `enwik9-preproc` and `dictionary/new_article_order` to a new folder, run `./enwik9-preproc c path_to_enwik9`, and the result is a file called `.ready4cmix` which is the preprocessed file ready for compression. To reproduce the original, the same tool is run in the other direction: `./enwik9-preproc d .ready4cmix`.

So the pipeline is preprocessing first, compression second, and the reverse on the way out. The preprocessing step reorders the corpus into an order that the models handle better, and the ordering file lives in the `dictionary/` directory. This is the same family of trick as the precomp preprocessing the project recommends for some files, and it explains why a plain `tar` of many similar files is not the interesting input for a compressor like this.

The version history also records where the improvements came from. Version 21, the current release, is described as carrying improvements from fx2-cmix, version 20 added the fxcm model and version 19 took improvements from STARLIT and cmix-hp. The project credits outside work rather than presenting itself as a closed system.

Where cmix is the wrong tool

The clearest case against cmix is the everyday one. A general compressor that returns in seconds and needs a few hundred megabytes is the right tool for a folder of documents you want to email. cmix is built for the opposite situation, where one file has to shrink as much as possible and the machine can sit there for hours with 32GB of RAM in use. If your workflow is compressing many files, you will be running the program many times, and each run re-does its work from scratch.

The second case is portability of the output. Because the fast build can change floating point behaviour and because `-march=native` ties the binary to one CPU, an archive produced on one machine may not reproduce or decompress on another. If the compressed file is meant to outlive the machine that made it, that is a serious drawback, and the portable `-O3` build trades away the speed that justifies the memory cost in the first place.

The third case is the licence. GPL-3.0 is fine for personal archiving and for internal tools, and it is a poor fit for a product that wants to ship the compressor inside a commercial binary without also shipping the corresponding source. The documentation does not discuss any alternative licence arrangement.

Editorial conclusion

cmix fits someone archiving a large file that has to fit in less space and who has a machine with 32GB of RAM and hours to spare, since it compresses one file per run and spends its memory budget on model mixing rather than on convenience. It is the wrong tool for a daily-use compressor on a laptop, for batch work over many files, and for anyone who needs a compressed file to open on an arbitrary other machine, because the fastest build options can change the bytes you get. Check three things before you start: that the target file is a single file you can archive with tar first, that the machine has the recommended 32GB of RAM, and which compiler flags you used, because the newest release is v21 from September 2024 while the last push was on 2026-09-27, so the source tree is moving ahead of the tagged version.

Frequently asked questions

Official sources

1. byronknoll/cmix on GitHub
2. License: GPL-3.0
3. Project website
4. README
5. Releases