Binary Refinery: turning malware triage into piped command line units

The unit model: one job each, stdin to stdout, composed with a pipe

The core idea in the README is simple to state and unusual to meet in practice. The Binary Refinery is a collection of Python scripts that implement transformations of binary data such as compression and encryption. Every script is designed to read input exclusively from stdin and write output to stdout. The main philosophy is that every script does one job, and individual units can be combined into pipelines with the piping operator on the command line to perform more complex tasks.

That constraint is what makes the toolkit composable. A pipeline of three units reads as a single expression of what happened to the sample, and each stage can be swapped without rewriting the rest. The project calls its main focus malware triage, and describes itself as an attempt to implement something like CyberChef on the command line.

The scripts are called units, and the package is named refinery. There is a binref command for finding units, and a full-text search over the description and help text of every unit is available from that command. For a toolkit whose value is the breadth of its transformations, search is not a convenience feature, it is the primary navigation method.

The repository also ships samples, a tutorials directory with notebooks including one on TBR files and a dropper, and a shells/ directory of command line support scripts.

Framing, multibin arguments and meta variables

A pipe-based toolkit faces a problem that a library does not: binary data has no natural delimiter, so a unit reading raw bytes from stdin cannot always tell where one item ends and the next begins. Binary Refinery solves this with the concept of framing, and the README lists framing as one of the three fundamental concepts of the toolkit, alongside multibin arguments and meta variables.

Framing is the mechanism that decides how a stream of bytes is divided into the units a unit expects to work on, which matters when the input is not a single file but a container of several parts. Multibin arguments extend the usual command line argument model so that a single argument can refer to more than one binary blob rather than exactly one file path.

Meta variables are the third piece, and they are the part that makes pipelines reusable. Rather than hardcoding an intermediate file on disk, a unit can emit a variable into the stream that a later unit in the same pipeline can refer to.

The automatically generated documentation contains the specification for these three concepts at the top level, and the README says the documentation also compiles the output of the -h or --help switch for each unit. The README is candid that reference documentation can be somewhat dry and recommends the tutorials instead. That recommendation matters: framing and meta variables are exactly the kind of ideas that are easier to pick up from a worked example than from a specification.

Installing into a virtual environment, and why that warning matters

The shortest path in the README asks for a Python virtual environment and Python 3.11 or later, then two pip commands:

python -m pip install -U pip
python -m pip install -U binary-refinery[extended]

The extended extra pulls in more than the plain package. The build system declared in pyproject.toml lists the toolchain the project compiles against, which shows the kinds of libraries the units rely on:

requires = [
    "cython",
    "colorama",
    "defusedxml",
    "lief~=0.17.0",
    "msgpack>=1.0.2",
    "pycryptodomex",
    "pure-magic-rs",
    "setuptools",
    "toml",
    "wheel",
]

LIEF for binary parsing, pycryptodomex for the cryptography units, pure-magic-rs for file type detection, msgpack for serialisation and defusedxml to keep XML parsing safe. The setup.py file confirms the floor on the interpreter, and it also reads the prefix from the environment at build time:

__prefix__ = os.getenv('REFINERY_PREFIX') or ''
__minver__ = '3.11'

The README is unusually insistent about isolating the installation. The package can pull in a lot of dependencies, installing it into a global Python is somewhat prone to version conflicts, and since the toolkit introduces a large number of new commands there is a good chance that some of them will clash with commands already on the system. Keeping them in their own environment is the way to prevent that. The setup.py source also shows why a Windows-specific build workaround exists: Cython extension modules get locked by the operating system when the setup imports them, so the custom build command renames the locked files before building.

Command prefixes and the REFINE-prefix environment variable

Because refinery installs a large number of commands, there is a second decision to make at install time. The README offers a prefix option that gets put in front of every command shim that is installed, and the example given is the prefix r., under which the emit unit is installed as the command r.emit. There is a practical side benefit: typing r. and pressing Tab twice lists every available refinery command.

The prefix is set through the REFINERY_PREFIX environment variable rather than a command line flag. On Linux the README shows it inlined with the install command:

REFINERY_PREFIX=r. pip install -U binary-refinery

On Windows the equivalent is a separate set command followed by the install:

set REFINERY_PREFIX=r.
pip install -U binary-refinery

One prefix value is special. The prefix ! has the effect that no shell commands are created at all, and refinery is installed only as a library. That is the option for a system where the commands would be noise, such as a shared analysis host.

The README notes that no prefix is assumed in the documentation and that avoiding clashes on most systems is a development goal of the project, with the prefix offered as a safety blanket. The author does not use one. If you are writing a tutorial or a runbook for other analysts, that decision is yours to make, but pick it deliberately, because the prefix is baked into the installed command names and changing it later means reinstalling.

Finding units: help text, the binref command and the tutorials

The documentation strategy is unusual and worth understanding before you start. The README states that the help text displayed when you run a unit with -h or --help is its main documentation. The automatically generated documentation is a compilation of that output for every unit, collected at the top level, plus the specification of the three fundamental concepts.

So the reference text and the runtime behaviour come from the same source. What a unit prints when you ask for help is what the documentation says, which means it cannot drift out of date in the way a separate manual can.

The README also points to a series of tutorials as an ongoing effort, and recommends checking them out. The repository listing shows a tutorials/ directory and the README names the first tutorial, a notebook on TBR files and a dropper. There is a public video walkthrough too, and the README references blog posts and conference streams where the toolkit has been demonstrated on real samples.

One detail in that list is worth remembering for anyone following older write-ups. The README notes that refinery is still in alpha and the interface can sometimes change, with units and parameters able to be removed or renamed, and states that specific command lines from older videos and blog posts may not work any more. When a tutorial from an earlier year fails on a parameter, the tutorial is the likely reason rather than the installation.

Release rhythm, documentation gaps and what alpha means here

The release policy is stated in two sentences and they pull in different directions. There is no release schedule, but releases happen very frequently and updating periodically is recommended. Bugfixes are not documented outside of git, while all other changes, meaning new features, are documented in the CHANGELOG.md file at the root of the repository.

That gap is a practical hazard. If a unit starts behaving differently, the changelog will not tell you why, and the commit history becomes the only record. The repository has no GitHub releases, so there is no tag-based changelog either, which means the pip package version is the only version marker you can rely on.

The project itself flags the alpha status, and the README gives the concrete consequences: the interface can change, units and parameters can be removed or renamed, and older command lines may stop working. For an analyst this is tolerable, because a broken command line is an inconvenience during a session. For anyone scripting refinery into a longer-lived pipeline, it is a different proposition, and the absence of a release schedule makes it harder to pin a version that you know will keep working.

The activity itself is current. The last push to the repository was on 2026-10-12, and the tree carries a pytest.ini, a codecov.yml and a .coveragerc, so the project is built with continuous integration and coverage measurement in view rather than being a script collection maintained by hand.

Limits: dependency weight, command collisions and single-purpose units

Three constraints follow directly from the design.

The first is dependency weight. The README warns that the package can pull in a lot of dependencies, that installing into a global Python is prone to version conflicts, and that the large number of new commands creates a real chance of clashing with existing ones. On a shared analyst workstation, or in a container image that also carries other tooling, the command shims are as much of a problem as the Python packages.

The second is that units are single-purpose by design, which is a strength for reuse and a weakness for a task that needs several steps done atomically. If the middle of a pipeline needs a decision based on the content of the previous stage, that logic has to live somewhere other than the pipe. Nothing in the README describes a conditional or branching facility, so the practical pattern is to run the pipeline, inspect the output, and then run a second pipeline.

The third is Windows-specific friction around the compiled extensions, which the setup.py workaround documents in detail: importing the modules during setup loads the Cython extensions, the operating system locks them, and the default in-place build cannot overwrite a locked file. The workaround renames the existing files first, which Windows permits even for locked files, and cleans them up afterwards. It is a narrow problem with a targeted fix, but it is the kind of detail that only surfaces on the platform that has it.

Finally, the toolkit does not replace an analyst. It transforms bytes; deciding which transformation to apply, and whether the output means anything, is still the work.

Binary Refinery against CyberChef on the command line

The README names the comparison itself: the project is an attempt to implement something like CyberChef on the command line. The difference in approach is worth stating precisely, because it determines when each one fits.

CyberChef is a recipe-driven environment where operations are assembled into a chain and executed in the browser, with the data moving between operation boxes. Refinery expresses the same idea as a shell pipeline, where the operations are separate programs joined by the pipe operator and the data flows through standard streams. Two consequences follow. First, a refinery pipeline composes with everything else on the machine: you can put any other command at either end of it, redirect the output to a file, or feed it into another pipeline. Second, because the units are individual Python scripts, they can be called from Python code, which the ! prefix makes explicit by installing the toolkit as a library with no shell commands at all.

The trade-off runs the other way as well. The browser model gives you a visual overview of a recipe and a point-and-click workflow. A command line gives you a text record that can be pasted into a ticket, version controlled, and re-run identically, and it lets the operating system's process handling carry the data instead of an in-page buffer.

The difference in maturity is worth weighing too. CyberChef is a browser application with a familiar interface, while this toolkit is in alpha and may rename its units. For a team that wants shared, repeatable triage recipes with a stable interface today, the browser tool is the safer bet. For someone who lives in a terminal, works on a REMnux host or in a container, and wants to compose triage with the rest of a shell pipeline, refinery is the more natural fit.

Editorial conclusion

Adopt Binary Refinery when you triage samples on the command line and want repeatable, composable transformations instead of a GUI recipe, particularly in a REMnux or containerised workflow. Do not adopt it for a pipeline that needs a stable interface, because the README states it is still in alpha and that units and parameters can be removed or renamed. Verify first that you install it into its own virtual environment, that your Python is 3.11 or later, and that the units you plan to depend on still carry the names you saw in an older write-up.

Frequently asked questions

Official sources

1. binref/refinery on GitHub
2. Issues
3. README