cuckoo3: automated malware analysis that reports what a sample actually did

A sandbox that answers with a report, not an alert

Cuckoo3 is an open source tool to test suspicious files or links in a controlled environment. The description of what it does is short: it tests them in sandboxed platform emulators and generates a report showing what the files or websites did during the test.

That framing matters, because it is different from how endpoint detection products work. A sandbox answers a question about one artefact: you hand it a file or a URL, it runs, and you get back an account of what happened. The audience is an analyst or a researcher who has a suspicious sample and wants to know what it does, rather than a security operations centre watching a fleet.

The project is maintained under the cert.ee organisation, and the homepage is the Cuckoo Sandbox service at cuckoo-hatch.cert.ee, which also serves as an online instance you can look at before setting anything up locally.

The repository layout follows the pipeline. `core/` and `common/` hold the shared logic, `machineries/` holds the analysis machinery, `processing/` the report generation side, `node/` the sandbox nodes, `web/` the frontend, and `INSTALL/` holds the installation files including the Quickstart walkthrough and the project logos. There are also `docs/` and `docsrfc/` for documentation and design proposals, `devtools/` for developer tooling, `monitor.zip` and `signatures.zip` as packaged data, and `install.sh` at the root.

The licence is EUPL-1.2, the European Union Public Licence, with the text in the LICENSE file.

One scripted installer does the whole stack

The Quickstart script is the entry point for anyone evaluating the project. It is a single command:

curl -sSf https://cuckoo-hatch.cert.ee/static/install/quickstart | sudo bash

The documentation describes what it does in six steps: it creates a new non-sudo Cuckoo user, installs Cuckoo3 and VMCloak under that user, downloads and prepares the virtual machines, configures Cuckoo, installs UWSGI and Nginx, and serves the frontend using UWSGI and Nginx.

Two details in that list carry most of the operational meaning. Running the analysis under a dedicated unprivileged user, separate from the person who installed it, is the standard containment pattern for a sandbox. And the frontend is served by UWSGI behind Nginx, which means the web interface is a normal reverse proxied deployment rather than a development server.

The full list of actions is in the Quickstart walkthrough at INSTALL/QUICKSTART.md, and the deeper guides live in the documentation at cuckoo-hatch.cert.ee/static/docs.

Note the shape of the install. This is not a `pip install` into an existing virtualenv and it is not a Docker Compose file. It is a script that provisions a machine, pulls down virtual machine images and configures services. That is the right shape for a sandbox, since the sandboxes themselves are the expensive part of the setup, but it does mean the install is invasive and not trivially repeatable on a shared host.

The host runs Linux, the sandboxes run Windows

The README carries a warning at the top of the document, and it is the first constraint anyone evaluating this needs to internalise. You can currently only set up Cuckoo3 on Linux (Ubuntu) machines with Python 3.10, and you run Windows sandboxes.

So the split is: the control plane is Linux, the analysis targets are Windows virtual machines. That is the natural fit for the threat picture most people care about, since most malware that reaches an analyst was written for Windows, but it means the analyst's workstation has to be Ubuntu and the resource cost is dominated by running one or more Windows VMs.

The documentation links to a Cuckoo3 requirements page for the detail on what the host needs. Since a sandbox's whole value depends on the isolation of the virtual machines it runs samples in, the host requirements are the part to read first, not last.

The virtual machines are prepared by the installer using VMCloak, which the Quickstart script installs alongside Cuckoo3. The `machineries/` directory is where the analysis machinery lives, and `node/` is where the sandbox node components sit, which is the layer that talks to the virtual machines during an analysis run.

Python version requirements differ between the docs and the metadata

There is a discrepancy in the project worth flagging, because it will bite during setup.

The README states that you can set up Cuckoo3 on Linux machines with Python 3.10. The pyproject.toml at the root says `requires-python = ">=3.11"`, and the ruff configuration in the same file sets `target-version = "py310"`. The dependency list in the project metadata is empty, with the only declared groups being development tools, pyright and ruff.

An empty `dependencies` list does not mean the project has no runtime dependencies. It means the runtime requirements are not expressed in pyproject.toml at all, and are instead installed by the Quickstart script, which installs Cuckoo3 and VMCloak under the created user. That is a reasonable way to distribute a system that also provisions machines and services, but it does mean the packaging metadata is not a complete description of what gets installed.

The version mismatch itself may just be documentation lag in one direction or the other. The practical approach is to read the requirements page linked from the README, which is where the project says the authoritative detail lives, and to check it against the Python on the host before running the installer. An installer that provisions a user, a web stack and virtual machines is a poor place to discover a version conflict halfway through.

The tooling configuration itself is modern and conventional: ruff with a line length of 88 and a select list of the error and pyflakes rule families, pyright for type checking, and formatting that follows the Black conventions with double quotes and space indentation.

The project says out loud that it is not production ready

The README has a section headed IMPORTANT, and it says that this is not a production ready solution just yet, and that the maintainers highly advise against using it in a production environment.

That is unusual to see stated so plainly, and it is the single most useful thing in the document for an adopter. A sandbox that is reachable by untrusted users is itself an attack surface, and the project is telling you not to put it there.

The same section carries a warning about old Cuckoo related sites such as cuckoosandbox.org and others that have been neglected or compromised, and states that content is only served from the cert.ee domain and that the project is not affiliated with other domains containing Cuckoo content. That is a security warning about the ecosystem rather than the code, and it matters: a large number of Cuckoo-derived deployments have accumulated over the years and the fork lineage is genuinely confusing.

The project also points at a roadmap on the cert-ee organisation's project board for what is planned next.

Taken together, the picture is of a system that is actively developed, documented, and honest about its current limits. That combination is rarer than it should be, and it is the main reason to consider the project even if the immediate plan is only an internal analysis capability.

Where a sandbox is the wrong tool

Three situations call for something else.

First, if the question is not about a single artefact's behaviour, a sandbox is the wrong instrument. Sandbox reports are about what one file did in one run in one environment. A sample that behaves differently because of anti-analysis timing, or that only acts on a specific target, will produce a report that describes a behaviour that does not generalise.

Second, if you need to protect endpoints in real time. Sandbox analysis is inherently retrospective and slow relative to what an endpoint protection agent does at file open time. If the requirement is blocking or alerting at the moment of execution, that is a different category of product.

Third, if you cannot isolate properly. A sandbox whose virtual machines are not genuinely disposable is worse than no sandbox, because it produces confident reports from a compromised environment. The VMCloak based setup in the Quickstart exists to address exactly this, and if your host cannot provide real isolation the honest answer is to use a hosted analysis service instead of your own installation.

The alternative worth naming is the online instance the project runs itself at cuckoo-hatch.cert.ee. The difference in approach is where the work happens: a hosted service runs your sample on someone else's isolated infrastructure, while a local installation keeps the analysis inside your own perimeter and lets you control the virtual machine images, which is what you need when you are analysing something sensitive or need to inspect the environment yourself.

For a first look, using the hosted instance before running the installer on your own hardware is the sensible order of operations.

Editorial conclusion

Adopt cuckoo3 if you need a private sandbox that reports what a suspicious file or link did during execution, and you have a Linux host with the resources to run disposable virtual machines. Do not put it in front of untrusted users on a production network, because the project states plainly that it is not production ready. Verify first which Python version your host has, since the documentation says 3.10 while the project metadata requires 3.11 or newer, and check that your host can run KVM-backed virtual machines before planning anything beyond a single analyst's setup.

Frequently asked questions

Official sources

1. cert-ee/cuckoo3 on GitHub
2. License: EUPL-1.2
3. Project website
4. README
5. Releases