cosmic-ray: mutation testing that runs your suite against broken code

Coverage says the code ran, not that the test checked

Cosmic Ray is a mutation testing tool for Python 3, and its job is to answer a question line coverage cannot. Coverage tells you that a line executed during the test run. It does not tell you whether the assertion on that line would have failed had the code been wrong.

The mechanism is deliberately blunt. Cosmic Ray makes small changes to your source code and runs your test suite for each one. If a mutation survives, meaning the test suite still passes with the deliberately broken code, that test was not covering that behaviour. The README shows what the mutations look like in an animated image in the documentation, which is a reasonable way to teach the idea since a mutation is easiest to understand when you watch one appear in a diff.

So the output is a number with a meaning: the mutation score, the share of mutants your tests kill. A project with 90 percent line coverage and a low mutation score has tests that execute a lot of code and assert very little about it.

The audience is developers who already have a test suite and want to measure its quality rather than its coverage. It is not a coverage tool, not a test generator and not a way to find bugs on its own. A tool like this is only as good as the suite it mutates, which is why the README's own contribution guidance is to use Cosmic Ray and submit reports for defects you come across, rather than to write new mutators first.

A console entry point, then a family of reporting tools

The packaging is the clearest description of what the tool does. The project scripts section registers one main command and a series of specialised ones:

[project.scripts]
cosmic-ray = "cosmic_ray.cli:main"
cr-html = "cosmic_ray.tools.html:report_html"
cr-report = "cosmic_ray.tools.report:report"
cr-badge = "cosmic_ray.tools.badge:generate_badge"
cr-rate = "cosmic_ray.tools.survival_rate:format_survival_rate"
cr-xml = "cosmic_ray.tools.xml:report_xml"

The single cosmic-ray command is the runner, driven by click, which is why the tool is a console application rather than a library you call. Everything else is a separate command that reads the results of a run and turns them into something else: an HTML report, a text report, a badge, a formatted survival rate, or XML for a CI system to consume.

The badge command is the interesting one for a team that wants to make mutation testing part of a pipeline without blocking anyone. A badge is the same shape as a coverage badge, and the survival rate formatter produces the figure it displays. The XML output is what a CI job would archive or parse, and the filter commands, cr-filter-operators, cr-filter-pragma, cr-filter-git and cr-filter-lines, exist to narrow a run before or after the fact: by operator, by pragma, by git revision and by line.

That set of tools is a design statement. Cosmic Ray is not only a mutator and a runner, it is a reporting pipeline whose output is meant to end up in a dashboard, a badge in a README, or a CI artifact.

Operators are plugins, loaded through an entry point group

The project registers a second entry point group, cosmic_ray.operator_providers, and that single line explains most of the architecture.

An operator is the thing that makes a change. If the set of mutations is a plugin boundary, then the tool's core is small: a workspace, a scheduler, a runner that executes the test suite, and a comparison of the result against the baseline. Everything specific to a mutation lives in a provider.

The dependency list supports that reading. The package is built on parso, a Python parser, which is what makes source level mutation possible rather than text level, and on attrs for the declarative data classes that describe mutants and their results. gitpython appears because the filters include one that works by git revision, which tells you the tool models the source tree as a set of revisions rather than as a single snapshot.

The remaining dependencies describe the shell around the core. click provides the command line, rich provides the terminal output, sqlalchemy provides the storage for results, aiohttp backs the http-workers command, stevedore is the plugin loader that reads the entry points, anybadge generates badges, yattag writes the HTML report, toml handles configuration and exit_codes manages process exit behaviour.

Two things follow for an adopter. The first is that adding a mutation type means writing a provider and registering it, which is an extension point rather than a patch. The second is that the runtime dependencies are not minimal, so Cosmic Ray is a development-time tool and should not end up in a production dependency tree, even though it is installable as a normal package.

Where mutation testing is the wrong investment

The cost model is the whole trade-off, and it is arithmetic rather than philosophical. Cosmic Ray runs your test suite once per mutant. A project with a fast suite and a hundred mutants finishes in minutes. A project with a suite that takes twenty minutes and five hundred mutants takes a week of continuous integration, and the result is a number that is a week out of date every time you run it.

That makes the tool a poor first move in a project whose test suite is slow or flaky, because a flaky suite produces meaningless mutation results. A mutant that appears to survive might have failed for an unrelated reason, and the false signal is worse than no signal, since someone will act on it.

There is also a scope limit that is easy to miss. Mutation testing measures the tests, not the code. It does not find the bug you shipped, because a bug that no test targets and no mutation happens to resemble will not show up as a surviving mutant in a meaningful way. It tells you where your tests are blind, which is valuable, and it is not the same as finding defects.

The second limitation is operational. A mutation score is a number, and a number invites a target. Once a team sets a threshold, the temptation is to write tests that kill mutants rather than tests that describe behaviour, and mutation score is then measuring the wrong thing. The README does not discuss this, so the guard against it is a team decision about how the number is used, not a feature of the tool.

The third is that Python only, at version 3.9 or later, and the project is a console tool rather than a library, so there is no way to embed a mutation run inside another harness.

Packaging, licence and a project that expects contributors to use it

The project is packaged with hatchling and hatch-vcs, which means the version is derived from the source control tags rather than written by hand. The dynamic version field and the hatch-vcs build requirement are the visible parts of that, and it is consistent with the release tags, which are named release/v8.7.0 and release/v8.4.6.

Those two releases are dated 2026-08-09 and 2026-04-02, so the gap between them is about four months, and the last push to the repository was on 2026-10-15. The repository is not archived, and the project describes itself as Development Status 4, Beta, in its classifiers while still shipping releases and CI badges for every supported Python version from 3.9 to 3.13.

The licence is MIT, recorded in the LICENCE.txt file at the root and confirmed by the classifier. The metadata also names Sixty North AS as the author, and the file layout has the markers of a company-maintained open source project: a .pylintrc, a .coveragerc, a code health configuration, a cliff.toml for generating the changelog, an AGENTS.md, a deploy_key.enc for the release pipeline, a readthedocs configuration, and a uv.lock pinning the development environment.

The contributing guidance is unusual and worth repeating. The easiest way to contribute is to use Cosmic Ray and submit reports for defects or any other issues you come across, with more detail in CONTRIBUTING.rst. That is a deliberate request for dogfooding reports rather than for new mutators, and it tells you where the maintainers think the remaining gaps are.

The documentation itself lives on readthedocs, and the README's own contribution to it is a pointer to the full documentation, with the badge in the header linking to the hosted docs.

Editorial conclusion

Adopt cosmic-ray when you already have a fast, deterministic test suite and want to know which of your tests would actually catch a regression, since that is the only situation where mutation scores mean something. Do not adopt it as a first step in a project whose test suite is slow or flaky, because the tool multiplies the cost of the suite by the number of mutants. Verify first how long a single test run takes in your project, since the whole design scales with that number, and check that the operators you care about are actually available through the operator_providers entry point before you plan a campaign around them.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. Releases
5. sixty-north/cosmic-ray on GitHub