Muter: mutation testing that checks your Swift test suite

A score for the tests, not for the code

Muter is described as a mutation testing utility used to help you determine the quality of your test suite, and it is inspired by Stryker, PITTest and Mull, the mutation testing tools for JavaScript, Java and JVM languages. The idea it implements is a small one with a long history. The tool edits your source code in a way that breaks it on purpose, runs your test suite against the broken version, and checks whether the tests noticed.

The output is a mutation score, which the README defines as the ratio of the number of mutants your test suite killed versus the total number of mutants introduced. The formula in the documentation is written out plainly: mutation score = number of mutants killed / total number of mutants. The worked example is that a suite killing 50 of 75 introduced mutants scores 67%, and the README says a well-engineered test suite should strive to be as close to 100% as possible.

What the score is for is stated in three bullets. It finds gaps in fault coverage by identifying missing groups of tests, assertions or test cases. It determines whether the assertions you write are meaningful and effective, meaning they withstand different code than what the test was originally written against. And it assesses how many tests fail as a result of one code change, which is a different question from whether the suite is green.

That third use is the one that is not really about mutation testing at all, and it is worth separating. How many tests fail on a single change tells you how tightly your suite is coupled to the code, which is a coupling metric rather than a quality metric. The repository carries a separate document on how a mutation score differs from test code coverage, and the badge in the README shows a mutation score of 67 alongside a codecov badge, which is the project using its own tool on itself.

Mutants, operators, and the promise that nothing is left behind

The mechanism is a mutation operator. Muter introduces changes to your source code based on the logic contained in your app, and the changes it introduces are called mutants, which it generates using mutation operators. The list of available operators lives in a separate document in the repository, Docs/mutation_operators.md, and that file is the part to read first, because the score only means something relative to the set of mutations that were applied.

The safety property is stated in a note that deserves more attention than it usually gets. Muter does all of its work on a complete copy of your codebase, so it is not possible for it to accidentally leave anything behind. That is the practical difference between a mutation tool you can run against a working tree and one you can only run in a throwaway container: the copy is a copy of the whole project, not a patch applied to the checkout you are working in.

Scores are produced at two levels. Muter gives a mutation score for the entire test suite and also generates individual scores for the files it has mutated, which is what makes the report actionable. A file at zero means nothing in the suite distinguishes the correct implementation from a broken one. A file at a high score means the assertions there are load bearing.

The report format the repository ships is muter.report.txt, and the top level also holds muter.conf.yml, the configuration file. Those two files, plus the operator list, are the whole configuration surface a reader can see from the repository.

Three ways to run it, and the one built for CI

Muter offers three modes, and the README recommends a different one for each job. Inside Xcode it is for rapidly diagnosing areas where you can begin improving your test code, which is the fast, local, iterative use. From the command line it gives detailed information about the health and quality of your entire test suite. In CI it is meant for mutation testing projects incrementally, enabling per-commit updates on how code changes impact the quality of the test suite, with the output of that step connected into a dashboard or communication channel of choice.

The incremental CI invocation is the interesting one, and it is a single command.

muter --files-to-mutate $(echo \"$(git diff --name-only HEAD HEAD~1 | tr '\n' ',')\")

The shape of it is worth reading. The file list comes from a git diff between the current commit and its parent, the newlines are turned into commas so the list can be passed as one argument, and the result is handed to the --files-to-mutate flag. In other words, CI mutates only the files the commit touched, which is what makes mutation testing affordable on every push rather than as an occasional full-repo job.

Running from Xcode has its own setup, and it is small. You open Muter's scheme editor through Product, Scheme, Edit Scheme, select the Run scheme, and in the Options tab you enable Use custom working directory. Any command line arguments go in the Arguments tab under Arguments Passed on Launch, and the README notes this is also how you pass arguments on launch. After that the standard run command, the keyboard shortcut for Run, starts a mutation run against the current working directory.

Installing Muter, and building it yourself when you need the newest features

The packaged route is Homebrew, and it is one command.

brew install muter-mutation-testing/formulae/muter

The repository carries a homebrew-formulae directory at the top level, which is the tap this command pulls from.

Building from source is the route for people who want the latest features and improvements, and it is three commands.

git clone https://github.com/muter-mutation-testing/muter.git
cd muter
make install prefix=$(brew --prefix)

The prefix argument is the reason the command looks like that. The Makefile defaults prefix to /usr/local and derives bindir and libdir from it, and passing the Homebrew prefix installs the binary over the Homebrew-managed location rather than into the system path. The install target depends on build-release, which runs a release build of the muter product with the sandbox disabled, and then installs the built binary into bindir.

Going back to the Homebrew version is a documented two-step, and the README is explicit that installing from source over a Homebrew install replaces it.

make uninstall prefix=$(brew --prefix)
brew unlink muter && brew link muter

The project targets macOS and Linux, and the Swift badge in the README says 5.9. The build system is Swift Package Manager, and the Makefile has targets for the everyday work: build, build-release, install, uninstall, clean, run, test filtered to muterTests, acceptance-test and regression-test with their own shell scripts, and a mutation-test target that cleans and then runs muter on the project itself. The CI variants run the same scripts from a Scripts/ci directory.

Initial configuration, and where the documentation stops

Setting the tool up starts with one command in the root of your project directory. Running muter init there makes Muter take its best guess at a configuration that will work for your project, and the configuration it writes is muter.conf.yml.

That is where the README available here ends, and the truncation matters. The table of contents in the README lists further sections for Running Muter, Assumptions, Best Practices and FAQs, and the repository carries a Docs/ directory alongside the operator list and the mutation score versus coverage document, so the writing for those sections exists in the project even though it is not reproduced in the text. A reader who wants the full workflow should look in Docs/ for them.

What can be said from the repository layout is that this is a project with a serious testing culture of its own. The top level has AcceptanceTests/ and RegressionTests/ directories with their own runner scripts, a Tests/ directory, a Repositories/ directory, a .gitmodules file, and CI scripts under Scripts/ci for pull requests and for regression runs. The README also carries badges for regression tests, acceptance tests and code coverage pointing at gist-backed endpoints, and a Bitrise build badge.

The mutation testing workflow in that Makefile is the last piece worth noting. The mutation-test target runs clean and then invokes muter with no arguments, which means Muter runs against the project it is sitting in. The tool is applied to itself as part of the normal target list.

Where Muter is the wrong tool

Three cases point elsewhere. The first is a team in another language. Muter is a Swift tool, it builds with the Swift toolchain, its operator list is for Swift source, and the inspiration it names, Stryker, PITTest and Mull, are the equivalents for JavaScript, Java and the JVM. A Java shop should be running PITTest and should not expect a Swift binary to analyse it.

The second is a project that cannot afford the runtime. Mutation testing cost scales with the number of mutants and with how long the test suite takes to run, because each mutant needs a suite run against it. The incremental CI command exists precisely because the full run is expensive, and it is worth noticing that its scope is the files a commit changed, so a mutation score computed that way is not the same number as a full repository score, and mixing the two across commits will make the trend line meaningless.

The third is a team that wants a different question answered. Mutation testing asks whether the tests would catch a broken implementation. It does not tell you whether the tests are fast, whether they are readable, or whether the code is well designed. The README itself points at a document explaining how a mutation score differs from test code coverage, and that document exists because the two numbers are routinely confused.

The alternative worth comparing against is Stryker for JavaScript or PITest for Java, and the difference in approach is mostly in the operator set and the ecosystem integration rather than in the idea. Muter distinguishes itself by running inside Xcode for the local iteration loop, which the Java equivalents do not offer because they are not hosted in a Swift IDE, and by keeping the mutation score per file so the report points at a specific file rather than a project total.

Editorial conclusion

Muter fits a Swift team that has coverage numbers it does not trust, and that wants to know which assertions actually detect a broken implementation rather than which lines were executed. It does not fit a project that needs mutation testing for a language other than Swift, and it does not fit a team that cannot absorb the runtime of mutating a whole codebase on every CI run, since the cost scales with the number of mutants and the length of your test suite. Before you adopt it, read the operator list in Docs/mutation_operators.md, because the mutations it applies are the whole meaning of the score, and note that the newest tagged release is 16 from 2023-09-16 even though the last push to the repository was on 2026-07-21.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. muter-mutation-testing/muter on GitHub
4. README
5. Releases