Welcome to cargo-mutants

cargo-mutants is a mutation testing tool for Rust. It helps you improve your program's quality by finding places where bugs can be inserted causing any tests to fail.

The goal of cargo-mutants is to be easy to run on any Rust source tree, and to tell you something interesting about areas where bugs might be lurking or the tests might be insufficient. (More about these goals.)

To get started:

1. Install cargo-mutants.
2. Run `cargo mutants in your Rust source tree.

For more resources see the repository at https://github.com/sourcefrog/cargo-mutants.

Installation

Install cargo-mutants from source:

cargo install --locked cargo-mutants

You can also use cargo binstall from cargo-binstall, or install binaries from GitHub releases.

Supported Rust versions

Building cargo-mutants requires a reasonably recent stable (or nightly or beta) Rust toolchain. The supported version is visible in https://crates.io/crates/cargo-mutants.

After installing cargo-mutants, you should be able to use it to run tests under any toolchain, even toolchains that are far too old to build cargo-mutants, using the standard + option to cargo:

cargo +1.48 mutants

Getting started

Just run cargo mutants in a Rust source directory, and it will point out functions that may be inadequately tested.

Prerequisites

For cargo-mutants to give useful results, your tree must already

1. Be built with cargo build, and
2. Have reliable non-flaky tests that run under either cargo test or cargo nextest.

If the tests are flaky, meaning that they can pass or fail depending on factors other than the source tree, then the cargo-mutants results will be meaningless.

Cross-compilation is not currently supported, so the tree must be buildable for the host platform.

Example

; cargo mutants
Found 14 mutants to test
Copy source to scratch directory ... 0 MB in 0.0s
Unmutated baseline ... ok in 1.6s build + 0.3s test
Auto-set test timeout to 20.0s
src/lib.rs:386: replace <impl Error for Error>::source -> Option<&(dyn std::error::Error + 'static)>
 with Default::default() ... NOT CAUGHT in 0.6s build + 0.3s test
src/lib.rs:485: replace copy_symlink -> Result<()> with Ok(Default::default()) ...
 NOT CAUGHT in 0.5s build + 0.3s test
14 mutants tested in 0:08: 2 missed, 9 caught, 3 unviable

In v0.5.1 of the cp_r crate, the copy_symlink function was reached by a test but not adequately tested, and the Error::source function was not tested at all.

Using the results

Tests fail in an clean tree?

If tests fail in a clean copy of the tree, there might be an (intermittent) failure in the source directory, or there might be some problem that stops them passing when run from a different location. Fix this first: cargo-mutants can't do anything until you have a tree where cargo test passes reliably when copied to a temporary directory.

Mutant outcomes

Assuming tests pass in a clean copy of the tree, cargo-mutants proceeds to generate every mutant it can, subject to any configured filters, and then runs cargo build and cargo test on each of them.

Each mutant results in one of the following outcomes:

• caught — A test failed with this mutant applied. This is a good sign about test coverage.

• missed — No test failed with this mutation applied, which seems to indicate a gap in test coverage. Or, it may be that the mutant is undistinguishable from the correct code. You may wish to add a better test, or mark that the function should be skipped.

• unviable — The attempted mutation doesn't compile. This is inconclusive about test coverage and no action is needed, but indicates an opportunity for cargo-mutants to either generate better mutants, or at least not generate unviable mutants.

• timeout — The mutation caused the test suite to run for a long time, until it was eventually killed. You might want to investigate the cause and potentially mark the function to be skipped.

By default only missed mutants and timeouts are printed to stdout, because they're the most actionable. Others can be shown with the --caught and --unviable options.

What to do about missed mutants?

Each missed mutant is a sign that there might be a gap in test coverage. What to do about them is up to you, bearing in mind your goals and priorities for your project, but here are some suggestions:

First, look at the overall list of missed mutants: there might be patterns such as a cluster of related functions all having missed mutants. Probably some will stand out as potentially more important to the correct function of your program.

You should first look for any mutations where it's very surprising that they were not caught by any tests, given what you know about the codebase. For example, if cargo-mutants reports that replacing an important function with Ok(()) is not caught then that seems important to investigate.

You should then look at the tests that you would think would catch the mutant: that might be unit tests within the relevant module, or some higher-level public-API or integration test, depending on how your project's tests are structured.

If you can't find any tests that you think should have caught the mutant, then perhaps you should add some. The right thing here is not necessarily to directly assert that the mutated behavior doesn't happen. For example, if the mutant changed a private function, you don't necessarily want to add a test for that private function, but instead ask yourself what public-API behavior would break if the private function was buggy, and then add a test for that.

Try to avoid writing tests that are too tightly targeted to the mutant, which is really just an example of something that could be wrong, and instead write tests that assert the correct behavior at the right level of abstraction, preferably through a public interface.

If it's not clear why the tests aren't already failing, it may help to manually inject the same mutation into your working tree and then run the tests under a debugger, or add trace statements to the test. (The --diff option or looking in the mutants.out directory will show you exactly what change cargo-mutants made.)

You may notice some messages about missed mutants in functions that you feel are not very important to test, such as Debug implementations. You can use the --exclude-re options to filter out these mutants, or mark them as skipped with #[mutants::skip]. (Or, you might decide that you do want to add unit tests for the Debug representation, but perhaps as a lower priority than investigating mutants in more important code.)

In some cases cargo-mutants will generate a mutant that is effectively the same as the original code, and so not really incorrect. cargo-mutants tries to avoid doing this, but if it does happen then you can mark the function as skipped.

Iterating on mutant coverage

After you've changed your program to address some of the missed mutants, you can run cargo mutants again with the --file option to re-test only functions from the changed files.

Hard-to-test cases

Some functions don't cause a test suite failure if emptied, but also cannot be removed. For example, functions to do with managing caches or that have other performance side effects.

Ideally, these should be tested, but doing so in a way that's not flaky can be difficult. cargo-mutants can help in a few ways:

• It helps to at least highlight to the developer that the function is not covered by tests, and so should perhaps be treated with extra care, or tested manually.
• A #[mutants::skip] annotation can be added to suppress warnings and explain the decision.
• Sometimes these effects can be tested by making the side-effect observable with, for example, a counter of the number of memory allocations or cache misses/hits.

Hangs and timeouts

Some mutations to the tree can cause the test suite to hang. For example, in this code, cargo-mutants might try changing should_stop to always return false, but this will cause the program to hang:

#![allow(unused)]
fn main() {
    while !should_stop() {
      // something
    }
}

In general you will want to skip functions which cause a hang when mutated, either by marking them with an attribute or in the configuration file.

Timeouts

To avoid hangs, cargo-mutants will kill the test suite after a timeout and continue to the next mutant.

By default, the timeout is set automatically. cargo-mutants measures the time to run the test suite in the unmodified tree, and then sets a timeout for mutated tests at 5x the time to run tests with no mutations, and a minimum of 20 seconds.

The minimum of 20 seconds can be overridden by the CARGO_MUTANTS_MINIMUM_TEST_TIMEOUT environment variable, measured in seconds.

You can also set an explicit timeout with the --timeout option, also measured in seconds. If this option is specified then the timeout is also applied to the unmutated tests.

The timeout does not apply to cargo check or cargo build, only cargo test.

Exit codes

cargo-mutants returns an exit code that can be used by scripts or CI.

• 0: Success! Every viable mutant that was tested was caught by a test.

• 1: Usage error: bad command-line arguments etc.

• 2: Found some mutants that were not covered by tests.

• 3: Some tests timed out: possibly the mutatations caused an infinite loop, or the timeout is too low.

• 4: The tests are already failing or hanging before any mutations are applied, so no mutations were tested.

For more detailed machine-readable information, use the mutants.out directory.

The mutants.out directory

A mutants.out directory is created in the original source directory. You can put the output directory elsewhere with the --output option.

On each run, any existing mutants.out is renamed to mutants.out.old, and any existing mutants.out.old is deleted.

The output directory contains:

• A lock.json, on which an fs2 lock is held while cargo-mutants is running, to avoid two tasks trying to write to the same directory at the same time. lock.json contains the start time, cargo-mutants version, username, and hostname. lock.json is left in mutants.out when the run completes, but the lock on it is released.

• A mutants.json file describing all the generated mutants. This file is completely written before testing begins.

• An outcomes.json file describing the results of all tests, and summary counts of each outcome.

• A logs/ directory, with one log file for each mutation plus the baseline unmutated case. The log contains the diff of the mutation plus the output from cargo. outcomes.json includes for each mutant the name of the log file.

• caught.txt, missed.txt, timeout.txt, unviable.txt, each listing mutants with the corresponding outcome.

The contents of the directory and the format of these files is subject to change in future versions.

These files are incrementally updated while cargo-mutants runs, so other programs can read them to follow progress.

There is generally no reason to include this directory in version control, so it is recommended that you add /mutants.out* to your .gitignore file. This will exclude both mutants.out and mutants.out.old.

Skipping untestable code

Some functions may be inherently hard to cover with tests, for example if:

• Generated mutants cause tests to hang.
• You've chosen to test the functionality by human inspection or some higher-level integration tests.
• The function has side effects or performance characteristics that are hard to test.
• You've decided the function is not important to test.

There are three ways to skip mutating some code:

1. Marking the function with an attribute within the source file.
2. Filtering by path in the config file or command line.
3. Filtering by function and mutant name in the config file or command line.

The results of all these filters can be previewed using the --list option.

Which filtering method to use?

• If some particular functions are hard to test with cargo-mutants, use an attribute, so that the skip is visible in the code.
• If a whole module is untestable, use a filter by path in the config file, so that the filter's stored in the source tree and covers any new code in that module.
• If you want to permanently ignore a class of functions, such as Debug implementations, use a regex filter in the config file.
• If you want to run cargo-mutants just once, focusing on a subset of files, functions, or mutants, use command line options to filter by name or path.

Skipping functions with an attribute

To mark functions as skipped, so they are not mutated:

1. Add a Cargo dependency on the mutants crate, version "0.0.3" or later. (This must be a regular dependency not a dev-dependency, because the annotation will be on non-test code.)

2. Mark functions with #[mutants::skip] or other attributes containing mutants::skip (e.g. #[cfg_attr(test, mutants::skip)]).

The mutants create is tiny and the attribute has no effect on the compiled code. It only flags the function for cargo-mutants. However, you can avoid the dependency by using the slightly longer #[cfg_attr(test, mutants::skip)] form.

Note: Currently, cargo-mutants does not (yet) evaluate attributes like cfg_attr, it only looks for the sequence mutants::skip in the attribute.

You may want to also add a comment explaining why the function is skipped.

For example:

#![allow(unused)]
fn main() {
use std::time::{Duration, Instant};

/// Returns true if the program should stop
#[cfg_attr(test, mutants::skip)] // Returning false would cause a hang
fn should_stop() -> bool {
    true
}

pub fn controlled_loop() {
    let start = Instant::now();
    for i in 0.. {
        println!("{}", i);
        if should_stop() {
            break;
        }
        if start.elapsed() > Duration::from_secs(60 * 5) {
            panic!("timed out");
        }
    }
}

mod test {
    #[test]
    fn controlled_loop_terminates() {
        super::controlled_loop()
    }
}
}

Filtering files

Two options (each with short and long names) control which files are mutated:

• -f GLOB, --file GLOB: Mutate only functions in files matching the glob.

• -e GLOB, --exclude GLOB: Exclude files that match the glob.

These options may be repeated.

If any -f options are given, only source files that match are considered; otherwise all files are considered. This list is then further reduced by exclusions.

If the glob contains / (or on Windows, \), then it matches against the path from the root of the source tree. For example, src/*/*.rs will exclude all files in subdirectories of src.

If the glob does not contain a path separator, it matches against filenames in any directory. / matches the path separator on both Unix and Windows.

Note that the glob must contain .rs (or a matching wildcard) to match source files with that suffix. For example, -f network will match src/network/mod.rs but it will not match src/network.rs.

Files that are excluded are still parsed (and so must be syntactically valid), and mod statements in them are followed to discover other source files. So, for example, you can exclude src/main.rs but still test mutants in other files referenced by mod statements in main.rs.

Since Rust does not currently allow attributes such as #[mutants::skip] on mod statements or at module scope filtering by filename is the only way to skip an entire module.

The results of filters can be previewed with the --list-files and --list options.

Examples:

• cargo mutants -f visit.rs -f change.rs -- test mutants only in files called visit.rs or change.rs (in any directory).

• cargo mutants -e console.rs -- test mutants in any file except console.rs.

• cargo mutants -f src/db/*.rs -- test mutants in any file in this directory.

Configuring filters by filename

Files may also be filtered with the exclude_globs and examine_globs options in .cargo/mutants.toml.

Exclusions in the config file may be particularly useful when there are modules that are inherently hard to automatically test, and the project has made a decision to accept lower test coverage for them.

From cargo-mutants 23.11.2 onwards, if the command line options are given then the corresponding config file option is ignored. This allows you to use the config file to test files that are normally expected to pass, and then to use the command line to test files that are not yet passing.

For example:

exclude_globs = ["src/main.rs", "src/cache/*.rs"] # like -e
examine_globs = ["src/important/*.rs"] # like -f: test *only* these files

Filtering functions and mutants

You can filter mutants by name, using the --re and --exclude-re command line options and the corresponding examine_re and exclude_re config file options.

These options are useful if you want to run cargo-mutants just once, focusing on a subset of functions or mutants.

These options filter mutants by the full name of the mutant, which includes the function name, file name, and a description of the change, as shown in the output of cargo mutants --list.

For example, one mutant name might be:

src/outcome.rs:157: replace <impl Serialize for ScenarioOutcome>::serialize -> Result<S::Ok, S::Error> with Ok(Default::default())

Within this name, your regex can match any substring, including for example:

• The filename
• The trait, impl Serialize
• The struct name, ScenarioOutcome
• The function name, serialize
• The mutated return value, with Ok(Default::default()), or any part of it.

The regex matches a substring, but can be anchored with ^ and $ to require that it match the whole name.

The regex syntax is defined by the regex crate.

These filters are applied after filtering by filename, and --re is applied before --exclude-re.

Examples:

• -E 'impl Debug' -- don't test impl Debug methods, because coverage of them might be considered unimportant.

• -F 'impl Serialize' -F 'impl Deserialize' -- test implementations of these two traits.

Configuring filters by name

Mutants can be filtered by name in the .cargo/mutants.toml file. The exclude_re and examine_re keys are each a list of strings.

This can be helpful if you want to systematically skip testing implementations of certain traits, or functions with certain names.

From cargo-mutants 23.11.2 onwards, if the command line options are given then the corresponding config file option is ignored.

For example:

exclude_re = ["impl Debug"] # same as -E

Controlling cargo-mutants

cargo mutants takes various options to control how it runs.

These options, can, in general, be passed on the command line, set in a .cargo/mutants.toml file in the source tree, or passed in CARGO_MUTANTS_ environment variables. Not every method of setting an option is available for every option, however, as some would not make sense or be useful.

For options that take a list of values, values from the configuration file are appended to values from the command line.

For options that take a single value, the value from the command line takes precedence.

--no-config can be used to disable reading the configuration file.

Execution order

By default, mutants are run in a randomized order, so as to surface results from different parts of the codebase earlier. This can be disabled with --no-shuffle, in which case mutants will run in order by file name and within each file in the order they appear in the source.

Source directory location

-d, --dir: Test the Rust tree in the given directory, rather than the source tree enclosing the working directory where cargo-mutants is launched.

Console output

-v, --caught: Also print mutants that were caught by tests.

-V, --unviable: Also print mutants that failed cargo build.

--no-times: Don't print elapsed times.

-L, --level, and $CARGO_MUTANTS_TRACE_LEVEL: set the verbosity of trace output to stdout. The default is info, and it can be increased to debug or trace.

Listing generated mutants

--list: Show what mutants could be generated, without running them.

--diff: With --list, also include a diff of the source change for each mutant.

--json: With --list, show the list in json for easier processing by other programs. (The same format is written to mutants.out/mutants.json when running tests.)

--check: Run cargo check on all generated mutants to find out which ones are viable, but don't actually run the tests. (This is primarily useful when debugging cargo-mutants.)

Workspace and package support

cargo-mutants supports testing Cargo workspaces that contain multiple packages. The entire workspace tree is copied.

By default, cargo-mutants has the same behavior as Cargo:

• If --workspace is given, all packages in the workspace are tested.
• If --package is given, the named packages are tested.
• If the starting directory (or -d directory) is in a package, that package is tested.
• Otherwise, the starting directory must be in a virtual workspace. If it specifies default members, they are tested. Otherwise, all packages are tested.

For each mutant, only the containing package's tests are run, on the theory that each package's tests are responsible for testing the package's code.

The baseline tests exercise all and only the packages for which mutants will be generated.

You can also use the --file options to restrict cargo-mutants to testing only files from some subdirectory, e.g. with -f "utils/**/*.rs". (Remember to quote globs on the command line, so that the shell doesn't expand them.) You can use --list or --list-files to preview the effect of filters.

Passing options to Cargo

cargo-mutants runs cargo build and cargo test (or, with --check, it runs cargo check.) Additional options can be passed in three different ways: to all cargo commands; to cargo test only; and to the test binaries run by cargo test.

There is not yet a way to pass options only to cargo build but not to cargo test.

Arguments to all cargo commands

To pass more arguments to every Cargo invocation, use --cargo-arg, or the additional_cargo_args configuration key. --cargo-arg can be repeated.

For example

cargo mutants -- --cargo-arg=--release

or in .cargo/mutants.toml:

additional_cargo_args = ["--all-features"]

Arguments to cargo test

Command-line options following a -- delimiter are passed through to cargo test (or to nextest, if you're using that).

For example, this can be used to pass --all-targets which (unobviously) excludes doctests. (If the doctests are numerous and slow, and not relied upon to catch bugs, this can improve performance.)

cargo mutants -- --all-targets

These options can also be configured statically with the additional_cargo_test_args key in .cargo/mutants.toml:

additional_cargo_test_args = ["--jobs=1"]

Arguments to test binaries

You can use a second double-dash to pass options through to the test targets:

cargo mutants -- -- --test-threads 1 --nocapture

(However, this may interact poorly with using additional_cargo_test_args in the configuration file, as the argument lists are currently appended without specially handling the -- separator.)

Build directories

cargo-mutants builds mutated code in a temporary directory, containing a copy of your source tree with each mutant successively applied. With --jobs, multiple build directories are used in parallel.

Build-in ignores

Files or directories matching these patterns are not copied:

.git
.hg
.bzr
.svn
_darcs
.pijul

gitignore

From 23.11.2, by default, cargo-mutants will not copy files that are excluded by gitignore patterns, to make copying faster in large trees.

This behavior can be turned off with --gitignore=false.

cargo-mutants with nextest

nextest is a tool for running Rust tests, as a replacement for cargo test.

You can use nextest to run your tests with cargo-mutants, instead of cargo test, by either passing the --test-tool=nextest option, or setting test_tool = "nextest" in .cargo/mutants.toml.

How nextest works

In the context of cargo-mutants the most important difference between cargo-test and nextest is that nextest runs each test in a separate process, and it can run tests from multiple test targets in parallel. (Nextest also has some nice UI improvements and other features, but they're not relevant here.)

This means that nextest can stop faster if a single test fails, whereas cargo test will continue running all the tests within the test binary.

This is beneficial for cargo-mutants, because it only needs to know whether at least one test caught the mutation, and so exiting as soon as there's a failure is better.

However, running each test individually also means there is more per-test startup cost, and so on some trees nextest may be slower.

In general, nextest will do relatively poorly on trees that have tests that are individually very fast, or on trees that establish shared or cached state across tests.

When to use nextest

There are at least two reasons why you might want to use nextest:

1. Some Rust source trees only support testing under nextest, and their tests fail under cargo test: in that case, you have to use this option! In particular, nextest's behavior of running each test in a separate process gives better isolation between tests.

2. Some trees might be faster under nextest than under cargo test, because they have a lot of tests that fail quickly, and the startup time is a small fraction of the time for the average test. This may or may not be true for your tree, so you can try it and see. Some trees, including cargo-mutants itself, are slower under nextest.

nextest and doctests

Caution: nextest currently does not run doctests, so behaviors that are only caught by doctests will show as missed when using nextest. (cargo-mutants could separately run the doctests, but currently does not.)

Baseline tests

Normally, cargo-mutants builds and runs your tree in a temporary directory before applying any mutations. This makes sure that your tests are in fact all passing, including in the copy of the tree that cargo-mutants will mutate.

Baseline tests can be skipped by passing the --baseline=skip command line option. (There is no config option for this.)

Performance effects

The performance gain from skipping the baseline is one run of the full test suite, plus one incremental build. When the baseline is run, its build is typically slow because it must do the initial build of the tree, but when it is skipped, the first mutant will have to do a full (rather than incremental) build instead.

This means that, in a run that tests many mutants, the relative performance gain from skipping the baseline will be relatively small. However, it may still be useful to skip baseline tests in some specific situations.

Timeouts

Normally, cargo-mutants uses the baseline test to establish an appropriate timeout for the test suite. If you skip the baseline, you should set --timeout manually.

Use cases for skipping baseline tests

--baseline=skip might be useful in these situations:

1. You are running cargo-mutants in a CI or build system that separately runs the tests before cargo-mutants. In this case, you can be confident that the tests are passing, and you can save time by skipping the baseline. In particular, if you are sharding work in CI, this avoids running the baseline on each shard.

2. You're repeatedly running cargo-mutants with different options, without changing the source code, perhaps with different --file or --exclude options.

3. You're developing cargo-mutants itself, and running it repeatedly on a tree that doesn't change.

Testing in place

By default, cargo-mutants copies your code to a temporary directory, where it applies mutations and then runs tests there.

With the --in-place option, it will instead mutate and test your code in the original source directory.

--in-place is currently incompatible with the --jobs option, because running multiple jobs requires making multiple copies of the tree.

Cautions

If you use --in-place then you shouldn't edit the code, commit, or run your own tests while tests are running, because cargo-mutants will be modifying the code at the same time. It's not the default because of the risk that users might accidentally do this.

cargo-mutants will try to restore the code to its original state after testing each mutant, but it's possible that it might fail to do so if it's interrupted or panics.

Why test in place?

Some situations where --in-place might be useful are:

• You're running cargo-mutants in CI with a source checkout that exists solely for testing, so it would be a waste of time and space to copy it.
• You've previously built the tree into target and want to avoid rebuilding it: the Rust toolchain currently doesn't reuse build products after cargo-mutants copies the tree, but it will reuse them with --in-place.
• The source tree is extremely large, and making a copy would use too much disk space, or take time that you don't want to spend. (In most cases copying the tree takes negligible time compared to running the tests, but if it contains many binary assets it might be significant.)
• You're investigating or debugging a problem where the tests don't pass in a copy of the tree. (Please report this as a bug if you can describe how to reproduce it.)

Generating mutants

cargo mutants generates mutants by inspecting the existing source code and applying a set of rules to generate new code that is likely to compile but have different behavior.

Mutants each have a "genre", each of which is described below.

Replace function body with value

The FnValue genre of mutants replaces a function's body with a value that is guessed to be of the right type.

This checks that the tests:

1. Observe any side effects of the original function.
2. Distinguish return values.

More mutation genres and patterns will be added in future releases.

... in the mutation patterns indicates that the type is recursively mutated. For example, Result<bool> can generate Ok(true) and Ok(false). The recursion can nest for types like Result<Option<String>>.

Some of these values may not be valid for all types: for example, returning Default::default() will work for many types, but not all. In this case the mutant is said to be "unviable": by default these are counted but not printed, although they can be shown with --unviable.

Binary operators

Binary operators are replaced with other binary operators in expressions like a == 0.

Equality operators are not currently replaced with comparisons like < or <= because they are too prone to generate false positives, for example when unsigned integers are compared to 0.

Generating error values

cargo-mutants can be configured to generate mutants that return an error value from functions that return a Result.

This will flag cases where no test fails if the function returns an error: that might happen if there are only tests for the error cases and not for the Ok case.

Since crates can choose to use any type for their error values, cargo-mutants must be told how to construct an appropriate error.

The --error command line option and the error_value configuration option specify an error value to use.

These options can be repeated or combined, which might be useful if there are multiple error types in the crate. On any one mutation site, probably only one of the error values will be viable, and cargo-mutants will discover that and use it.

The error value can be any Rust expression that evaluates to a value of the error type. It should not include the Err wrapper, because cargo-mutants will add that.

For example, if your crate uses anyhow::Error as its error type, you might use --error '::anyhow::anyhow!("error")'.

If you have your own error type, you might use --error 'crate::MyError::Generic'.

Since the correct error type is a property of the source tree, the configuration should typically go into .cargo/mutants.toml rather than being specified on the command line:

error_values = ["::anyhow::anyhow!(\"mutated\")"]

To see only the mutants generated by this configuration, you can use a command like this:

cargo r mutants -F anyhow -vV -j4

Improving performance

Most of the runtime for cargo-mutants is spent in running the program test suite and in running incremental builds: both are done once per viable mutant.

So, anything you can do to make the cargo build and cargo test suite faster will have a multiplicative effect on cargo mutants run time, and of course will also make normal development more pleasant.

https://matklad.github.io/2021/09/04/fast-rust-builds.html has good general advice on making Rust builds and tests faster.

Avoid doctests

Rust doctests are pretty slow, because every doctest example becomes a separate test binary. If you're using doctests only as testable documentation and not to assert correctness of the code, you can skip them with cargo mutants -- --all-targets.

Optimized builds

On some but not all projects, cargo-mutants can be faster if you use -C --release, which will make the build slower but may make the tests faster. Typically this will help on projects with very long CPU-intensive test suites.

cargo-mutants now shows the breakdown of build versus test time which may help you work out if this will help: if the tests are much slower than the build it's worth trying more more compiler optimizations.

On projects like this you might also choose just to turn up optimization for all debug builds in .cargo/config.toml.

Ramdisks

cargo-mutants causes the Rust toolchain (and, often, the program under test) to read and write many temporary files. Setting the temporary directory onto a ramdisk can improve performance significantly. This is particularly important with parallel builds, which might otherwise hit disk bandwidth limits. For example on Linux:

sudo mkdir /ram
sudo mount -t tmpfs /ram /ram  # or put this in fstab, or just change /tmp
sudo chmod 1777 /ram
env TMPDIR=/ram cargo mutants

Using the Mold linker

Using the Mold linker on Unix can give a 20% performance improvement, depending on the tree. Because cargo-mutants does many incremental builds, link time is important, especially if the test suite is relatively fast.

Because of limitations in the way cargo-mutants runs Cargo, the standard way of configuring Mold for Rust in ~/.cargo/config.toml won't work.

Instead, set the RUSTFLAGS environment variable to -Clink-arg=-fuse-ld=mold.

Parallelism

After the initial test of the unmutated tree, cargo-mutants can run multiple builds and tests of the tree in parallel on a single machine. Separately, you can shard work across multiple machines.

Caution: cargo build and cargo test internally spawn many threads and processes and can be very resource hungry. Don't set --jobs too high, or your machine may thrash, run out of memory, or overheat.

Background

Even though cargo builds, rustc, and Rust's test framework launch multiple processes or threads, they typically spend some time waiting for straggler tasks, during which time some CPU cores are idle. For example, a cargo build commonly ends up waiting for a single-threaded linker for several seconds.

Running one or more build or test tasks in parallel can use up this otherwise wasted capacity. This can give significant performance improvements, depending on the tree under test and the hardware resources available.

Timeouts

Because tests may be slower with high parallelism, or may exhibit more variability in execution time, you may see some spurious timeouts, and you may need to set --timeout manually to allow enough safety margin. (User feedback on this is welcome.)

Non-hermetic tests

If your test suite is non-hermetic -- for example, if it talks to an external database -- then running multiple jobs in parallel may cause test flakes. cargo-mutants is just running multiple copies of cargo test simultaneously: if that doesn't work in your tree, then you can't use this option.

Choosing a job count

You should set the number of jobs very conservatively, starting at -j2 or -j3.

Higher settings are only likely to be helpful on very large machines, perhaps with >100 cores and >256GB RAM.

Unlike with make, setting -j proportionally to the number of cores is unlikely to work out well, because so the Rust build and test tools already parallelize very aggressively.

The best setting will depend on many factors including the behavior of your program's test suite, the amount of memory on your system, and your system's behavior under high load. Ultimately you'll need to experiment to find the best setting.

To tune the number of jobs, you can watch htop or some similar program while the tests are running, to see whether cores are fully utilized or whether the system is running out of memory. On laptop or desktop machines you might also want to watch the temperature of the CPU.

As well as using more CPU and RAM, higher -j settings will also use more disk space in your temporary directory: Rust target directories can commonly be 2GB or more, and there will be one per parallel job, plus whatever temp files your test suite might create.

Interaction with --test-threads

The Rust test framework exposes a --test-threads option controlling how many threads run inside a test binary. cargo-mutants doesn't set this, but you can set it from the command line, along with other parameters to the test binary. You might need to set this if your test suite is non-hermetic with regard to global process state.

Limiting the number of threads inside a single test binary would tend to make that binary less resource-hungry, and so might allow you to set a higher -j option.

Reducing the number of test threads to increase -j seems unlikely to help performance in most trees.

Sharding

In addition to running multiple jobs locally, cargo-mutants can also run jobs on multiple machines, to get an overall job faster.

Each job tests a subset of mutants, selected by a shard. Shards are described as k/n, where n is the number of shards and k is the index of the shard, from 0 to n-1.

There is no runtime coordination between shards: they each independently discover the available mutants and then select a subset based on the --shard option.

If any shard fails then that would indicate that some mutants were missed, or there was some other problem.

Consistency across shards

CAUTION: All shards must be run with the same arguments, and the same sharding k, or the results will be meaningless, as they won't agree on how to divide the work.

Sharding can be combined with filters or shuffling, as long as the filters are set consistently in all shards. Sharding can also combine with --in-diff, again as long as all shards see the same diff.

Setting up sharding

Your CI system or other tooling is responsible for launching multiple shards, and for collecting the results. You're responsible for choosing the number of shards (see below).

For example, in GitHub Actions, you could use a matrix job to run multiple shards:

  cargo-mutants:
    runs-on: ubuntu-latest
    # needs: [build, incremental-mutants]
    strategy:
      matrix:
        shard: [0, 1, 2, 3, 4, 5, 6, 7]
    steps:
      - uses: actions/checkout@v3
      - uses: dtolnay/rust-toolchain@master
        with:
          toolchain: beta
      - uses: Swatinem/rust-cache@v2
      - run: cargo install cargo-mutants
      - name: Mutants
        run: |
          cargo mutants --no-shuffle -vV --shard ${{ matrix.shard }}/8
      - name: Archive mutants.out
        uses: actions/upload-artifact@v3
        if: always()
        with:
          name: mutants.out
          path: mutants.out

Note that the number of shards is set to match the /8 in the --shard argument.

Performance of sharding

Each mutant does some constant upfront work:

• Any CI setup including starting the machine, getting a checkout, installing a Rust toolchain, and installing cargo-mutants
• An initial clean build of the code under test
• A baseline run of the unmutated code

Then, for each mutant in its shard, it does an incremental build and runs all the tests.

Each shard runs the same number of mutants, +/-1. Typically this will mean they each take roughly the same amount of time, although it's possible that some shards are unlucky in drawing mutants that happen to take longer to test.

A rough model for the overall execution time for all of the shards, allowing for this work occurring in parallel, is

SHARD_STARTUP + (CLEAN_BUILD + TEST) + (N_MUTANTS/K) * (INCREMENTAL_BUILD + TEST)

The total cost in CPU seconds can be modelled as:

K * (SHARD_STARTUP + CLEAN_BUILD + TEST) + N_MUTANTS * (INCREMENTAL_BUILD + TEST)

As a result, at very large k the cost of the initial setup work will dominate, but overall time to solution will be minimized.

Choosing a number of shards

Because there's some constant overhead for every shard there will be diminishing returns and increasing ineffiency if you use too many shards. (In the extreme cases where there are more shards than mutants, some of them will do the setup work, then find they have nothing to do and immediately exit.)

As a rule of thumb, you should probably choose k such that each worker runs at least 10 mutants, and possibly much more. 8 to 32 shards might be a good place to start.

The optimal setting probably depends on how long your tree takes to build from zero and incrementally, how long the tests take to run, and the performance of your CI system.

If your CI system offers a choice of VM sizes you might experiment with using smaller or larger VMs and more or less shards: the optimal setting probably also depends on your tree's ability to exploit larger machines.

You should also think about cost and capacity constraints in your CI system, and the risk of starving out other users.

cargo-mutants has no internal scaling constraints to prevent you from setting k very large, if cost, efficiency and CI capacity are not a concern.

Sampling mutants

An option like --shard 1/100 can be used to run 1% of all the generated mutants for testing cargo-mutants, to get a sense of whether it works or to see how it performs on some tree.

Testing code changed in a diff

If you're working on a large project or one with a long test suite, you may not want to test the entire codebase every time you make a change. You can use cargo-mutants --in-diff to test only mutants generated from recently changed code.

The --in-diff DIFF_FILE option tests only mutants that overlap with regions changed in the diff.

The diff is expected to either have a prefix of b/ on the new filename, which is the format produced by git diff, or no prefix.

Some ways you could use --in-diff:

1. Before submitting code, check your uncommitted changes with git diff.
2. In CI, or locally, check the diff between the current branch and the base branch of the pull request.

Changes to non-Rust files, or files from which no mutants are produced, are ignored.

--in-diff is applied on the output of other filters including --package and --regex. For example, cargo mutants --in-diff --package foo will only test mutants in the foo package that overlap with the diff.

Caution

--in-diff makes tests faster by covering the mutants that are most likely to be missed in the changed code. However, it's certainly possible that edits in one region cause code in a different region or a different file to no longer be well tested. Incremental tests are helpful for giving faster feedback, but they're not a substitute for a full test run.

The diff is only matched against the code under test, not the test code. So, a diff that only deletes or changes test code won't cause any mutants to run, even though it may have a very material effect on test coverage.

Example

In this diff, we've added a new function two to src/lib.rs, and the existing code is unaltered. With --in-diff, cargo-mutants will only test mutants that affect the function two.

```diff
--- a/src/lib.rs    2023-11-12 13:05:25.774658230 -0800
+++ b/src/lib.rs    2023-11-12 12:54:04.373806696 -0800
@@ -2,6 +2,10 @@
     "one".to_owned()
 }

+pub fn two() -> String {
+    format!("{}", 2)
+}
+
 #[cfg(test)]
 mod test_super {
     use super::*;
@@ -10,4 +14,9 @@
     fn test_one() {
         assert_eq!(one(), "one");
     }
+
+    #[test]
+    fn test_two() {
+        assert_eq!(two(), "2");
+    }
 }

Integrations

Shell completion

The --completions SHELL emits completion scripts for the given shell.

The right place to install these depends on your shell and operating system.

For example, for Fish¹:

cargo mutants --completions fish >~/.config/fish/conf.d/cargo-mutants-completions.fish

vim-cargomutants

vim-cargomutants provides commands view cargo-mutants results, see the diff of mutations, and to launch cargo-mutants from within vim.

Continuous integration

You might want to use cargo-mutants in your continuous integration (CI) system, to ensure that no uncaught mutants are merged into your codebase.

There are at least two complementary ways to use cargo-mutants in CI:

1. Check for mutants produced in the code changed in a pull request. This is typically much faster than testing all mutants, and is a good way to ensure that newly merged code is well tested, and to facilitate conversations about how to test the PR.

2. Checking that all mutants are caught, on PRs or on the development branch.

Recommendations for CI

• Use the --in-place option to avoid copying the tree.

Installing into CI

The recommended way to install cargo-mutants is using install-action, which will fetch a binary from cargo-mutants most recent GitHub release, which is faster than building from source. You could alternatively use baptiste0928/cargo-install which will build it from source in your worker and cache the result.

Example workflow

Here is an example of a GitHub Actions workflow that runs mutation tests and uploads the results as an artifact. This will fail if it finds any uncaught mutants.

name: cargo-mutants

on:
  push:
    branches:
      - main
  pull_request:
    # Only test PR if it changes something that's likely to affect the results, because
    # mutant tests can take a long time. Adjust these paths to suit your project.
    paths:
      - ".cargo/**"
      - ".github/workflows/mutants.yml"
      - "Cargo.*"
      - "src/**"
      - "testdata/**"
      - "tests/**"

jobs:
  cargo-mutants:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
      - uses: taiki-e/install-action@v2
        name: Install cargo-mutants using install-action
        with:
          tool: cargo-mutants
      - name: Run mutant tests
        run: cargo mutants -vV --in-place
      - name: Archive results
        uses: actions/upload-artifact@v3
        if: always()
        with:
          name: mutants-out
          path: mutants.out

Incremental tests of pull requests

You can use --in-diff to test only the code that has changed in a pull request. This can be useful for incremental testing in CI, where you want to test only the code that has changed since the last commit.

For example, you can use the following workflow to test only the code that has changed in a pull request:

name: Tests

permissions:
  contents: read

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  incremental-mutants:
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Relative diff
        run: |
          git branch -av
          git diff origin/${{ github.base_ref }}.. | tee git.diff
      - uses: Swatinem/rust-cache@v2
      - uses: taiki-e/install-action@v2
        name: Install cargo-mutants using install-action
        with:
          tool: cargo-mutants
      - name: Mutants
        run: |
          cargo mutants --no-shuffle -j 2 -vV --in-diff git.diff
      - name: Archive mutants.out
        uses: actions/upload-artifact@v3
        if: always()
        with:
          name: mutants-incremental.out
          path: mutants.out

How cargo-mutants works

The basic approach is:

• Build a list of mutations:

  • Run cargo metadata to find directories containing Rust source files.
  • Walk all source files and parse each one looking for functions.
  • Skip functions that should not be mutated for any of several reasons: because they're tests, because they have a #[mutants::skip] attribute, etc.
  • For each function, depending on its return type, generate every mutation pattern that produces a result of that type.

• Make a copy of the source tree into a scratch directory, excluding version-control directories like .git and the /target directory. The same directory is reused across all the mutations to benefit from incremental builds.

  • After copying the tree, cargo-mutants scans the top-level Cargo.toml and any .cargo/config.toml for relative dependencies. If there are any, the paths are rewritten to be absolute, so that they still work when cargo is run in the scratch directory.
  • Before applying any mutations, check that cargo test succeeds in the scratch directory: perhaps a test is already broken, or perhaps the tree doesn't build when copied because it relies on relative paths to find dependencies, etc. This is called the "baseline" test.
  • If running more than one parallel job, make the appropriate number of additional scratch directories.

• For each mutation:

  • Apply the mutation to the scratch tree by patching the affected file.
  • Run cargo build: if this fails, the mutant is unviable, and that's ok.
  • Run cargo test in the tree, saving output to a log file.
  • If the the tests fail, that's good: the mutation was somehow caught.
  • If the tests succeed, that might mean test coverage was inadequate, or it might mean we accidentally generated a no-op mutation.
  • Revert the mutation to return the tree to its clean state.

The file is parsed using the syn crate, but mutations are applied textually, rather than to the token stream, so that unmutated code retains its prior formatting, comments, line numbers, etc. This makes it possible to show a text diff of the mutation and should make it easier to understand any error messages from the build of the mutated code.

For more details, see DESIGN.md.

Goals

The goal of cargo-mutants is to be easy to run on any Rust source tree, and to tell you something interesting about areas where bugs might be lurking or the tests might be insufficient.

The detailed goals in this section are intended to generally guide development priorities and tradeoffs. For example, the goal of ease means that we will generally prefer to automatically infer reasonable default behavior rather than requiring the user to configure anything at first. The goal of being interesting means that we will generally only enable by default features that seem reasonably likely to say something important about test quality to at least some users.

Ease

Being easy to use means:

• cargo-mutants requires no changes to the source tree or other setup: just install and run. So, if it does not find anything interesting to say about a well-tested tree, it didn't cost you much. (This worked out really well: cargo install cargo-mutants && cargo mutants will do it.)

• There is no chance that running cargo-mutants will change the released behavior of your program (other than by helping you to fix bugs!), because you don't need to change the source to use it.

• cargo-mutants should be reasonably fast even on large Rust trees. The overall run time is, roughly, the product of the number of viable mutations multiplied by the time to run the test suite for each mutation. Typically, one cargo mutants run will give you all the information it can find about missing test coverage in the tree, and you don't need to run it again as you iterate on tests, so it's relatively OK if it takes a while.

  (There is currently very little overhead beyond the cost to do an incremental build and run the tests for each mutant, but that can still take a while for large trees that produce many mutants especially if their test suite takes a while.)

• cargo-mutants should run correctly on any Rust source trees that are built and tested by Cargo, that will build and run their tests in a copy of the tree, and that have hermetic tests.

• cargo-mutants shouldn't crash or hang, even if it generates mutants that cause the software under test to crash or hang.

• The results should be reproducible, assuming the build and test suite is deterministic.

• cargo-mutants should avoid generating unviable mutants that don't compile, because that wastes time. However, when it's uncertain whether the mutant will build, it's worth trying things that might find interesting results even if they might fail to build. (It does currently generate some unviable mutants, but typically not too many, and they don't have a large effect on runtime in most trees.)

• Realistically, cargo-mutants may generate some mutants that aren't caught by tests but also aren't interesting, or aren't feasible to test. In those cases it should be easy to permanently dismiss them (e.g. by adding a #[mutants::skip] attribute or a config file.)

Interestingness

Showing interesting results mean:

• cargo-mutants should tell you about places where the code could be wrong and the test suite wouldn't catch it. If it doesn't find any interesting results on typical trees, there's no point. Aspirationally, it will even find useful results in code with high line coverage, when there is code that is reached by a test, but no test depends on its behavior.

• In superbly-tested projects cargo-mutants may find nothing to say, but hey, at least it was easy to run, and hopefully the assurance that the tests really do seem to be good is useful data.

• Most, ideally all, findings should indicate something that really should be tested more, or that may already be buggy, or that's at least worth looking at.

• It should be easy to understand what the output is telling you about a potential bug that wouldn't be caught. (This seems true today.) It might take some thought to work out why the existing tests don't cover it, or how to check it, but at least you know where to begin.

• As much as possible cargo-mutants should avoid generating trivial mutants, where the mutated code is effectively equivalent to the original code, and so it's not interesting that the test suite doesn't catch the change.

• For trees that are thoroughly tested, you can use cargo mutants in CI to check that they remain so.

How is mutation testing different to coverage measurement?

Coverage measurements tell you which lines of code (or other units) are reached while running a test. They don't tell you whether the test really checks anything about the behavior of the code.

For example, a function that writes a file and returns a Result might be covered by a test that checks the return value, but not by a test that checks that the file was actually written. cargo-mutants will try mutating the function to simply return Ok(()) and report that this was not caught by any tests.

Historically, rust coverage measurements have required manual setup of several OS and toolchain-dependent tools, although this is improving. Because cargo-mutants just runs cargo it has no OS-specific or tight toolchain integrations, and so is simple to install and run on any Rust source tree. cargo-mutants also needs no special tools to view or interpret the results.

Coverage tools also in some cases produce output that is hard to interpret, with lines sometimes shown as covered or not due to toolchain quirks that aren't easy to map to direct changes to the test suite. cargo-mutants produces a direct list of changes that are not caught by the test suite, which can be quickly reviewed and prioritized.

One drawback of mutation testing is that it runs the whole test suite once per generated mutant, so it can be slow on large trees with slow test suites. There are some techniques to speed up cargo-mutants, including running multiple tests in parallel.

How is mutation testing different to fuzzing?

Fuzzing is a technique for finding bugs by feeding pseudo-random inputs to a program, and is particularly useful on programs that parse complex or untrusted inputs such as binary file formats or network protocols.

Mutation testing makes algorithmically-generated changes to a copy of the program source, and measures whether the test suite catches the change.

The two techniques are complementary. Although some bugs might be found by either technique, fuzzing will tend to find bugs that are triggered by complex or unusual inputs, whereas mutation testing will tend to point out logic that might be correct but that's not tested.

Limitations, caveats, known bugs, and future enhancements

Cases where cargo-mutants can't help

cargo-mutants can only help if the test suite is hermetic: if the tests are flaky or non-deterministic, or depend on external state, it will draw the wrong conclusions about whether the tests caught a bug.

If you rely on testing the program's behavior by manual testing, or by an integration test not run by cargo test, then cargo-mutants can't know this, and will only tell you about gaps in the in-tree tests. It may still be helpful to run mutation tests on only some selected modules that do have in-tree tests.

Running cargo-mutants on your code won't, by itself, make your code better. It only helps suggest places you might want to improve your tests, and that might indirectly find bugs, or prevent future bugs. Sometimes the results will point out real current bugs. But it's on you to follow up. (However, it's really easy to run, so you might as well look!)

cargo-mutants typically can't do much to help with crates that primarily generate code using macros or build scripts, because it can't "see" the code that's generated. (You can still run it, but it's may generate very few mutants.)

Stability

cargo-mutants behavior, output formats, command-line syntax, json output formats, etc, may change from one release to the next.

Limitations and known bugs

cargo-mutants currently only supports mutation testing of Rust code that builds using cargo and where the tests are run using cargo test. Support for other tools such as Bazel or Nextest could in principle be added.

cargo-mutants sees the AST of the tree but doesn't fully "understand" the types, so sometimes generates unviable mutants or misses some opportunities to generate interesting mutants.

cargo-mutants reads CARGO_ENCODED_RUSTFLAGS and RUSTFLAGS environment variables, and sets CARGO_ENCODED_RUSTFLAGS. It does not read .cargo/config.toml files, and so any rust flags set there will be ignored.

cargo-mutants does not yet understand conditional compilation, such as #[cfg(target_os = "linux")]. It will report functions for other platforms as missed, when it should know to skip them.

Support for other build tools

cargo-mutants currently only works with Cargo, but could in principle be extended to work with other build tools such as Bazel.

cargo-mutants contains two main categories of code, which are mostly independent:

1. Code for reading Rust source code, parsing it, and mutating it: this is not specific to Cargo.

2. Code for finding the modules to mutate and their source files, finding the tree to copy, adjusting paths after it is copied, and finally running builds and tests. This is very Cargo-specific, but should not be too hard to generalize.

The main precondition for supporting Bazel is a realistic test case: preferably an open source Rust tree built with Bazel, or at least a contributor with a Bazel-based Rust tree who is willing to help test and debug and to produce some test cases.

(See https://github.com/sourcefrog/cargo-mutants/issues/77 for more discussion.)

Caution on side effects

cargo-mutants builds and runs code with machine-generated modifications. This is generally fine, but if the code under test has side effects such as writing or deleting files, running it with mutations might conceivably have unexpected effects, such as deleting the wrong files, in the same way that a bug might.

If you're concerned about this, run cargo-mutants in a container or virtual machine.

cargo-mutants never modifies the original source tree, other than writing a mutants.out directory, and that can be sent elsewhere with the --output option. All mutations are applied and tested in a copy of the source tree.

How to help

Experience reports in GitHub Discussions or issues are very welcome:

• Did it find a bug or important coverage gap?
• Did it fail to build and test your tree? (Some cases that aren't supported yet are already listed in this doc or the bug tracker.)

It's especially helpful if you can either point to an open source tree that will reproduce the problem (or success) or at least describe how to reproduce it.

If you are interested in contributing a patch, please read CONTRIBUTING.md.

cargo-mutants changelog

Unreleased

24.1.2

• New: --in-place option tests mutations in the original source tree, without copying the tree. This is faster and uses less disk space, but it's incompatible with --jobs, and you must be careful not to edit or commit the source tree while tests are running.

24.1.1

• New: Mutate +, -, *, /, %, &, ^, |, <<, >> binary ops, and their corresponding assignment ops like +=.

• New: --baseline=skip option to skip running tests in an unmutated tree, when they're already been checked externally.

• Changed: Stop generating mutations of || and && to != and ||, because it seems to raise too many low-value false positives that may be hard to test.

• Fixed: Colors in command-line help and error messages.

24.1.0

• New! cargo mutants --test-tool nextest, or test_tool = "nextest" in .cargo/mutants.toml runs tests under Nextest. Some trees have tests that only work under Nextest, and this allows them to be tested. In other cases Nextest may be significantly faster, because it will exit soon after the first test failure.

• Fixed: Fixed spurious "Patch input contains repeated filenames" error when --in-diff is given a patch that deletes multiple files.

23.12.2

• New: A --shard k/n allows you to split the work across n independent parallel cargo mutants invocations running on separate machines to get a faster overall solution on large suites. You, or your CI system, are responsible for launching all the shards and checking whether any of them failed.

• Improved: Better documentation about -j, with stronger recommendations not to set it too high.

• New: Binary releases on GitHub through cargo-dist.

23.12.1

• Improved progress bars and console output, including putting the outcome of each mutant on the left, and the overall progress bar at the bottom. Improved display of estimated remaining time, and other times.

• Fixed: Correctly traverse mod statements within package top source files that are not named lib.rs or main.rs, by following the path setting of each target within the manifest.

• Improved: Don't generate function mutants that have the same AST as the code they're replacing.

23.12.0

An exciting step forward: cargo-mutants can now generate mutations smaller than a whole function. To start with, several binary operators are mutated.

• New: Mutate == to != and vice versa.

• New: Mutate && to || and vice versa, and mutate both of them to == and !=.

• New: Mutate <, <=, >, >=.

• Changed: If no mutants are generated then cargo mutants now exits successfully, showing a warning. (Previously it would exit with an error.) This works better with --in-diff in CI, where it's normal that some changes may not have any mutants.

• Changed: Include column numbers in text listings of mutants and output to disambiguate smaller-than-function mutants, for example if there are several operators that can be changed on one line. This also applies to the names used for regex matching, so may break some regexps that match the entire line (sorry). The new option --line-col=false turns them both off in --list output.

• Changed: In the mutants.json format, replaced the function, line, and return_type fields with a function submessage (including the name and return type) and a span indicating the entire replaced region, to better handle smaller-than-function mutants. Also, the function includes the line-column span of the entire function.

23.11.2

• Changed: If --file or --exclude are set on the command line, then they replace the corresponding config file options. Similarly, if --re is given then the examine_re config key is ignored, and if --exclude-re is given then exclude_regex is ignored. (Previously the values were combined.) This makes it easier to use the command line to test files or mutants that are normally not tested.

• Improved: By default, files matching gitignore patterns (including in parent directories, per-user configuration, and info/exclude) are excluded from copying to temporary build directories. This should improve performance in some large trees with many files that are not part of the build. This behavior can be turned off with --gitignore=false.

• Improved: Run cargo metadata with --no-deps, so that it doesn't download and compute dependency information, which can save time in some situations.

• Added: Alternative aliases for command line options, so you don't need to remember if it's "regex" or "re": --regex, --examine-re, --examine-regex (all for names to include) and --exclude-regex.

• Added: Accept --manifest-path as an alternative to -d, for consistency with other cargo commands.

23.11.1

• New --in-diff FILE option tests only mutants that are in the diff from the given file. This is useful to avoid testing mutants from code that has not changed, either locally or in CI.

23.11.0

• Changed: cargo mutants now tries to match the behavior of cargo test when run within a workspace. If run in a package directory, it tests only that package. If run in a workspace that is not a package (a "virtual workspace"), it tests the configured default packages, or otherwise all packages. This can all be overridden with the --package or --workspace options.

• New: generate key-value map values from types like BTreeMap<String, Vec<u8>>.

• Changed: Send trace messages to stderr rather stdout, in part so that it won't pollute json output.

23.10.0

• The baseline test (with no mutants) now tests only the packages in which mutants will be generated, subject to any file or regex filters. This should both make baseline tests faster, and allow testing workspaces in which some packages have non-hermetic tests.

23.9.1

• Mutate the known collection types BinaryHeap, BTreeSet, HashSet, LinkedList, and VecDeque to generate empty and one-element collections using T::new() and T::from_iter(..).

• Mutate known container types like Arc, Box, Cell, Mutex, Rc, RefCell into T::new(a).

• Mutate unknown types that look like containers or collections T<A> or T<'a, A>' and try to construct them from an A with T::from_iter, T::new, and T::from.

• Minimum Rust version updated to 1.70.

• Mutate Cow<'_, T> into Owned and Borrowed variants.

• Mutate functions returning &[T] and &mut [T] to return leaked vecs of values.

• Mutate (A, B, C, ...) into the product of all replacements for a, b, c, ...

• The combination of options --list --diff --json is now supported, and emits a diff key in the JSON.

• Mutate -> impl Iterator<Item = A> to produce empty and one-element iterators of the item type.

23.9.0

• Fixed a bug causing an assertion failure when cargo-mutants was run from a subdirectory of a workspace. Thanks to Adam Chalmers!

• Generate HttpResponse::Ok().finish() as a mutation of an Actix HttpResponse.

23.6.0

• Generate Box::leak(Box::new(...)) as a mutation of functions returning &mut.

• Add a concept of mutant "genre", which is included in the json listing of mutants. The only genre today is FnValue, in which a function body is replaced by a value. This will in future allow filtering by genre.

• Recurse into return types, so that for example Result<bool> can generate Ok(true) and Ok(false), and Some<T> generates None and every generated value of T. Similarly for Box<T>, Vec<T>, Rc<T>, Arc<T>.

• Generate specific values for integers: [0, 1] for unsigned integers, [0, 1, -1] for signed integers; [1] for NonZero unsigned integers and [1, -1] for NonZero signed integers.

• Generate specific values for floats: [0.0, 1.0, -1.0].

• Generate (fixed-length) array values, like [0; 256], [1; 256] using every recursively generated value for the element type.

23.5.0

"Pickled crab"

Released 2023-05-27

• cargo mutants can now successfully test packages that transitively depend on a different version of themselves, such as itertools. Previously, cargo-mutants used the cargo --package option, which is ambiguous in this case, and now it uses --manifest-path instead.

• Mutate functions returning &'_ str (whether a lifetime is named or not) to return "xyzzy" and "".

• Switch to CalVer numbering.

1.2.3

Released 2023-05-05

• Mutate functions returning String to String::new() rather than "".into(): same result but a bit more idiomatic.

• New --leak-dirs option, for debugging cargo-mutants.

• Update to syn 2.0, adding support for new Rust syntax.

• Minimum supported Rust version increased to 1.65 due to changes in dependencies.

• New --error option, to cause functions returning Result to be mutated to return the specified error.

• New --no-config option, to disable reading .cargo/mutants.toml.

1.2.2

Released 2023-04-01

• Don't mutate unsafe fns.

• Don't mutate functions that never return (i.e. -> !).

• Minimum supported Rust version increased to 1.64 due to changes in dependencies.

• Some command-line options can now also be configured through environment variables: CARGO_MUTANTS_JOBS, CARGO_MUTANTS_TRACE_LEVEL.

• New command line option --minimum-test-timeout and config file variable minimum_test_timeout join existing environment variable CARGO_MUTANTS_MINIMUM_TEST_TIMEOUT, to allow boosting the minimum, especially for test environments with poor or uneven throughput.

• Changed: Renamed fields in outcomes.json from cargo_result to process_status and from command to argv.

• Warn if no mutants were generated or if all mutants were unviable.

1.2.1

Released 2023-01-05

• Converted most of the docs to a book available at https://mutants.rs/.

• Fixed: Correctly find submodules that don't use mmod.rsnaming, e.g. when descending fromsrc/foo.rstosrc/foo/bar.rs. Also handle module names that are raw identifiers usingr#`. (Thanks to @kpreid for the report.)

1.2.0

Thankful mutants!

• Fixed: Files that are excluded by filters are also excluded from --list-files.

• Fixed: --exclude-re and --re can match against the return type as shown in --list.

• New: A .cargo/mutants.toml file can be used to configure standard filters and cargo args for a project.

1.1.1

Released 2022-10-31

Spooky mutants!

• Fixed support for the Mold linker, or for other options passed via RUSTFLAGS or CARGO_ENCODED_RUSTFLAGS. (See the instructions in README.md).

• Source trees are walked by following mod statements rather than globbing the directory. This is more correct if there are files that are not referenced by mod statements. Once attributes on modules are stable in Rust (https://github.com/rust-lang/rust/issues/54727) this opens a path to skip mods using attributes.

1.1.0

Released 2022-10-30

Fearless concurrency!

• cargo-mutants can now run multiple cargo build and test tasks in parallel, to make better use of machine resources and find mutants faster, controlled by --jobs.

• The minimum Rust version to build cargo-mutants is now 1.63.0. It can still be used to test code under older toolchains.

1.0.3

Released 2022-09-29

• cargo-mutants is now finds no uncaught mutants in itself! Various tests were added and improved, particularly around handling timeouts.

• New: --re and --exclude-re options to filter by mutant name, including the path. The regexps match against the strings printed by --list.

1.0.2

Released 2022-09-24

• New: cargo mutants --completions SHELL to generate shell completions using clap_complete.

• Changed: cargo-mutants no longer builds in the source directory, and no longer copies the target/ directory to the scratch directory. Since cargo-mutants now sets RUSTFLAGS to avoid false failures from warnings, it is unlikely to match the existing build products in the source directory target/, and in fact building there is just likely to cause rebuilds in the source. The behavior now is as if --no-copy-target was always passed. That option is still accepted, but it has no effect.

• Changed: cargo-mutants finds all possible mutations before doing the baseline test, so that you can see earlier how many there will be.

• New: Set INSTA_UPDATE=no so that tests that use the Insta library don't write updates back into the source directory, and so don't falsely pass.

1.0.1

Released 2022-09-12

• Fixed: Don't try to mutate functions within test targets, e.g. within tests/**/*.rs.

• New: missed.txt, caught.txt, timeout.txt and unviable.txt files are written in to the output directory to make results easier to review later.

• New: --output creates the specified directory if it does not exist.

• Internal: Switched from Argh to Clap for command-line parsing. There may be some small changes in CLI behavior and help formatting.

1.0.0

Released 2022-08-21

A 1.0 release to celebrate that with the addition of workspace handling, cargo-mutants gives useful results on many Rust projects.

• New: Supports workspaces containing multiple packages. Mutants are generated for all relevant targets in all packages, and mutants are subject to the tests of their own package. cargo mutants --list-files --json and cargo mutants --list --json now includes package names for each file or mutant.

• Improved: Generate mutations in cdylib, rlib, and ever other *lib target. For example, this correctly exercises Wasm projects.

• Improved: Write mutants.out/outcomes.json after the source-tree build and baseline tests so that it can be observed earlier on.

• Improved: mutants.out/outcomes.json includes the commands run.

0.2.11

Released 2022-08-20

• New --exclude command line option to exclude source files from mutants generation, matching a glob.

• New: CARGO_MUTANTS_MINIMUM_TEST_TIMEOUT sets a minimum timeout for cargo tests, in seconds. This can be used to allow more time on slow CI builders. If unset the default is still 20s.

• Added: A new mutants.out/debug.log with internal debugging information.

• Improved: The time for check, build, and test is now shown separately in progress bars and output, to give a better indication of which is taking more time in the tree under test. Also, times are show in seconds with one decimal place, and they are styled more consistently.

• Improved: More consistent use of 'unviable' and other terms for outcomes in the UI.

0.2.10

Released 2022-08-07

cargo-mutants 0.2.10 comes with improved docs, and the new -C option can be used to pass options like --release or --all-features to cargo.

• Added: --cargo-arg (or -C for short) allows passing arguments to cargo commands (check, build, and test), for example to set --release or --features.

• Improved: Works properly if run from a subdirectory of a crate, or if -d points to a subdirectory of a crate.

• Improved: Various docs.

• Improved: Relative dependencies within the source tree are left as relative paths, and will be built within the scratch directory. Relative dependencies outside the source tree are still rewritten as absolute paths.

0.2.9

Released 2022-07-30

• Faster: cargo mutants no longer runs cargo check before building, in cases where the build products are wanted or tests will be run. This saves a significant amount of work in build phases; in some trees cargo mutants is now 30% faster. (In trees where most of the time is spent running tests the effect will be less.)

• Fixed: Open log files in append mode to fix messages from other processes occasionally being partly overwritten.

• Improved: cargo mutants should now give useful results in packages that use #![deny(unused)] or other mechanisms to reject warnings. Mutated functions often ignore some parameters, which would previously be rejected by this configuration without proving anything interesting about test coverage. Now, --cap-lints=allow is passed in RUSTFLAGS while building mutants, so that they're not falsely rejected and the tests can be exercised.

• Improved: The build dir name includes the root package name.

• Improved: The progress bar shows more information.

• Improved: The final message shows how many mutants were tested and how long it took.

0.2.8

Released 2022-07-18

• New: Summarize the overall number of mutants generated, caught, missed, etc, at the end.

• Fixed: Works properly with crates that have relative path dependencies in Cargo.toml or .cargo/config.toml, by rewriting them to absolute paths in the scratch directory.

0.2.7

Released 2022-07-11

• New: You can skip functions by adding #[cfg_attr(test, mutants::skip), in which case the mutants crate can be only a dev-dependency.

• Improved: Don't generate pointless mutations of functions with an empty body (ignoring comments.)

• Improved: Remove extra whitespace from the display of function names and return types: the new formatting is closer to the spacing used in idiomatic Rust.

• Improved: Show the last line of compiler/test output while running builds, so that it's more clear where time is being spent.

• Docs: Instructions on how to check for missed mutants from CI.

0.2.6

Released 2022-04-17

• Improved: Find source files by looking at cargo metadata output, rather than assuming they're in src/**/*.rs. This makes cargo mutants work properly on trees where it previously failed to find the source.

• New --version option.

• New: Write a lock.json into the mutants.out directory including the start timestamp, cargo-mutants version, hostname and username. Take a lock on this file while cargo mutants is running, so that it doesn't crash or get confused if two tasks try to write to the same directory at the same time.

• New: Restored a --list-files option.

• Changed: Error if no mutants are generated, which probably indicates a bug or configuration error(?)

0.2.5

Released 2022-04-14

• New --file command line option to mutate only functions in source files matching a glob.

• Improved: Don't attempt to mutate functions called new or implementations of Default. cargo-mutants can not yet generate good mutations for these so they are generally false positives.

• Improved: Better display of <impl Foo for Bar>::foo and similar type paths.

• New: --output directory to write mutants.out somewhere other than the source directory.

0.2.4

Released 2022-03-26

• Fix: Ignore errors setting file mtimes during copies, which can cause failures on Windows if some files are readonly.

• Fix: Log file names now include only the source file relative path, the line number, and a counter, so they are shorter, and shouldn't cause problems on filesystems with length limits.

• Change: version-control directories like .git are not copied with the source tree: they should have no effect on the build, so copying them is just a waste.

• Changed/improved json logs in mutants.out:

  • Show durations as fractional seconds.

  • Outcomes include a "summary" field.

0.2.3

Released 2022-03-23

• Switch from Indicatif to Nutmeg to draw progress bars and output. This fixes a bug where terminal output line-wraps badly, and adds a projection for the total estimated time to completion.

• Change: Mutants are now tested in random order by default, so that repeated runs are more likely to surface interesting new findings early, rather than repeating previous results. The previous behavior of testing mutants in the deterministic order they're encountered in the tree can be restored with --no-shuffle.

0.2.2

Released 2022-02-16

• The progress bar now shows which mutant is being tested out of how many total.

• The automatic timeout is now set to the minimum of 20 seconds, or 5x the time of the tests in a baseline tree, to reduce the incidence of false timeouts on machines with variable throughput.

• Ctrl-c (or SIGINT) interrupts the program during copying the tree. Previously it was not handled until the copy was complete.

• New --no-copy-target option.

0.2.1

Released 2022-02-10

• Arguments to cargo test can be passed on the command line after --. This allows, for example, skipping doctests or setting the number of test threads. https://github.com/sourcefrog/cargo-mutants/issues/15

0.2.0

Released 2022-02-06

• A new --timeout SECS option to limit the runtime of any cargo test invocation, so that mutations that cause tests to hang don't cause cargo mutants to hang.

  A default timeout is set based on the time to run tests in an unmutated tree. There is no timeout by default on the unmutated tree.

  On Unix, the cargo subprocesses run in a new process group. As a consequence ctrl-c is explicitly caught and propagated to the child processes.

• Show a progress bar while looking for mutation opportunities, and show the total number found.

• Show how many mutation opportunities were found, before testing begins.

• New --shuffle option tests mutants in random order.

• By default, the output now only lists mutants that were missed or that timed out. Mutants that were caught, and mutants that did not build, can be printed with --caught and --unviable respectively.

0.1.0

Released 2021-11-30

• Logs and other information are written into mutants.out in the source directory, rather than target/mutants.

• New --all-logs option prints all Cargo output to stdout, which is verbose but useful for example in CI, by making all the output directly available in captured stdout.

• The output distinguishes check or build failures (probably due to an unviable mutant) from test failures (probably due to lacking coverage.)

• A new file mutants.out/mutants.json lists all the generated mutants.

• Show function return types in some places, to make it easier to understand whether the mutants were useful or viable.

• Run cargo check --tests and cargo build --tests in the source directory to freshen the build and download any dependencies, before copying it to a scratch directory.

• New --check option runs cargo check on generated mutants to see if they are viable, without actually running the tests. This is useful in tuning cargo-mutants to generate better mutants.

• New --no-times output hides times (and tree sizes) from stdout, mostly to make the output deterministic and easier to match in tests.

• Mutate methods too!

0.0.4

Released 2021-11-10

• Fixed cargo install cargo-mutants (sometimes?) failing due to the derive feature not getting set on the serde dependency.

• Show progress while copying the tree.

• Respect the $CARGO environment variable so that the same toolchain is used to run tests as was used to invoke cargo mutants. Concretely, cargo +nightly mutants should work correctly.

0.0.3

Released 2021-11-06

• Skip functions or modules marked #[test], #[cfg(test)] or #[mutants::skip].

• Early steps towards type-guided mutations:

  • Generate mutations of true and false for functions that return bool
  • Empty and arbitrary strings for functions returning String.
  • Return Ok(Default::default()) for functions that return Result<_, _>.

• Rename --list-mutants to just --list.

• New --list --json.

• Colored output makes test names and mutations easier to read (for me at least.)

• Return distinct exit codes for different situations including that uncaught mutations were found.

0.0.2

• Functions that should not be mutated can be marked with #[mutants::skip] from the mutants helper crate.

0.0.1

First release.