Welcome to cargo-mutants

cargo-mutants is a mutation testing tool for Rust. It helps you improve your program's quality by finding places where bugs can be inserted causing any tests to fail.

The goal of cargo-mutants is to be easy to run on any Rust source tree, and to tell you something interesting about areas where bugs might be lurking or the tests might be insufficient. (More about these goals.)

To get started:

  1. Install cargo-mutants.
  2. Run `cargo mutants in your Rust source tree.

For more resources see the repository at https://github.com/sourcefrog/cargo-mutants.

Installation

Install cargo-mutants from source:

cargo install --locked cargo-mutants

Supported Rust versions

Building cargo-mutants requires a reasonably recent stable (or nightly or beta) Rust toolchain. The supported version is visible in https://crates.io/crates/cargo-mutants.

After installing cargo-mutants, you should be able to use it to run tests under any toolchain, even toolchains that are far too old to build cargo-mutants, using the standard + option to cargo:

cargo +1.48 mutants

Getting started

Just run cargo mutants in a Rust source directory, and it will point out functions that may be inadequately tested.

Example

; cargo mutants
Found 14 mutants to test
Copy source to scratch directory ... 0 MB in 0.0s
Unmutated baseline ... ok in 1.6s build + 0.3s test
Auto-set test timeout to 20.0s
src/lib.rs:386: replace <impl Error for Error>::source -> Option<&(dyn std::error::Error + 'static)>
 with Default::default() ... NOT CAUGHT in 0.6s build + 0.3s test
src/lib.rs:485: replace copy_symlink -> Result<()> with Ok(Default::default()) ...
 NOT CAUGHT in 0.5s build + 0.3s test
14 mutants tested in 0:08: 2 missed, 9 caught, 3 unviable

In v0.5.1 of the cp_r crate, the copy_symlink function was reached by a test but not adequately tested, and the Error::source function was not tested at all.

Using the results

Tests fail in an clean tree?

If tests fail in a clean copy of the tree, there might be an (intermittent) failure in the source directory, or there might be some problem that stops them passing when run from a different location. Fix this first: cargo-mutants can't do anything until you have a tree where cargo test passes reliably when copied to a temporary directory.

Mutant outcomes

Assuming tests pass in a clean copy of the tree, cargo-mutants proceeds to generate every mutant it can, subject to any configured filters, and then runs cargo build and cargo test on each of them.

Each mutant results in one of the following outcomes:

  • caught — A test failed with this mutant applied. This is a good sign about test coverage.

  • missed — No test failed with this mutation applied, which seems to indicate a gap in test coverage. Or, it may be that the mutant is undistinguishable from the correct code. You may wish to add a better test, or mark that the function should be skipped.

  • unviable — The attempted mutation doesn't compile. This is inconclusive about test coverage and no action is needed, but indicates an opportunity for cargo-mutants to either generate better mutants, or at least not generate unviable mutants.

  • timeout — The mutation caused the test suite to run for a long time, until it was eventually killed. You might want to investigate the cause and potentially mark the function to be skipped.

By default only missed mutants and timeouts are printed to stdout, because they're the most actionable. Others can be shown with the --caught and --unviable options.

What to do about missed mutants?

Each missed mutant is a sign that there might be a gap in test coverage. What to do about them is up to you, bearing in mind your goals and priorities for your project, but here are some suggestions:

First, look at the overall list of missed mutants: there might be patterns such as a cluster of related functions all having missed mutants. Probably some will stand out as potentially more important to the correct function of your program.

You should first look for any mutations where it's very surprising that they were not caught by any tests, given what you know about the codebase. For example, if cargo-mutants reports that replacing an important function with Ok(()) is not caught then that seems important to investigate.

You should then look at the tests that you would think would catch the mutant: that might be unit tests within the relevant module, or some higher-level public-API or integration test, depending on how your project's tests are structured.

If you can't find any tests that you think should have caught the mutant, then perhaps you should add some. The right thing here is not necessarily to directly assert that the mutated behavior doesn't happen. For example, if the mutant changed a private function, you don't necessarily want to add a test for that private function, but instead ask yourself what public-API behavior would break if the private function was buggy, and then add a test for that.

Try to avoid writing tests that are too tightly targeted to the mutant, which is really just an example of something that could be wrong, and instead write tests that assert the correct behavior at the right level of abstraction, preferably through a public interface.

If it's not clear why the tests aren't already failing, it may help to manually inject the same mutation into your working tree and then run the tests under a debugger, or add trace statements to the test. (The --diff option or looking in the mutants.out directory will show you exactly what change cargo-mutants made.)

You may notice some messages about missed mutants in functions that you feel are not very important to test, such as Debug implementations. You can use the --exclude-re options to filter out these mutants, or mark them as skipped with #[mutants::skip]. (Or, you might decide that you do want to add unit tests for the Debug representation, but perhaps as a lower priority than investigating mutants in more important code.)

In some cases cargo-mutants will generate a mutant that is effectively the same as the original code, and so not really incorrect. cargo-mutants tries to avoid doing this, but if it does happen then you can mark the function as skipped.

Iterating on mutant coverage

After you've changed your program to address some of the missed mutants, you can run cargo mutants again with the --file option to re-test only functions from the changed files.

Hard-to-test cases

Some functions don't cause a test suite failure if emptied, but also cannot be removed. For example, functions to do with managing caches or that have other performance side effects.

Ideally, these should be tested, but doing so in a way that's not flaky can be difficult. cargo-mutants can help in a few ways:

  • It helps to at least highlight to the developer that the function is not covered by tests, and so should perhaps be treated with extra care, or tested manually.
  • A #[mutants::skip] annotation can be added to suppress warnings and explain the decision.
  • Sometimes these effects can be tested by making the side-effect observable with, for example, a counter of the number of memory allocations or cache misses/hits.

Hangs and timeouts

Some mutations to the tree can cause the test suite to hang. For example, in this code, cargo-mutants might try changing should_stop to always return false, but this will cause the program to hang:

#![allow(unused)]
fn main() {
    while !should_stop() {
      // something
    }
}

In general you will want to skip functions which cause a hang when mutated, either by marking them with an attribute or in the configuration file.

Timeouts

To avoid hangs, cargo-mutants will kill the test suite after a timeout and continue to the next mutant.

By default, the timeout is set automatically. cargo-mutants measures the time to run the test suite in the unmodified tree, and then sets a timeout for mutated tests at 5x the time to run tests with no mutations, and a minimum of 20 seconds.

The minimum of 20 seconds can be overriden by the CARGO_MUTANTS_MINIMUM_TEST_TIMEOUT environment variable, measured in seconds.

You can also set an explicit timeout with the --timeout option, also measured in seconds. If this option is specified then the timeout is also applied to the unmutated tests.

The timeout does not apply to cargo check or cargo build, only cargo test.

Exit codes

cargo-mutants returns an exit code that can be used by scripts or CI.

  • 0: Success! Every viable mutant that was tested was caught by a test.

  • 1: Usage error: bad command-line arguments etc.

  • 2: Found some mutants that were not covered by tests.

  • 3: Some tests timed out: possibly the mutatations caused an infinite loop, or the timeout is too low.

  • 4: The tests are already failing or hanging before any mutations are applied, so no mutations were tested.

For more detailed machine-readable information, use the mutants.out directory.

The mutants.out directory

A mutants.out directory is created in the original source directory. You can put the output directory elsewhere with the --output option.

On each run, any existing mutants.out is renamed to mutants.out.old, and any existing mutants.out.old is deleted.

The output directory contains:

  • A lock.json, on which an fs2 lock is held while cargo-mutants is running, to avoid two tasks trying to write to the same directory at the same time. lock.json contains the start time, cargo-mutants version, username, and hostname. lock.json is left in mutants.out when the run completes, but the lock on it is released.

  • A mutants.json file describing all the generated mutants. This file is completely written before testing begins.

  • An outcomes.json file describing the results of all tests, and summary counts of each outcome.

  • A logs/ directory, with one log file for each mutation plus the baseline unmutated case. The log contains the diff of the mutation plus the output from cargo. outcomes.json includes for each mutant the name of the log file.

  • caught.txt, missed.txt, timeout.txt, unviable.txt, each listing mutants with the corresponding outcome.

The contents of the directory and the format of these files is subject to change in future versions.

These files are incrementally updated while cargo-mutants runs, so other programs can read them to follow progress.

There is generally no reason to include this directory in version control, so it is recommended that you add /mutants.out* to your .gitignore file. This will exclude both mutants.out and mutants.out.old.

Skipping untestable code

Some functions may be inherently hard to cover with tests, for example if:

  • Generated mutants cause tests to hang.
  • You've chosen to test the functionality by human inspection or some higher-level integration tests.
  • The function has side effects or performance characteristics that are hard to test.
  • You've decided the function is not important to test.

There are three ways to skip mutating some code:

  1. Marking the function with an attribute within the source file.
  2. Filtering by path in the config file or command line.
  3. Filtering by function and mutant name in the config file or command line.

The results of all these filters can be previewed using the --list option.

Which filtering method to use?

  • If some particular functions are hard to test with cargo-mutants, use an attribute, so that the skip is visible in the code.
  • If a whole module is untestable, use a filter by path in the config file, so that the filter's stored in the source tree and covers any new code in that module.
  • If you want to permanently ignore a class of functions, such as Debug implementations, use a regex filter in the config file.
  • If you want to run cargo-mutants just once, focusing on a subset of files, functions, or mutants, use command line options to filter by name or path.

Skipping functions with an attribute

To mark functions as skipped, so they are not mutated:

  1. Add a Cargo dependency on the mutants crate, version "0.0.3" or later. (This must be a regular dependency not a dev-dependency, because the annotation will be on non-test code.)

  2. Mark functions with #[mutants::skip] or other attributes containing mutants::skip (e.g. #[cfg_attr(test, mutants::skip)]).

The mutants create is tiny and the attribute has no effect on the compiled code. It only flags the function for cargo-mutants. However, you can avoid the dependency by using the slightly longer #[cfg_attr(test, mutants::skip)] form.

Note: Currently, cargo-mutants does not (yet) evaluate attributes like cfg_attr, it only looks for the sequence mutants::skip in the attribute.

You may want to also add a comment explaining why the function is skipped.

For example:

#![allow(unused)]
fn main() {
use std::time::{Duration, Instant};

/// Returns true if the program should stop
#[cfg_attr(test, mutants::skip)] // Returning false would cause a hang
fn should_stop() -> bool {
    true
}

pub fn controlled_loop() {
    let start = Instant::now();
    for i in 0.. {
        println!("{}", i);
        if should_stop() {
            break;
        }
        if start.elapsed() > Duration::from_secs(60 * 5) {
            panic!("timed out");
        }
    }
}

mod test {
    #[test]
    fn controlled_loop_terminates() {
        super::controlled_loop()
    }
}
}

Filtering files

Two options (each with short and long names) control which files are mutated:

  • -f GLOB, --file GLOB: Mutate only functions in files matching the glob.

  • -e GLOB, --exclude GLOB: Exclude files that match the glob.

These options may be repeated.

If any -f options are given, only source files that match are considered; otherwise all files are considered. This list is then further reduced by exclusions.

If the glob contains / (or on Windows, \), then it matches against the path from the root of the source tree. For example, src/*/*.rs will exclude all files in subdirectories of src.

If the glob does not contain a path separator, it matches against filenames in any directory. / matches the path separator on both Unix and Windows.

Note that the glob must contain .rs (or a matching wildcard) to match source files with that suffix. For example, -f network will match src/network/mod.rs but it will not match src/network.rs.

Files that are excluded are still parsed (and so must be syntactically valid), and mod statements in them are followed to discover other source files. So, for example, you can exclude src/main.rs but still test mutants in other files referenced by mod statements in main.rs.

Since Rust does not currently allow attributes such as #[mutants::skip] on mod statements or at module scope filtering by filename is the only way to skip an entire module.

The results of filters can be previewed with the --list-files and --list options.

Examples:

  • cargo mutants -f visit.rs -f change.rs -- test mutants only in files called visit.rs or change.rs (in any directory).

  • cargo mutants -e console.rs -- test mutants in any file except console.rs.

  • cargo mutants -f src/db/*.rs -- test mutants in any file in this directory.

Configuring filters by filename

Files may also be filtered with the exclude_globs and examine_globs options in .cargo/mutants.toml.

Exclusions in the config file may be particularly useful when there are modules that are inherently hard to automatically test, and the project has made a decision to accept lower test coverage for them.

From cargo-mutants 23.11.2 onwards, if the command line options are given then the corresponding config file option is ignored. This allows you to use the config file to test files that are normally expected to pass, and then to use the command line to test files that are not yet passing.

For example:

exclude_globs = ["src/main.rs", "src/cache/*.rs"] # like -e
examine_globs = ["src/important/*.rs"] # like -f: test *only* these files

Filtering functions and mutants

You can filter mutants by name, using the --re and --exclude-re command line options and the corresponding examine_re and exclude_re config file options.

These options are useful if you want to run cargo-mutants just once, focusing on a subset of functions or mutants.

These options filter mutants by the full name of the mutant, which includes the function name, file name, and a description of the change, as shown in the output of cargo mutants --list.

For example, one mutant name might be:

src/outcome.rs:157: replace <impl Serialize for ScenarioOutcome>::serialize -> Result<S::Ok, S::Error> with Ok(Default::default())

Within this name, your regex can match any substring, including for example:

  • The filename
  • The trait, impl Serialize
  • The struct name, ScenarioOutcome
  • The function name, serialize
  • The mutated return value, with Ok(Defualt::default()), or any part of it.

The regex matches a substring, but can be anchored with ^ and $ to require that it match the whole name.

The regex syntax is defined by the regex crate.

These filters are applied after filtering by filename, and --re is applied before --exclude-re.

Examples:

  • -E 'impl Debug' -- don't test impl Debug methods, because coverage of them might be considered unimportant.

  • -F 'impl Serialize' -F 'impl Deserialize' -- test implementations of these two traits.

Configuring filters by name

Mutants can be filtered by name in the .cargo/mutants.toml file. The exclude_re and examine_re keys are each a list of strings.

This can be helpful if you want to systematically skip testing implementations of certain traits, or functions with certain names.

From cargo-mutants 23.11.2 onwards, if the command line options are given then the corresponding config file option is ignored.

For example:

exclude_re = ["impl Debug"] # same as -E

Controlling cargo-mutants

cargo mutants takes various options to control how it runs.

These options, can, in general, be passed on the command line, set in a .cargo/mutants.toml file in the source tree, or passed in CARGO_MUTANTS_ environment variables. Not every method of setting an option is available for every option, however, as some would not make sense or be useful.

For options that take a list of values, values from the configuration file are appended to values from the command line.

For options that take a single value, the value from the command line takes precedence.

--no-config can be used to disable reading the configuration file.

Execution order

By default, mutants are run in a randomized order, so as to surface results from different parts of the codebase earlier. This can be disabled with --no-shuffle, in which case mutants will run in order by file name and within each file in the order they appear in the source.

Source directory location

-d, --dir: Test the Rust tree in the given directory, rather than the source tree enclosing the working directory where cargo-mutants is launched.

Console output

-v, --caught: Also print mutants that were caught by tests.

-V, --unviable: Also print mutants that failed cargo build.

--no-times: Don't print elapsed times.

-L, --level, and $CARGO_MUTANTS_TRACE_LEVEL: set the verbosity of trace output to stdout. The default is info, and it can be increased to debug or trace.

Listing generated mutants

--list: Show what mutants could be generated, without running them.

--diff: With --list, also include a diff of the source change for each mutant.

--json: With --list, show the list in json for easier processing by other programs. (The same format is written to mutants.out/mutants.json when running tests.)

--check: Run cargo check on all generated mutants to find out which ones are viable, but don't actually run the tests. (This is primarily useful when debugging cargo-mutants.)

Workspace and package support

cargo-mutants supports testing Cargo workspaces that contain multiple packages. The entire workspace tree is copied.

By default, cargo-mutants has the same behavior as Cargo:

  • If --workspace is given, all packages in the workspace are tested.
  • If --package is given, the named packages are tested.
  • If the starting directory (or -d directory) is in a package, that package is tested.
  • Otherwise, the starting directory must be in a virtual workspace. If it specifies default members, they are tested. Otherwise, all packages are tested.

For each mutant, only the containing package's tests are run, on the theory that each package's tests are responsible for testing the package's code.

The baseline tests exercise all and only the packages for which mutants will be generated.

You can also use the --file options to restrict cargo-mutants to testing only files from some subdirectory, e.g. with -f "utils/**/*.rs". (Remember to quote globs on the command line, so that the shell doesn't expand them.) You can use --list or --list-files to preview the effect of filters.

Passing options to Cargo

cargo-mutants runs cargo build and cargo test (or, with --check, it runs cargo check.) Additional options can be passed in three different ways: to all cargo commands; to cargo test only; and to the test binaries run by cargo test.

There is not yet a way to pass options only to cargo build but not to cargo test.

Arguments to all cargo commands

To pass more arguments to every Cargo invocation, use --cargo-arg, or the additional_cargo_args configuration key. --cargo-arg can be repeated.

For example

cargo mutants -- --cargo-arg=--release

or in .cargo/mutants.toml:

additional_cargo_args = ["--all-features"]

Arguments to cargo test

Command-line options following a -- delimiter are passed through to cargo test. For example, this can be used to pass --all-targets which (unobviously) excludes doctests. (If the doctests are numerous and slow, and not relied upon to catch bugs, this can improve performance.)

cargo mutants -- --all-targets

These options can also be configured statically with the additional_cargo_test_args key in .cargo/mutants.toml:

additional_cargo_test_args = ["--jobs=1"]

Arguments to test binaries

You can use a second double-dash to pass options through to the test targets:

cargo mutants -- -- --test-threads 1 --nocapture

(However, this may interact poorly with using additional_cargo_test_args in the configuration file, as the argument lists are currently appended without specially handling the -- separator.)

Build directories

cargo-mutants builds mutated code in a temporary directory, containing a copy of your source tree with each mutant successively applied. With --jobs, multiple build directories are used in parallel.

Build-in ignores

Files or directories matching these patterns are not copied:

.git
.hg
.bzr
.svn
_darcs
.pijul

gitignore

From 23.11.2, by default, cargo-mutants will not copy files that are excluded by gitignore patterns, to make copying faster in large trees.

This behavior can be turned off with --gitignore=false.

Generating mutants

cargo mutants generates mutants by inspecting the existing source code and applying a set of rules to generate new code that is likely to compile but have different behavior.

Mutants each have a "genre", each of which is described below.

Replace function body with value

The FnValue genre of mutants replaces a function's body with a value that is guessed to be of the right type.

This checks that the tests:

  1. Observe any side effects of the original function.
  2. Distinguish return values.

More mutation genres and patterns will be added in future releases.

Return typeMutation pattern
()() (return unit, with no side effects)
signed integers0, 1, -1
unsigned integers0, 1
floats0.0, 1.0, -1.0
NonZeroI*1, -1
NonZeroU*1
booltrue, false
StringString::new(), "xyzzy".into()
&'_ str ."", "xyzzy"
&mut ...Box::leak(Box::new(...))
Result<T>Ok(...) , and an error if configured
Option<T>Some(...), None
Box<T>Box::new(...)
Vec<T>vec![], vec![...]
Arc<T>Arc::new(...)
Rc<T>Rc::new(...)
BinaryHeap, BTreeSet, HashSet, LinkedList, VecDequeempty and one-element collections
BTreeMap, HashMapempty map and the product of all key and value replacements
Cow<'_, T>Cow::Borrowed(t), Cow::Owned(t.to_owned())
[T; L][r; L] for all replacements of T
&[T], &mut [T]Leaked empty and one-element vecs
&T&... (all replacements for T)
HttpResponseHttpResponse::Ok().finish
(A, B, ...)(a, b, ...) for the product of all replacements of A, B, ...
impl IteratorEmpty and one-element iterators of the inner type
(any other)Default::default()

... in the mutation patterns indicates that the type is recursively mutated. For example, Result<bool> can generate Ok(true) and Ok(false). The recursion can nest for types like Result<Option<String>>.

Some of these values may not be valid for all types: for example, returning Default::default() will work for many types, but not all. In this case the mutant is said to be "unviable": by default these are counted but not printed, although they can be shown with --unviable.

Binary operators

Binary operators are replaced with other binary operators in expressions like a == 0.

OperatorReplacements
==!=
!===
&&\|\|, ==, !=
\|\|&&, ==, !=
<==, >
>==, <
<===, >=
>===, <=

Equality operators are not currently replaced with comparisons like < or <= because they are too prone to generate false positives, for example when unsigned integers are compared to 0.

Logical && and || are replaced with == and != which function as XNOR and XOR respectively, although they are fairly often unviable due to needing parenthesis when the original operator does not.

Generating error values

cargo-mutants can be configured to generate mutants that return an error value from functions that return a Result.

This will flag cases where no test fails if the function returns an error: that might happen if there are only tests for the error cases and not for the Ok case.

Since crates can choose to use any type for their error values, cargo-mutants must be told how to construct an appropriate error.

The --error command line option and the error_value configuration option specify an error value to use.

These options can be repeated or combined, which might be useful if there are multiple error types in the crate. On any one mutation site, probably only one of the error values will be viable, and cargo-mutants will discover that and use it.

The error value can be any Rust expression that evaluates to a value of the error type. It should not include the Err wrapper, because cargo-mutants will add that.

For example, if your crate uses anyhow::Error as its error type, you might use --error '::anyhow::anyhow!("error")'.

If you have your own error type, you might use --error 'crate::MyError::Generic'.

Since the correct error type is a property of the source tree, the configuration should typically go into .cargo/mutants.toml rather than being specified on the command line:

error_values = ["::anyhow::anyhow!(\"mutated\")"]

To see only the mutants generated by this configuration, you can use a command like this:

cargo r mutants -F anyhow -vV -j4

Improving performance

Most of the runtime for cargo-mutants is spent in running the program test suite and in running incremental builds: both are done once per viable mutant.

So, anything you can do to make the cargo build and cargo test suite faster will have a multiplicative effect on cargo mutants run time, and of course will also make normal development more pleasant.

https://matklad.github.io/2021/09/04/fast-rust-builds.html has good general advice on making Rust builds and tests faster.

Avoid doctests

Rust doctests are pretty slow, because every doctest example becomes a separate test binary. If you're using doctests only as testable documentation and not to assert correctness of the code, you can skip them with cargo mutants -- --all-targets.

Optimized builds

On some but not all projects, cargo-mutants can be faster if you use -C --release, which will make the build slower but may make the tests faster. Typically this will help on projects with very long CPU-intensive test suites.

cargo-mutants now shows the breakdown of build versus test time which may help you work out if this will help: if the tests are much slower than the build it's worth trying more more compiler optimizations.

On projects like this you might also choose just to turn up optimization for all debug builds in .cargo/config.toml.

Ramdisks

cargo-mutants causes the Rust toolchain (and, often, the program under test) to read and write many temporary files. Setting the temporary directory onto a ramdisk can improve performance significantly. This is particularly important with parallel builds, which might otherwise hit disk bandwidth limits. For example on Linux:

sudo mkdir /ram
sudo mount -t tmpfs /ram /ram  # or put this in fstab, or just change /tmp
sudo chmod 1777 /ram
env TMPDIR=/ram cargo mutants

Using the Mold linker

Using the Mold linker on Unix can give a 20% performance improvement, depending on the tree. Because cargo-mutants does many incremental builds, link time is important, especially if the test suite is relatively fast.

Because of limitations in the way cargo-mutants runs Cargo, the standard way of configuring Mold for Rust in ~/.cargo/config.toml won't work.

Instead, set the RUSTFLAGS environment variable to -Clink-arg=-fuse-ld=mold.

Parallelism

After the initial test of the unmutated tree, cargo-mutants can test multiple mutants in parallel. This can give significant performance improvements, depending on the tree under test and the hardware resources available.

Even though cargo builds, rustc, and Rust's test framework launch multiple processes or threads, they typically can't use all available CPU cores all the time, and many cargo test runs will end up using only one core waiting for the last task to complete. Running multiple jobs in parallel makes use of resources that would otherwise be idle.

By default, only one job is run at a time.

To run more, use the --jobs or -j option, or set the CARGO_MUTANTS_JOBS environment variable.

Setting this higher than the number of CPU cores is unlikely to be helpful.

The best setting will depend on many factors including the behavior of your program's test suite, the amount of memory on your system, and your system's behavior under high thermal load.

-j 4 may be a good starting point. Start there and watch memory and CPU usage, and tune towards a setting where all cores are fully utilized without apparent thrashing, memory exhaustion, or thermal issues.

Because tests may be slower with high parallelism, you may see some spurious timeouts, and you may need to set --timeout manually to allow enough safety margin.

Incremental tests of modified code

If you're working on a large project or one with a long test suite, you may not want to test the entire codebase every time you make a change. You can use cargo-mutants --in-diff to test only mutants generated from recently changed code.

The --in-diff DIFF_FILE option tests only mutants that overlap with regions changed in the diff.

The diff is expected to either have a prefix of b/ on the new filename, which is the format produced by git diff, or no prefix.

Some ways you could use --in-diff:

  1. Before submitting code, check your uncommitted changes with git diff.
  2. In CI, or locally, check the diff between the current branch and the base branch of the pull request.

Changes to non-Rust files, or files from which no mutants are produced, are ignored.

--in-diff is applied on the output of other filters including --package and --regex. For example, cargo mutants --in-diff --package foo will only test mutants in the foo package that overlap with the diff.

Caution

--in-diff makes tests faster by covering the mutants that are most likely to be missed in the changed code. However, it's certainly possible that edits in one region cause code in a different region or a different file to no longer be well tested. Incremental tests are helpful for giving faster feedback, but they're not a substitute for a full test run.

Example

In this diff, we've added a new function two to src/lib.rs, and the existing code is unaltered. With --in-diff, cargo-mutants will only test mutants that affect the function two.


```diff
--- a/src/lib.rs    2023-11-12 13:05:25.774658230 -0800
+++ b/src/lib.rs    2023-11-12 12:54:04.373806696 -0800
@@ -2,6 +2,10 @@
     "one".to_owned()
 }

+pub fn two() -> String {
+    format!("{}", 2)
+}
+
 #[cfg(test)]
 mod test_super {
     use super::*;
@@ -10,4 +14,9 @@
     fn test_one() {
         assert_eq!(one(), "one");
     }
+
+    #[test]
+    fn test_two() {
+        assert_eq!(two(), "2");
+    }
 }

Integrations

Shell completion

The --completions SHELL emits completion scripts for the given shell.

The right place to install these depends on your shell and operating system.

For example, for Fish1:

cargo mutants --completions fish >~/.config/fish/conf.d/cargo-mutants-completions.fish
1

This command installs them to conf.d instead of completions because you may have completions for several cargo plugins.

vim-cargomutants

vim-cargomutants provides commands view cargo-mutants results, see the diff of mutations, and to launch cargo-mutants from within vim.

Continuous integration

Here is an example of a GitHub Actions workflow that runs mutation tests and uploads the results as an artifact. This will fail if it finds any uncaught mutants.

name: cargo-mutants

on: [pull_request, push]

jobs:
  cargo-mutants:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
      - name: Install cargo-mutants
        run: cargo install --locked cargo-mutants
      - name: Run mutant tests
        run: cargo mutants -- --all-features
      - name: Archive results
        uses: actions/upload-artifact@v3
        if: failure()
        with:
          name: mutation-report
          path: mutants.out

The workflow used by cargo-mutants on itself can be seen at https://github.com/sourcefrog/cargo-mutants/blob/main/.github/workflows/mutate-self.yaml.

Incremental tests of pull requests

You can use --in-diff to test only the code that has changed in a pull request. This can be useful for incremental testing in CI, where you want to test only the code that has changed since the last commit.

For example, you can use the following workflow to test only the code that has changed in a pull request:

name: Tests

permissions:
  contents: read

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  incremental-mutants:
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Relative diff
        run: |
          git branch -av
          git diff origin/${{ github.base_ref }}.. | tee git.diff
      - uses: Swatinem/rust-cache@v2
      - run: cargo install cargo-mutants
      - name: Mutants
        run: |
          cargo mutants --no-shuffle -j 2 -vV --in-diff git.diff
      - name: Archive mutants.out
        uses: actions/upload-artifact@v3
        if: always()
        with:
          name: mutants-incremental.out
          path: mutants.out

How cargo-mutants works

The basic approach is:

  • Build a list of mutations:

    • Run cargo metadata to find directories containing Rust source files.
    • Walk all source files and parse each one looking for functions.
    • Skip functions that should not be mutated for any of several reasons: because they're tests, because they have a #[mutants::skip] attribute, etc.
    • For each function, depending on its return type, generate every mutation pattern that produces a result of that type.

  • Make a copy of the source tree into a scratch directory, excluding version-control directories like .git and the /target directory. The same directory is reused across all the mutations to benefit from incremental builds.

    • After copying the tree, cargo-mutants scans the top-level Cargo.toml and any .cargo/config.toml for relative dependencies. If there are any, the paths are rewritten to be absolute, so that they still work when cargo is run in the scratch directory.
    • Before applying any mutations, check that cargo test succeeds in the scratch directory: perhaps a test is already broken, or perhaps the tree doesn't build when copied because it relies on relative paths to find dependencies, etc.
    • If running more than one parallel job, make the appropriate number of additional scratch directories.

  • For each mutation:

    • Apply the mutation to the scratch tree by patching the affected file.
    • Run cargo build: if this fails, the mutant is unviable, and that's ok.
    • Run cargo test in the tree, saving output to a log file.
    • If the the tests fail, that's good: the mutation was somehow caught.
    • If the tests succeed, that might mean test coverage was inadequate, or it might mean we accidentally generated a no-op mutation.
    • Revert the mutation to return the tree to its clean state.

The file is parsed using the syn crate, but mutations are applied textually, rather than to the token stream, so that unmutated code retains its prior formatting, comments, line numbers, etc. This makes it possible to show a text diff of the mutation and should make it easier to understand any error messages from the build of the mutated code.

For more details, see DESIGN.md.

Goals

The goal of cargo-mutants is to be easy to run on any Rust source tree, and to tell you something interesting about areas where bugs might be lurking or the tests might be insufficient.

The detailed goals in this section are intended to generally guide development priorities and tradeoffs. For example, the goal of ease means that we will generally prefer to automatically infer reasonable default behavior rather than requiring the user to configure anything at first. The goal of being interesting means that we will generally only enable by default features that seem reasonably likely to say something important about test quality to at least some users.

Ease

Being easy to use means:

  • cargo-mutants requires no changes to the source tree or other setup: just install and run. So, if it does not find anything interesting to say about a well-tested tree, it didn't cost you much. (This worked out really well: cargo install cargo-mutants && cargo mutants will do it.)

  • There is no chance that running cargo-mutants will change the released behavior of your program (other than by helping you to fix bugs!), because you don't need to change the source to use it.

  • cargo-mutants should be reasonably fast even on large Rust trees. The overall run time is, roughly, the product of the number of viable mutations multiplied by the time to run the test suite for each mutation. Typically, one cargo mutants run will give you all the information it can find about missing test coverage in the tree, and you don't need to run it again as you iterate on tests, so it's relatively OK if it takes a while.

    (There is currently very little overhead beyond the cost to do an incremental build and run the tests for each mutant, but that can still take a while for large trees that produce many mutants especially if their test suite takes a while.)

  • cargo-mutants should run correctly on any Rust source trees that are built and tested by Cargo, that will build and run their tests in a copy of the tree, and that have hermetic tests.

  • cargo-mutants shouldn't crash or hang, even if it generates mutants that cause the software under test to crash or hang.

  • The results should be reproducible, assuming the build and test suite is deterministic.

  • cargo-mutants should avoid generating unviable mutants that don't compile, because that wastes time. However, when it's uncertain whether the mutant will build, it's worth trying things that might find interesting results even if they might fail to build. (It does currently generate some unviable mutants, but typically not too many, and they don't have a large effect on runtime in most trees.)

  • Realistically, cargo-mutants may generate some mutants that aren't caught by tests but also aren't interesting, or aren't feasible to test. In those cases it should be easy to permanently dismiss them (e.g. by adding a #[mutants::skip] attribute or a config file.)

Interestingness

Showing interesting results mean:

  • cargo-mutants should tell you about places where the code could be wrong and the test suite wouldn't catch it. If it doesn't find any interesting results on typical trees, there's no point. Aspirationally, it will even find useful results in code with high line coverage, when there is code that is reached by a test, but no test depends on its behavior.

  • In superbly-tested projects cargo-mutants may find nothing to say, but hey, at least it was easy to run, and hopefully the assurance that the tests really do seem to be good is useful data.

  • Most, ideally all, findings should indicate something that really should be tested more, or that may already be buggy, or that's at least worth looking at.

  • It should be easy to understand what the output is telling you about a potential bug that wouldn't be caught. (This seems true today.) It might take some thought to work out why the existing tests don't cover it, or how to check it, but at least you know where to begin.

  • As much as possible cargo-mutants should avoid generating trivial mutants, where the mutated code is effectively equivalent to the original code, and so it's not interesting that the test suite doesn't catch the change.

  • For trees that are thoroughly tested, you can use cargo mutants in CI to check that they remain so.

How is mutation testing different to coverage measurement?

Coverage measurements tell you which lines of code (or other units) are reached while running a test. They don't tell you whether the test really checks anything about the behavior of the code.

For example, a function that writes a file and returns a Result might be covered by a test that checks the return value, but not by a test that checks that the file was actually written. cargo-mutants will try mutating the function to simply return Ok(()) and report that this was not caught by any tests.

Historically, rust coverage measurements have required manual setup of several OS and toolchain-dependent tools, although this is improving. Because cargo-mutants just runs cargo it has no OS-specific or tight toolchain integrations, and so is simple to install and run on any Rust source tree. cargo-mutants also needs no special tools to view or interpret the results.

Coverage tools also in some cases produce output that is hard to interpret, with lines sometimes shown as covered or not due to toolchain quirks that aren't easy to map to direct changes to the test suite. cargo-mutants produces a direct list of changes that are not caught by the test suite, which can be quickly reviewed and prioritized.

One drawback of mutation testing is that it runs the whole test suite once per generated mutant, so it can be slow on large trees with slow test suites. There are some techniques to speed up cargo-mutants, including running multiple tests in parallel.

How is mutation testing different to fuzzing?

Fuzzing is a technique for finding bugs by feeding pseudo-random inputs to a program, and is particularly useful on programs that parse complex or untrusted inputs such as binary file formats or network protocols.

Mutation testing makes algorithmically-generated changes to a copy of the program source, and measures whether the test suite catches the change.

The two techniques are complementary. Although some bugs might be found by either technique, fuzzing will tend to find bugs that are triggered by complex or unusual inputs, whereas mutation testing will tend to point out logic that might be correct but that's not tested.

Limitations, caveats, known bugs, and future enhancements

Cases where cargo-mutants can't help

cargo-mutants can only help if the test suite is hermetic: if the tests are flaky or non-deterministic, or depend on external state, it will draw the wrong conclusions about whether the tests caught a bug.

If you rely on testing the program's behavior by manual testing, or by an integration test not run by cargo test, then cargo-mutants can't know this, and will only tell you about gaps in the in-tree tests. It may still be helpful to run mutation tests on only some selected modules that do have in-tree tests.

Running cargo-mutants on your code won't, by itself, make your code better. It only helps suggest places you might want to improve your tests, and that might indirectly find bugs, or prevent future bugs. Sometimes the results will point out real current bugs. But it's on you to follow up. (However, it's really easy to run, so you might as well look!)

cargo-mutants typically can't do much to help with crates that primarily generate code using macros or build scripts, because it can't "see" the code that's generated. (You can still run it, but it's may generate very few mutants.)

Stability

cargo-mutants behavior, output formats, command-line syntax, json output formats, etc, may change from one release to the next.

Limitations and known bugs

cargo-mutants currently only supports mutation testing of Rust code that builds using cargo and where the tests are run using cargo test. Support for other tools such as Bazel or Nextest could in principle be added.

cargo-mutants sees the AST of the tree but doesn't fully "understand" the types, so sometimes generates unviable mutants or misses some opportunities to generate interesting mutants.

cargo-mutants reads CARGO_ENCODED_RUSTFLAGS and RUSTFLAGS environment variables, and sets CARGO_ENCODED_RUSTFLAGS. It does not read .cargo/config.toml files, and so any rust flags set there will be ignored.

cargo-mutants does not yet understand conditional compilation, such as #[cfg(target_os = "linux")]. It will report functions for other platforms as missed, when it should know to skip them.

Support for other build tools

cargo-mutants currently only works with Cargo, but could in principle be extended to work with other build tools such as Bazel.

cargo-mutants contains two main categories of code, which are mostly independent:

  1. Code for reading Rust source code, parsing it, and mutating it: this is not specific to Cargo.

  2. Code for finding the modules to mutate and their source files, finding the tree to copy, adjusting paths after it is copied, and finally running builds and tests. This is very Cargo-specific, but should not be too hard to generalize.

The main precondition for supporting Bazel is a realistc test case: preferably an open source Rust tree built with Bazel, or at least a contributor with a Bazel-based Rust tree who is willing to help test and debug and to produce some test cases.

(See https://github.com/sourcefrog/cargo-mutants/issues/77 for more discussion.)

Caution on side effects

cargo-mutants builds and runs code with machine-generated modifications. This is generally fine, but if the code under test has side effects such as writing or deleting files, running it with mutations might conceivably have unexpected effects, such as deleting the wrong files, in the same way that a bug might.

If you're concerned about this, run cargo-mutants in a container or virtual machine.

cargo-mutants never modifies the original source tree, other than writing a mutants.out directory, and that can be sent elsewhere with the --output option. All mutations are applied and tested in a copy of the source tree.

How to help

Experience reports in GitHub Discussions or issues are very welcome:

  • Did it find a bug or important coverage gap?
  • Did it fail to build and test your tree? (Some cases that aren't supported yet are already listed in this doc or the bug tracker.)

It's especially helpful if you can either point to an open source tree that will reproduce the problem (or success) or at least describe how to reproduce it.

If you are interested in contributing a patch, please read CONTRIBUTING.md.

cargo-mutants changelog

23.12.0

An exciting step forward: cargo-mutants can now generate mutations smaller than a whole function. To start with, several binary operators are mutated.

  • New: Mutate == to != and vice versa.

  • New: Mutate && to || and vice versa, and mutate both of them to == and !=.

  • New: Mutate <, <=, >, >=.

  • Changed: If no mutants are generated then cargo mutants now exits successfully, showing a warning. (Previously it would exit with an error.) This works better with --in-diff in CI, where it's normal that some changes may not have any mutants.

  • Changed: Include column numbers in text listings of mutants and output to disambiguate smaller-than-function mutants, for example if there are several operators that can be changed on one line. This also applies to the names used for regex matching, so may break some regexps that match the entire line (sorry). The new option --line-col=false turns them both off in --list output.

  • Changed: In the mutants.json format, replaced the function, line, and return_type fields with a function submessage (including the name and return type) and a span indicating the entire replaced region, to better handle smaller-than-function mutants. Also, the function includes the line-column span of the entire function.

23.11.2

  • Changed: If --file or --exclude are set on the command line, then they replace the corresponding config file options. Similarly, if --re is given then the examine_re config key is ignored, and if --exclude-re is given then exclude_regex is ignored. (Previously the values were combined.) This makes it easier to use the command line to test files or mutants that are normally not tested.

  • Improved: By default, files matching gitignore patterns (including in parent directories, per-user configuration, and info/exclude) are excluded from copying to temporary build directories. This should improve performance in some large trees with many files that are not part of the build. This behavior can be turned off with --gitignore=false.

  • Improved: Run cargo metadata with --no-deps, so that it doesn't download and compute dependency information, which can save time in some situations.

  • Added: Alternative aliases for command line options, so you don't need to remember if it's "regex" or "re": --regex, --examine-re, --examine-regex (all for names to include) and --exclude-regex.

  • Added: Accept --manifest-path as an alternative to -d, for consistency with other cargo commands.

23.11.1

  • New --in-diff FILE option tests only mutants that are in the diff from the given file. This is useful to avoid testing mutants from code that has not changed, either locally or in CI.

23.11.0

  • Changed: cargo mutants now tries to match the behavior of cargo test when run within a workspace. If run in a package directory, it tests only that package. If run in a workspace that is not a package (a "virtual workspace"), it tests the configured default packages, or otherwise all packages. This can all be overridden with the --package or --workspace options.

  • New: generate key-value map values from types like BTreeMap<String, Vec<u8>>.

  • Changed: Send trace messages to stderr rather stdout, in part so that it won't pollute json output.

23.10.0

  • The baseline test (with no mutants) now tests only the packages in which mutants will be generated, subject to any file or regex filters. This should both make baseline tests faster, and allow testing workspaces in which some packages have non-hermetic tests.

23.9.1

  • Mutate the known collection types BinaryHeap, BTreeSet, HashSet, LinkedList, and VecDeque to generate empty and one-element collections using T::new() and T::from_iter(..).

  • Mutate known container types like Arc, Box, Cell, Mutex, Rc, RefCell into T::new(a).

  • Mutate unknown types that look like containers or collections T<A> or T<'a, A>' and try to construct them from an A with T::from_iter, T::new, and T::from.

  • Minimum Rust version updated to 1.70.

  • Mutate Cow<'_, T> into Owned and Borrowed variants.

  • Mutate functions returning &[T] and &mut [T] to return leaked vecs of values.

  • Mutate (A, B, C, ...) into the product of all replacements for a, b, c, ...

  • The combination of options --list --diff --json is now supported, and emits a diff key in the JSON.

  • Mutate -> impl Iterator<Item = A> to produce empty and one-element iterators of the item type.

23.9.0

  • Fixed a bug causing an assertion failure when cargo-mutants was run from a subdirectory of a workspace. Thanks to Adam Chalmers!

  • Generate HttpResponse::Ok().finish() as a mutation of an Actix HttpResponse.

23.6.0

  • Generate Box::leak(Box::new(...)) as a mutation of functions returning &mut.

  • Add a concept of mutant "genre", which is included in the json listing of mutants. The only genre today is FnValue, in which a function body is replaced by a value. This will in future allow filtering by genre.

  • Recurse into return types, so that for example Result<bool> can generate Ok(true) and Ok(false), and Some<T> generates None and every generated value of T. Similarly for Box<T>, Vec<T>, Rc<T>, Arc<T>.

  • Generate specific values for integers: [0, 1] for unsigned integers, [0, 1, -1] for signed integers; [1] for NonZero unsigned integers and [1, -1] for NonZero signed integers.

  • Generate specific values for floats: [0.0, 1.0, -1.0].

  • Generate (fixed-length) array values, like [0; 256], [1; 256] using every recursively generated value for the element type.

23.5.0

"Pickled crab"

Released 2023-05-27

  • cargo mutants can now successfully test packages that transitively depend on a different version of themselves, such as itertools. Previously, cargo-mutants used the cargo --package option, which is ambiguous in this case, and now it uses --manifest-path instead.

  • Mutate functions returning &'_ str (whether a lifetime is named or not) to return "xyzzy" and "".

  • Switch to CalVer numbering.

1.2.3

Released 2023-05-05

  • Mutate functions returning String to String::new() rather than "".into(): same result but a bit more idiomatic.

  • New --leak-dirs option, for debugging cargo-mutants.

  • Update to syn 2.0, adding support for new Rust syntax.

  • Minimum supported Rust version increased to 1.65 due to changes in dependencies.

  • New --error option, to cause functions returning Result to be mutated to return the specified error.

  • New --no-config option, to disable reading .cargo/mutants.toml.

1.2.2

Released 2023-04-01

  • Don't mutate unsafe fns.

  • Don't mutate functions that never return (i.e. -> !).

  • Minimum supported Rust version increased to 1.64 due to changes in dependencies.

  • Some command-line options can now also be configured through environment variables: CARGO_MUTANTS_JOBS, CARGO_MUTANTS_TRACE_LEVEL.

  • New command line option --minimum-test-timeout and config file variable minimum_test_timeout join existing environment variable CARGO_MUTANTS_MINIMUM_TEST_TIMEOUT, to allow boosting the minimum, especially for test environments with poor or uneven throughput.

  • Changed: Renamed fields in outcomes.json from cargo_result to process_status and from command to argv.

  • Warn if no mutants were generated or if all mutants were unviable.

1.2.1

Released 2023-01-05

  • Converted most of the docs to a book available at https://mutants.rs/.

  • Fixed: Correctly find submodules that don't use mmod.rsnaming, e.g. when descending fromsrc/foo.rstosrc/foo/bar.rs. Also handle module names that are raw identifiers usingr#`. (Thanks to @kpreid for the report.)

1.2.0

Thankful mutants!

  • Fixed: Files that are excluded by filters are also excluded from --list-files.

  • Fixed: --exclude-re and --re can match against the return type as shown in --list.

  • New: A .cargo/mutants.toml file can be used to configure standard filters and cargo args for a project.

1.1.1

Released 2022-10-31

Spooky mutants!

  • Fixed support for the Mold linker, or for other options passed via RUSTFLAGS or CARGO_ENCODED_RUSTFLAGS. (See the instructions in README.md).

  • Source trees are walked by following mod statements rather than globbing the directory. This is more correct if there are files that are not referenced by mod statements. Once attributes on modules are stable in Rust (https://github.com/rust-lang/rust/issues/54727) this opens a path to skip mods using attributes.

1.1.0

Released 2022-10-30

Fearless concurrency!

  • cargo-mutants can now run multiple cargo build and test tasks in parallel, to make better use of machine resources and find mutants faster, controlled by --jobs.

  • The minimum Rust version to build cargo-mutants is now 1.63.0. It can still be used to test code under older toolchains.

1.0.3

Released 2022-09-29

  • cargo-mutants is now finds no uncaught mutants in itself! Various tests were added and improved, particularly around handling timeouts.

  • New: --re and --exclude-re options to filter by mutant name, including the path. The regexps match against the strings printed by --list.

1.0.2

Released 2022-09-24

  • New: cargo mutants --completions SHELL to generate shell completions using clap_complete.

  • Changed: cargo-mutants no longer builds in the source directory, and no longer copies the target/ directory to the scratch directory. Since cargo-mutants now sets RUSTFLAGS to avoid false failures from warnings, it is unlikely to match the existing build products in the source directory target/, and in fact building there is just likely to cause rebuilds in the source. The behavior now is as if --no-copy-target was always passed. That option is still accepted, but it has no effect.

  • Changed: cargo-mutants finds all possible mutations before doing the baseline test, so that you can see earlier how many there will be.

  • New: Set INSTA_UPDATE=no so that tests that use the Insta library don't write updates back into the source directory, and so don't falsely pass.

1.0.1

Released 2022-09-12

  • Fixed: Don't try to mutate functions within test targets, e.g. within tests/**/*.rs.

  • New: missed.txt, caught.txt, timeout.txt and unviable.txt files are written in to the output directory to make results easier to review later.

  • New: --output creates the specified directory if it does not exist.

  • Internal: Switched from Argh to Clap for command-line parsing. There may be some small changes in CLI behavior and help formatting.

1.0.0

Released 2022-08-21

A 1.0 release to celebrate that with the addition of workspace handling, cargo-mutants gives useful results on many Rust projects.

  • New: Supports workspaces containing multiple packages. Mutants are generated for all relevant targets in all packages, and mutants are subject to the tests of their own package. cargo mutants --list-files --json and cargo mutants --list --json now includes package names for each file or mutant.

  • Improved: Generate mutations in cdylib, rlib, and ever other *lib target. For example, this correctly exercises Wasm projects.

  • Improved: Write mutants.out/outcomes.json after the source-tree build and baseline tests so that it can be observed earlier on.

  • Improved: mutants.out/outcomes.json includes the commands run.

0.2.11

Released 2022-08-20

  • New --exclude command line option to exclude source files from mutants generation, matching a glob.

  • New: CARGO_MUTANTS_MINIMUM_TEST_TIMEOUT sets a minimum timeout for cargo tests, in seconds. This can be used to allow more time on slow CI builders. If unset the default is still 20s.

  • Added: A new mutants.out/debug.log with internal debugging information.

  • Improved: The time for check, build, and test is now shown separately in progress bars and output, to give a better indication of which is taking more time in the tree under test. Also, times are show in seconds with one decimal place, and they are styled more consistently.

  • Improved: More consistent use of 'unviable' and other terms for outcomes in the UI.

0.2.10

Released 2022-08-07

cargo-mutants 0.2.10 comes with improved docs, and the new -C option can be used to pass options like --release or --all-features to cargo.

  • Added: --cargo-arg (or -C for short) allows passing arguments to cargo commands (check, build, and test), for example to set --release or --features.

  • Improved: Works properly if run from a subdirectory of a crate, or if -d points to a subdirectory of a crate.

  • Improved: Various docs.

  • Improved: Relative dependencies within the source tree are left as relative paths, and will be built within the scratch directory. Relative dependencies outside the source tree are still rewritten as absolute paths.

0.2.9

Released 2022-07-30

  • Faster: cargo mutants no longer runs cargo check before building, in cases where the build products are wanted or tests will be run. This saves a significant amount of work in build phases; in some trees cargo mutants is now 30% faster. (In trees where most of the time is spent running tests the effect will be less.)

  • Fixed: Open log files in append mode to fix messages from other processes occasionally being partly overwritten.

  • Improved: cargo mutants should now give useful results in packages that use #![deny(unused)] or other mechanisms to reject warnings. Mutated functions often ignore some parameters, which would previously be rejected by this configuration without proving anything interesting about test coverage. Now, --cap-lints=allow is passed in RUSTFLAGS while building mutants, so that they're not falsely rejected and the tests can be exercised.

  • Improved: The build dir name includes the root package name.

  • Improved: The progress bar shows more information.

  • Improved: The final message shows how many mutants were tested and how long it took.

0.2.8

Released 2022-07-18

  • New: Summarize the overall number of mutants generated, caught, missed, etc, at the end.

  • Fixed: Works properly with crates that have relative path dependencies in Cargo.toml or .cargo/config.toml, by rewriting them to absolute paths in the scratch directory.

0.2.7

Released 2022-07-11

  • New: You can skip functions by adding #[cfg_attr(test, mutants::skip), in which case the mutants crate can be only a dev-dependency.

  • Improved: Don't generate pointless mutations of functions with an empty body (ignoring comments.)

  • Improved: Remove extra whitespace from the display of function names and return types: the new formatting is closer to the spacing used in idiomatic Rust.

  • Improved: Show the last line of compiler/test output while running builds, so that it's more clear where time is being spent.

  • Docs: Instructions on how to check for missed mutants from CI.

0.2.6

Released 2022-04-17

  • Improved: Find source files by looking at cargo metadata output, rather than assuming they're in src/**/*.rs. This makes cargo mutants work properly on trees where it previously failed to find the source.

  • New --version option.

  • New: Write a lock.json into the mutants.out directory including the start timestamp, cargo-mutants version, hostname and username. Take a lock on this file while cargo mutants is running, so that it doesn't crash or get confused if two tasks try to write to the same directory at the same time.

  • New: Restored a --list-files option.

  • Changed: Error if no mutants are generated, which probably indicates a bug or configuration error(?)

0.2.5

Released 2022-04-14

  • New --file command line option to mutate only functions in source files matching a glob.

  • Improved: Don't attempt to mutate functions called new or implementations of Default. cargo-mutants can not yet generate good mutations for these so they are generally false positives.

  • Improved: Better display of <impl Foo for Bar>::foo and similar type paths.

  • New: --output directory to write mutants.out somewhere other than the source directory.

0.2.4

Released 2022-03-26

  • Fix: Ignore errors setting file mtimes during copies, which can cause failures on Windows if some files are readonly.

  • Fix: Log file names now include only the source file relative path, the line number, and a counter, so they are shorter, and shouldn't cause problems on filesystems with length limits.

  • Change: version-control directories like .git are not copied with the source tree: they should have no effect on the build, so copying them is just a waste.

  • Changed/improved json logs in mutants.out:

    • Show durations as fractional seconds.

    • Outcomes include a "summary" field.

0.2.3

Released 2022-03-23

  • Switch from Indicatif to Nutmeg to draw progress bars and output. This fixes a bug where terminal output line-wraps badly, and adds a projection for the total estimated time to completion.

  • Change: Mutants are now tested in random order by default, so that repeated runs are more likely to surface interesting new findings early, rather than repeating previous results. The previous behavior of testing mutants in the deterministic order they're encountered in the tree can be restored with --no-shuffle.

0.2.2

Released 2022-02-16

  • The progress bar now shows which mutant is being tested out of how many total.

  • The automatic timeout is now set to the minimum of 20 seconds, or 5x the time of the tests in a baseline tree, to reduce the incidence of false timeouts on machines with variable throughput.

  • Ctrl-c (or SIGINT) interrupts the program during copying the tree. Previously it was not handled until the copy was complete.

  • New --no-copy-target option.

0.2.1

Released 2022-02-10

0.2.0

Released 2022-02-06

  • A new --timeout SECS option to limit the runtime of any cargo test invocation, so that mutations that cause tests to hang don't cause cargo mutants to hang.

    A default timeout is set based on the time to run tests in an unmutated tree. There is no timeout by default on the unmutated tree.

    On Unix, the cargo subprocesses run in a new process group. As a consequence ctrl-c is explicitly caught and propagated to the child processes.

  • Show a progress bar while looking for mutation opportunities, and show the total number found.

  • Show how many mutation opportunities were found, before testing begins.

  • New --shuffle option tests mutants in random order.

  • By default, the output now only lists mutants that were missed or that timed out. Mutants that were caught, and mutants that did not build, can be printed with --caught and --unviable respectively.

0.1.0

Released 2021-11-30

  • Logs and other information are written into mutants.out in the source directory, rather than target/mutants.

  • New --all-logs option prints all Cargo output to stdout, which is verbose but useful for example in CI, by making all the output directly available in captured stdout.

  • The output distinguishes check or build failures (probably due to an unviable mutant) from test failures (probably due to lacking coverage.)

  • A new file mutants.out/mutants.json lists all the generated mutants.

  • Show function return types in some places, to make it easier to understand whether the mutants were useful or viable.

  • Run cargo check --tests and cargo build --tests in the source directory to freshen the build and download any dependencies, before copying it to a scratch directory.

  • New --check option runs cargo check on generated mutants to see if they are viable, without actually running the tests. This is useful in tuning cargo-mutants to generate better mutants.

  • New --no-times output hides times (and tree sizes) from stdout, mostly to make the output deterministic and easier to match in tests.

  • Mutate methods too!

0.0.4

Released 2021-11-10

  • Fixed cargo install cargo-mutants (sometimes?) failing due to the derive feature not getting set on the serde dependency.

  • Show progress while copying the tree.

  • Respect the $CARGO environment variable so that the same toolchain is used to run tests as was used to invoke cargo mutants. Concretely, cargo +nightly mutants should work correctly.

0.0.3

Released 2021-11-06

  • Skip functions or modules marked #[test], #[cfg(test)] or #[mutants::skip].

  • Early steps towards type-guided mutations:

    • Generate mutations of true and false for functions that return bool
    • Empty and arbitrary strings for functions returning String.
    • Return Ok(Default::default()) for functions that return Result<_, _>.

  • Rename --list-mutants to just --list.

  • New --list --json.

  • Colored output makes test names and mutations easier to read (for me at least.)

  • Return distinct exit codes for different situations including that uncaught mutations were found.

0.0.2

  • Functions that should not be mutated can be marked with #[mutants::skip] from the mutants helper crate.

0.0.1

First release.