Welcome to cargo-mutants
cargo-mutants is a mutation testing tool for Rust. It helps you improve your program's quality by finding places where bugs can be inserted causing any tests to fail.
The goal of cargo-mutants is to be easy to run on any Rust source tree, and to tell you something interesting about areas where bugs might be lurking or the tests might be insufficient. (More about these goals.)
To get started:
- Install cargo-mutants.
- Run `cargo mutants in your Rust source tree.
For more resources see the repository at https://github.com/sourcefrog/cargo-mutants.
Installation
Install cargo-mutants from source:
cargo install --locked cargo-mutants
Supported Rust versions
Building cargo-mutants requires a reasonably recent stable (or nightly or beta) Rust toolchain. The supported version is visible in https://crates.io/crates/cargo-mutants.
After installing cargo-mutants, you should be able to use it to run tests under
any toolchain, even toolchains that are far too old to build cargo-mutants, using the standard +
option to cargo
:
cargo +1.48 mutants
Getting started
Just run cargo mutants
in a Rust source directory, and it will point out
functions that may be inadequately tested.
Example
; cargo mutants
Found 14 mutants to test
Copy source to scratch directory ... 0 MB in 0.0s
Unmutated baseline ... ok in 1.6s build + 0.3s test
Auto-set test timeout to 20.0s
src/lib.rs:386: replace <impl Error for Error>::source -> Option<&(dyn std::error::Error + 'static)>
with Default::default() ... NOT CAUGHT in 0.6s build + 0.3s test
src/lib.rs:485: replace copy_symlink -> Result<()> with Ok(Default::default()) ...
NOT CAUGHT in 0.5s build + 0.3s test
14 mutants tested in 0:08: 2 missed, 9 caught, 3 unviable
In v0.5.1 of the cp_r
crate, the copy_symlink
function was reached by a test
but not adequately tested, and the Error::source
function was not tested at all.
Using the results
Tests fail in an clean tree?
If tests fail in a clean copy of the tree, there might be an (intermittent)
failure in the source directory, or there might be some problem that stops them
passing when run from a different location. Fix this first: cargo-mutants can't do anything until you have a tree where cargo test
passes reliably when copied to a temporary directory.
Mutant outcomes
Assuming tests pass in a clean copy of the tree, cargo-mutants proceeds to generate every mutant it can, subject to any configured filters, and then runs cargo build
and cargo test
on each of them.
Each mutant results in one of the following outcomes:
-
caught — A test failed with this mutant applied. This is a good sign about test coverage.
-
missed — No test failed with this mutation applied, which seems to indicate a gap in test coverage. Or, it may be that the mutant is undistinguishable from the correct code. You may wish to add a better test, or mark that the function should be skipped.
-
unviable — The attempted mutation doesn't compile. This is inconclusive about test coverage and no action is needed, but indicates an opportunity for cargo-mutants to either generate better mutants, or at least not generate unviable mutants.
-
timeout — The mutation caused the test suite to run for a long time, until it was eventually killed. You might want to investigate the cause and potentially mark the function to be skipped.
By default only missed mutants and timeouts are printed to stdout, because they're the most actionable. Others can be shown with the --caught
and --unviable
options.
What to do about missed mutants?
Each missed mutant is a sign that there might be a gap in test coverage. What to do about them is up to you, bearing in mind your goals and priorities for your project, but here are some suggestions:
First, look at the overall list of missed mutants: there might be patterns such as a cluster of related functions all having missed mutants. Probably some will stand out as potentially more important to the correct function of your program.
You should first look for any mutations where it's very surprising that they
were not caught by any tests, given what you know about the codebase. For
example, if cargo-mutants reports that replacing an important function with
Ok(())
is not caught then that seems important to investigate.
You should then look at the tests that you would think would catch the mutant: that might be unit tests within the relevant module, or some higher-level public-API or integration test, depending on how your project's tests are structured.
If you can't find any tests that you think should have caught the mutant, then perhaps you should add some. The right thing here is not necessarily to directly assert that the mutated behavior doesn't happen. For example, if the mutant changed a private function, you don't necessarily want to add a test for that private function, but instead ask yourself what public-API behavior would break if the private function was buggy, and then add a test for that.
Try to avoid writing tests that are too tightly targeted to the mutant, which is really just an example of something that could be wrong, and instead write tests that assert the correct behavior at the right level of abstraction, preferably through a public interface.
If it's not clear why the tests aren't already failing, it may help to manually
inject the same mutation into your working tree and then run the tests under a
debugger, or add trace statements to the test. (The --diff
option or looking
in the mutants.out
directory will show you exactly what change cargo-mutants
made.)
You may notice some messages about missed mutants in functions that you feel are
not very important to test, such as Debug
implementations. You can use the
--exclude-re
options to filter out these mutants, or mark them as
skipped with #[mutants::skip]
. (Or, you might decide that you do want to add
unit tests for the Debug
representation, but perhaps as a lower priority than
investigating mutants in more important code.)
In some cases cargo-mutants will generate a mutant that is effectively the same as the original code, and so not really incorrect. cargo-mutants tries to avoid doing this, but if it does happen then you can mark the function as skipped.
Iterating on mutant coverage
After you've changed your program to address some of the missed mutants, you can
run cargo mutants
again with the --file
option to re-test
only functions from the changed files.
Hard-to-test cases
Some functions don't cause a test suite failure if emptied, but also cannot be removed. For example, functions to do with managing caches or that have other performance side effects.
Ideally, these should be tested, but doing so in a way that's not flaky can be difficult. cargo-mutants can help in a few ways:
- It helps to at least highlight to the developer that the function is not covered by tests, and so should perhaps be treated with extra care, or tested manually.
- A
#[mutants::skip]
annotation can be added to suppress warnings and explain the decision. - Sometimes these effects can be tested by making the side-effect observable with, for example, a counter of the number of memory allocations or cache misses/hits.
Hangs and timeouts
Some mutations to the tree can cause the test suite to hang. For example, in
this code, cargo-mutants might try changing should_stop
to always return
false
, but this will cause the program to hang:
#![allow(unused)] fn main() { while !should_stop() { // something } }
In general you will want to skip functions which cause a hang when mutated, either by marking them with an attribute or in the configuration file.
Timeouts
To avoid hangs, cargo-mutants will kill the test suite after a timeout and continue to the next mutant.
By default, the timeout is set automatically. cargo-mutants measures the time to run the test suite in the unmodified tree, and then sets a timeout for mutated tests at 5x the time to run tests with no mutations, and a minimum of 20 seconds.
The minimum of 20 seconds can be overriden by the
CARGO_MUTANTS_MINIMUM_TEST_TIMEOUT
environment variable, measured in seconds.
You can also set an explicit timeout with the --timeout
option, also measured
in seconds. If this option is specified then the timeout is also applied to the
unmutated tests.
The timeout does not apply to cargo check
or cargo build
, only cargo test
.
Exit codes
cargo-mutants returns an exit code that can be used by scripts or CI.
-
0: Success! Every viable mutant that was tested was caught by a test.
-
1: Usage error: bad command-line arguments etc.
-
2: Found some mutants that were not covered by tests.
-
3: Some tests timed out: possibly the mutatations caused an infinite loop, or the timeout is too low.
-
4: The tests are already failing or hanging before any mutations are applied, so no mutations were tested.
For more detailed machine-readable information, use the mutants.out
directory.
The mutants.out
directory
A mutants.out
directory is created in the original source directory. You can put the output directory elsewhere with the --output
option.
On each run, any existing mutants.out
is renamed to mutants.out.old
, and any
existing mutants.out.old
is deleted.
The output directory contains:
-
A
lock.json
, on which an fs2 lock is held while cargo-mutants is running, to avoid two tasks trying to write to the same directory at the same time.lock.json
contains the start time, cargo-mutants version, username, and hostname.lock.json
is left inmutants.out
when the run completes, but the lock on it is released. -
A
mutants.json
file describing all the generated mutants. This file is completely written before testing begins. -
An
outcomes.json
file describing the results of all tests, and summary counts of each outcome. -
A
logs/
directory, with one log file for each mutation plus the baseline unmutated case. The log contains the diff of the mutation plus the output from cargo.outcomes.json
includes for each mutant the name of the log file. -
caught.txt
,missed.txt
,timeout.txt
,unviable.txt
, each listing mutants with the corresponding outcome.
The contents of the directory and the format of these files is subject to change in future versions.
These files are incrementally updated while cargo-mutants runs, so other programs can read them to follow progress.
There is generally no reason to include this directory in version control, so it is recommended that you add /mutants.out*
to your .gitignore
file. This will exclude both mutants.out
and mutants.out.old
.
Skipping untestable code
Some functions may be inherently hard to cover with tests, for example if:
- Generated mutants cause tests to hang.
- You've chosen to test the functionality by human inspection or some higher-level integration tests.
- The function has side effects or performance characteristics that are hard to test.
- You've decided the function is not important to test.
There are three ways to skip mutating some code:
- Marking the function with an attribute within the source file.
- Filtering by path in the config file or command line.
- Filtering by function and mutant name in the config file or command line.
The results of all these filters can be previewed using the --list
option.
Which filtering method to use?
- If some particular functions are hard to test with cargo-mutants, use an attribute, so that the skip is visible in the code.
- If a whole module is untestable, use a filter by path in the config file, so that the filter's stored in the source tree and covers any new code in that module.
- If you want to permanently ignore a class of functions, such as
Debug
implementations, use a regex filter in the config file. - If you want to run cargo-mutants just once, focusing on a subset of files, functions, or mutants, use command line options to filter by name or path.
Skipping functions with an attribute
To mark functions as skipped, so they are not mutated:
-
Add a Cargo dependency on the mutants crate, version "0.0.3" or later. (This must be a regular
dependency
not adev-dependency
, because the annotation will be on non-test code.) -
Mark functions with
#[mutants::skip]
or other attributes containingmutants::skip
(e.g.#[cfg_attr(test, mutants::skip)]
).
The mutants
create is tiny and the attribute has no effect on the compiled
code. It only flags the function for cargo-mutants. However, you can avoid the
dependency by using the slightly longer #[cfg_attr(test, mutants::skip)]
form.
Note: Currently, cargo-mutants
does not (yet) evaluate attributes like
cfg_attr
, it only looks for the sequence mutants::skip
in the attribute.
You may want to also add a comment explaining why the function is skipped.
For example:
#![allow(unused)] fn main() { use std::time::{Duration, Instant}; /// Returns true if the program should stop #[cfg_attr(test, mutants::skip)] // Returning false would cause a hang fn should_stop() -> bool { true } pub fn controlled_loop() { let start = Instant::now(); for i in 0.. { println!("{}", i); if should_stop() { break; } if start.elapsed() > Duration::from_secs(60 * 5) { panic!("timed out"); } } } mod test { #[test] fn controlled_loop_terminates() { super::controlled_loop() } } }
Filtering files
Two options (each with short and long names) control which files are mutated:
-
-f GLOB
,--file GLOB
: Mutate only functions in files matching the glob. -
-e GLOB
,--exclude GLOB
: Exclude files that match the glob.
These options may be repeated.
If any -f
options are given, only source files that match are
considered; otherwise all files are considered. This list is then further
reduced by exclusions.
If the glob contains /
(or on Windows, \
), then it matches against the path from the root of the source
tree. For example, src/*/*.rs
will exclude all files in subdirectories of src
.
If the glob does not contain a path separator, it matches against filenames
in any directory. /
matches the path separator on both Unix and Windows.
Note that the glob must contain .rs
(or a matching wildcard) to match
source files with that suffix. For example, -f network
will match
src/network/mod.rs
but it will not match src/network.rs
.
Files that are excluded are still parsed (and so must be syntactically
valid), and mod
statements in them are followed to discover other
source files. So, for example, you can exclude src/main.rs
but still
test mutants in other files referenced by mod
statements in main.rs
.
Since Rust does not currently allow attributes such as #[mutants::skip]
on mod
statements or at module scope filtering by filename is the only way to skip an entire module.
The results of filters can be previewed with the --list-files
and --list
options.
Examples:
-
cargo mutants -f visit.rs -f change.rs
-- test mutants only in files calledvisit.rs
orchange.rs
(in any directory). -
cargo mutants -e console.rs
-- test mutants in any file exceptconsole.rs
. -
cargo mutants -f src/db/*.rs
-- test mutants in any file in this directory.
Configuring filters by filename
Files may also be filtered with the exclude_globs
and examine_globs
options in .cargo/mutants.toml
.
Exclusions in the config file may be particularly useful when there are modules that are inherently hard to automatically test, and the project has made a decision to accept lower test coverage for them.
From cargo-mutants 23.11.2 onwards, if the command line options are given then the corresponding config file option is ignored. This allows you to use the config file to test files that are normally expected to pass, and then to use the command line to test files that are not yet passing.
For example:
exclude_globs = ["src/main.rs", "src/cache/*.rs"] # like -e
examine_globs = ["src/important/*.rs"] # like -f: test *only* these files
Filtering functions and mutants
You can filter mutants by name, using the --re
and --exclude-re
command line
options and the corresponding examine_re
and exclude_re
config file options.
These options are useful if you want to run cargo-mutants just once, focusing on a subset of functions or mutants.
These options filter mutants by the full name of the mutant, which includes the
function name, file name, and a description of the change, as shown in the output of cargo mutants --list
.
For example, one mutant name might be:
src/outcome.rs:157: replace <impl Serialize for ScenarioOutcome>::serialize -> Result<S::Ok, S::Error> with Ok(Default::default())
Within this name, your regex can match any substring, including for example:
- The filename
- The trait,
impl Serialize
- The struct name,
ScenarioOutcome
- The function name,
serialize
- The mutated return value,
with Ok(Defualt::default())
, or any part of it.
The regex matches a substring, but can be anchored with ^
and $
to require that
it match the whole name.
The regex syntax is defined by the regex
crate.
These filters are applied after filtering by filename, and --re
is applied before
--exclude-re
.
Examples:
-
-E 'impl Debug'
-- don't testimpl Debug
methods, because coverage of them might be considered unimportant. -
-F 'impl Serialize' -F 'impl Deserialize'
-- test implementations of these two traits.
Configuring filters by name
Mutants can be filtered by name in the .cargo/mutants.toml
file. The exclude_re
and examine_re
keys are each a list of strings.
This can be helpful if you want to systematically skip testing implementations of certain traits, or functions with certain names.
From cargo-mutants 23.11.2 onwards, if the command line options are given then the corresponding config file option is ignored.
For example:
exclude_re = ["impl Debug"] # same as -E
Controlling cargo-mutants
cargo mutants
takes various options to control how it runs.
These options, can, in general, be passed on the command line, set in a .cargo/mutants.toml
file in the source tree, or passed in CARGO_MUTANTS_
environment variables. Not every
method of setting an option is available for every option, however, as some would not
make sense or be useful.
For options that take a list of values, values from the configuration file are appended to values from the command line.
For options that take a single value, the value from the command line takes precedence.
--no-config
can be used to disable reading the configuration file.
Execution order
By default, mutants are run in a randomized order, so as to surface results from
different parts of the codebase earlier. This can be disabled with
--no-shuffle
, in which case mutants will run in order by file name and within each file in the order they appear in
the source.
Source directory location
-d
, --dir
: Test the Rust tree in the given directory, rather than the source tree
enclosing the working directory where cargo-mutants is launched.
Console output
-v
, --caught
: Also print mutants that were caught by tests.
-V
, --unviable
: Also print mutants that failed cargo build
.
--no-times
: Don't print elapsed times.
-L
, --level
, and $CARGO_MUTANTS_TRACE_LEVEL
: set the verbosity of trace
output to stdout. The default is info
, and it can be increased to debug
or
trace
.
Listing generated mutants
--list
: Show what mutants could be generated, without running them.
--diff
: With --list
, also include a diff of the source change for each mutant.
--json
: With --list
, show the list in json for easier processing by other programs.
(The same format is written to mutants.out/mutants.json
when running tests.)
--check
: Run cargo check
on all generated mutants to find out which ones are viable, but don't actually run the tests. (This is primarily useful when debugging cargo-mutants.)
Workspace and package support
cargo-mutants supports testing Cargo workspaces that contain multiple packages. The entire workspace tree is copied.
By default, cargo-mutants has the same behavior as Cargo:
- If
--workspace
is given, all packages in the workspace are tested. - If
--package
is given, the named packages are tested. - If the starting directory (or
-d
directory) is in a package, that package is tested. - Otherwise, the starting directory must be in a virtual workspace. If it specifies default members, they are tested. Otherwise, all packages are tested.
For each mutant, only the containing package's tests are run, on the theory that each package's tests are responsible for testing the package's code.
The baseline tests exercise all and only the packages for which mutants will be generated.
You can also use the --file
options to restrict cargo-mutants to testing only files
from some subdirectory, e.g. with -f "utils/**/*.rs"
. (Remember to quote globs
on the command line, so that the shell doesn't expand them.) You can use --list
or
--list-files
to preview the effect of filters.
Passing options to Cargo
cargo-mutants runs cargo build
and cargo test
(or, with --check
, it runs
cargo check
.) Additional options can be passed in three different ways: to all
cargo
commands; to cargo test
only; and to the test binaries run by cargo test
.
There is not yet a way to pass options only to cargo build
but not to cargo test
.
Arguments to all cargo
commands
To pass more arguments to every Cargo invocation, use --cargo-arg
, or the additional_cargo_args
configuration key.
--cargo-arg
can be repeated.
For example
cargo mutants -- --cargo-arg=--release
or in .cargo/mutants.toml
:
additional_cargo_args = ["--all-features"]
Arguments to cargo test
Command-line options following a --
delimiter are passed through to
cargo test
. For example, this can be used to pass --all-targets
which (unobviously)
excludes doctests. (If the doctests are numerous and slow, and not relied upon to catch bugs, this can improve performance.)
cargo mutants -- --all-targets
These options can also be configured statically with the additional_cargo_test_args
key in .cargo/mutants.toml
:
additional_cargo_test_args = ["--jobs=1"]
Arguments to test binaries
You can use a second double-dash to pass options through to the test targets:
cargo mutants -- -- --test-threads 1 --nocapture
(However, this may interact poorly with using additional_cargo_test_args
in the configuration file,
as the argument lists are currently appended without specially handling the --
separator.)
Build directories
cargo-mutants builds mutated code in a temporary directory, containing a copy of your source tree with each mutant successively applied. With --jobs
, multiple build directories are used in parallel.
Build-in ignores
Files or directories matching these patterns are not copied:
.git
.hg
.bzr
.svn
_darcs
.pijul
gitignore
From 23.11.2, by default, cargo-mutants will not copy files that are excluded by gitignore patterns, to make copying faster in large trees.
This behavior can be turned off with --gitignore=false
.
Generating mutants
cargo mutants generates mutants by inspecting the existing source code and applying a set of rules to generate new code that is likely to compile but have different behavior.
Mutants each have a "genre", each of which is described below.
Replace function body with value
The FnValue
genre of mutants replaces a function's body with a value that is guessed to be of the right type.
This checks that the tests:
- Observe any side effects of the original function.
- Distinguish return values.
More mutation genres and patterns will be added in future releases.
Return type | Mutation pattern |
---|---|
() | () (return unit, with no side effects) |
signed integers | 0, 1, -1 |
unsigned integers | 0, 1 |
floats | 0.0, 1.0, -1.0 |
NonZeroI* | 1, -1 |
NonZeroU* | 1 |
bool | true , false |
String | String::new() , "xyzzy".into() |
&'_ str . | "" , "xyzzy" |
&mut ... | Box::leak(Box::new(...)) |
Result<T> | Ok(...) , and an error if configured |
Option<T> | Some(...) , None |
Box<T> | Box::new(...) |
Vec<T> | vec![] , vec![...] |
Arc<T> | Arc::new(...) |
Rc<T> | Rc::new(...) |
BinaryHeap , BTreeSet , HashSet , LinkedList , VecDeque | empty and one-element collections |
BTreeMap , HashMap | empty map and the product of all key and value replacements |
Cow<'_, T> | Cow::Borrowed(t) , Cow::Owned(t.to_owned()) |
[T; L] | [r; L] for all replacements of T |
&[T] , &mut [T] | Leaked empty and one-element vecs |
&T | &... (all replacements for T) |
HttpResponse | HttpResponse::Ok().finish |
(A, B, ...) | (a, b, ...) for the product of all replacements of A, B, ... |
impl Iterator | Empty and one-element iterators of the inner type |
(any other) | Default::default() |
...
in the mutation patterns indicates that the type is recursively mutated.
For example, Result<bool>
can generate Ok(true)
and Ok(false)
.
The recursion can nest for types like Result<Option<String>>
.
Some of these values may not be valid for all types: for example, returning
Default::default()
will work for many types, but not all. In this case the
mutant is said to be "unviable": by default these are counted but not printed,
although they can be shown with --unviable
.
Binary operators
Binary operators are replaced with other binary operators in expressions
like a == 0
.
Operator | Replacements |
---|---|
== | != |
!= | == |
&& | \|\| , == , != |
\|\| | && , == , != |
< | == , > |
> | == , < |
<= | == , >= |
>= | == , <= |
Equality operators are not currently replaced with comparisons like <
or <=
because they are
too prone to generate false positives, for example when unsigned integers are compared to 0.
Logical &&
and ||
are replaced with ==
and !=
which function as XNOR and XOR respectively,
although they are fairly often unviable due to needing parenthesis when the original operator does not.
Generating error values
cargo-mutants can be configured to generate mutants that return an error value from functions that return a Result.
This will flag cases where no test fails if the function returns an error: that might happen if there are only tests for the error cases and not for the Ok case.
Since crates can choose to use any type for their error values, cargo-mutants must be told how to construct an appropriate error.
The --error
command line option and the error_value
configuration option specify an error value to use.
These options can be repeated or combined, which might be useful if there are multiple error types in the crate. On any one mutation site, probably only one of the error values will be viable, and cargo-mutants will discover that and use it.
The error value can be any Rust expression that evaluates to a value of the error type. It should not include the Err
wrapper, because cargo-mutants will add that.
For example, if your crate uses anyhow::Error
as its error type, you might use --error '::anyhow::anyhow!("error")'
.
If you have your own error type, you might use --error 'crate::MyError::Generic'
.
Since the correct error type is a property of the source tree, the configuration should typically go into .cargo/mutants.toml
rather than being specified on the command line:
error_values = ["::anyhow::anyhow!(\"mutated\")"]
To see only the mutants generated by this configuration, you can use a command like this:
cargo r mutants -F anyhow -vV -j4
Improving performance
Most of the runtime for cargo-mutants is spent in running the program test suite and in running incremental builds: both are done once per viable mutant.
So, anything you can do to make the cargo build
and cargo test
suite faster
will have a multiplicative effect on cargo mutants
run time, and of course
will also make normal development more pleasant.
https://matklad.github.io/2021/09/04/fast-rust-builds.html has good general advice on making Rust builds and tests faster.
Avoid doctests
Rust doctests are pretty slow, because every doctest example becomes a separate
test binary. If you're using doctests only as testable documentation and not to
assert correctness of the code, you can skip them with cargo mutants -- --all-targets
.
Optimized builds
On some but not all projects, cargo-mutants can be faster if you use -C --release
, which will make the build slower but may make the tests faster. Typically this will help on projects with very long CPU-intensive test suites.
cargo-mutants now shows the breakdown of build versus test time which may help you work out if this will help: if the tests are much slower than the build it's worth trying more more compiler optimizations.
On projects like this you might also choose just to turn up optimization for all debug builds in .cargo/config.toml
.
Ramdisks
cargo-mutants causes the Rust toolchain (and, often, the program under test) to read and write many temporary files. Setting the temporary directory onto a ramdisk can improve performance significantly. This is particularly important with parallel builds, which might otherwise hit disk bandwidth limits. For example on Linux:
sudo mkdir /ram
sudo mount -t tmpfs /ram /ram # or put this in fstab, or just change /tmp
sudo chmod 1777 /ram
env TMPDIR=/ram cargo mutants
Using the Mold linker
Using the Mold linker on Unix can give a 20% performance improvement, depending on the tree. Because cargo-mutants does many incremental builds, link time is important, especially if the test suite is relatively fast.
Because of limitations in the way cargo-mutants runs Cargo, the standard way of configuring Mold for Rust in ~/.cargo/config.toml
won't work.
Instead, set the RUSTFLAGS
environment variable to -Clink-arg=-fuse-ld=mold
.
Parallelism
After the initial test of the unmutated tree, cargo-mutants can test multiple mutants in parallel. This can give significant performance improvements, depending on the tree under test and the hardware resources available.
Even though cargo builds, rustc, and Rust's test framework launch multiple
processes or threads, they typically can't use all available CPU cores all the
time, and many cargo test
runs will end up using only one core waiting for the
last task to complete. Running multiple jobs in parallel makes use of resources
that would otherwise be idle.
By default, only one job is run at a time.
To run more, use the --jobs
or -j
option, or set the CARGO_MUTANTS_JOBS
environment variable.
Setting this higher than the number of CPU cores is unlikely to be helpful.
The best setting will depend on many factors including the behavior of your program's test suite, the amount of memory on your system, and your system's behavior under high thermal load.
-j 4
may be a good starting point. Start there and watch memory and CPU usage,
and tune towards a setting where all cores are fully utilized without apparent
thrashing, memory exhaustion, or thermal issues.
Because tests may be slower with high parallelism, you may see some spurious
timeouts, and you may need to set --timeout
manually to allow enough safety
margin.
Incremental tests of modified code
If you're working on a large project or one with a long test suite, you may not want to test the entire codebase every time you make a change. You can use cargo-mutants --in-diff
to test only mutants generated from recently changed code.
The --in-diff DIFF_FILE
option tests only mutants that overlap with regions changed in the diff.
The diff is expected to either have a prefix of b/
on the new filename, which is the format produced by git diff
, or no prefix.
Some ways you could use --in-diff
:
- Before submitting code, check your uncommitted changes with
git diff
. - In CI, or locally, check the diff between the current branch and the base branch of the pull request.
Changes to non-Rust files, or files from which no mutants are produced, are ignored.
--in-diff
is applied on the output of other filters including --package
and --regex
. For example, cargo mutants --in-diff --package foo
will only test mutants in the foo
package that overlap with the diff.
Caution
--in-diff
makes tests faster by covering the mutants that are most likely to be missed in the changed code. However, it's certainly possible that edits in one region cause code in a different region or a different file to no longer be well tested. Incremental tests are helpful for giving faster feedback, but they're not a substitute for a full test run.
Example
In this diff, we've added a new function two
to src/lib.rs
, and the existing code is unaltered. With --in-diff
, cargo-mutants
will only test mutants that affect the function two
.
```diff
--- a/src/lib.rs 2023-11-12 13:05:25.774658230 -0800
+++ b/src/lib.rs 2023-11-12 12:54:04.373806696 -0800
@@ -2,6 +2,10 @@
"one".to_owned()
}
+pub fn two() -> String {
+ format!("{}", 2)
+}
+
#[cfg(test)]
mod test_super {
use super::*;
@@ -10,4 +14,9 @@
fn test_one() {
assert_eq!(one(), "one");
}
+
+ #[test]
+ fn test_two() {
+ assert_eq!(two(), "2");
+ }
}
Integrations
Shell completion
The --completions SHELL
emits completion scripts for the given shell.
The right place to install these depends on your shell and operating system.
For example, for Fish1:
cargo mutants --completions fish >~/.config/fish/conf.d/cargo-mutants-completions.fish
This command installs them to conf.d
instead of completions
because you may have completions for several cargo
plugins.
vim-cargomutants
vim-cargomutants
provides commands
view cargo-mutants results, see the diff of mutations, and to launch cargo-mutants
from within vim.
Continuous integration
Here is an example of a GitHub Actions workflow that runs mutation tests and uploads the results as an artifact. This will fail if it finds any uncaught mutants.
name: cargo-mutants
on: [pull_request, push]
jobs:
cargo-mutants:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions-rs/toolchain@v1
with:
toolchain: stable
- name: Install cargo-mutants
run: cargo install --locked cargo-mutants
- name: Run mutant tests
run: cargo mutants -- --all-features
- name: Archive results
uses: actions/upload-artifact@v3
if: failure()
with:
name: mutation-report
path: mutants.out
The workflow used by cargo-mutants on itself can be seen at https://github.com/sourcefrog/cargo-mutants/blob/main/.github/workflows/mutate-self.yaml.
Incremental tests of pull requests
You can use --in-diff
to test only the code that has changed in a pull request. This can be useful for incremental testing in CI, where you want to test only the code that has changed since the last commit.
For example, you can use the following workflow to test only the code that has changed in a pull request:
name: Tests
permissions:
contents: read
on:
push:
branches:
- main
pull_request:
jobs:
incremental-mutants:
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Relative diff
run: |
git branch -av
git diff origin/${{ github.base_ref }}.. | tee git.diff
- uses: Swatinem/rust-cache@v2
- run: cargo install cargo-mutants
- name: Mutants
run: |
cargo mutants --no-shuffle -j 2 -vV --in-diff git.diff
- name: Archive mutants.out
uses: actions/upload-artifact@v3
if: always()
with:
name: mutants-incremental.out
path: mutants.out
How cargo-mutants works
The basic approach is:
-
Build a list of mutations:
- Run
cargo metadata
to find directories containing Rust source files. - Walk all source files and parse each one looking for functions.
- Skip functions that should not be mutated for any of several reasons:
because they're tests, because they have a
#[mutants::skip]
attribute, etc. - For each function, depending on its return type, generate every mutation pattern that produces a result of that type.
- Run
-
Make a copy of the source tree into a scratch directory, excluding version-control directories like
.git
and the/target
directory. The same directory is reused across all the mutations to benefit from incremental builds.- After copying the tree, cargo-mutants scans the top-level
Cargo.toml
and any.cargo/config.toml
for relative dependencies. If there are any, the paths are rewritten to be absolute, so that they still work when cargo is run in the scratch directory. - Before applying any mutations, check that
cargo test
succeeds in the scratch directory: perhaps a test is already broken, or perhaps the tree doesn't build when copied because it relies on relative paths to find dependencies, etc. - If running more than one parallel job, make the appropriate number of additional scratch directories.
- After copying the tree, cargo-mutants scans the top-level
-
For each mutation:
- Apply the mutation to the scratch tree by patching the affected file.
- Run
cargo build
: if this fails, the mutant is unviable, and that's ok. - Run
cargo test
in the tree, saving output to a log file. - If the the tests fail, that's good: the mutation was somehow caught.
- If the tests succeed, that might mean test coverage was inadequate, or it might mean we accidentally generated a no-op mutation.
- Revert the mutation to return the tree to its clean state.
The file is parsed using the syn
crate, but mutations
are applied textually, rather than to the token stream, so that unmutated code
retains its prior formatting, comments, line numbers, etc. This makes it
possible to show a text diff of the mutation and should make it easier to
understand any error messages from the build of the mutated code.
For more details, see DESIGN.md.
Goals
The goal of cargo-mutants is to be easy to run on any Rust source tree, and to tell you something interesting about areas where bugs might be lurking or the tests might be insufficient.
The detailed goals in this section are intended to generally guide development priorities and tradeoffs. For example, the goal of ease means that we will generally prefer to automatically infer reasonable default behavior rather than requiring the user to configure anything at first. The goal of being interesting means that we will generally only enable by default features that seem reasonably likely to say something important about test quality to at least some users.
Ease
Being easy to use means:
-
cargo-mutants requires no changes to the source tree or other setup: just install and run. So, if it does not find anything interesting to say about a well-tested tree, it didn't cost you much. (This worked out really well:
cargo install cargo-mutants && cargo mutants
will do it.) -
There is no chance that running cargo-mutants will change the released behavior of your program (other than by helping you to fix bugs!), because you don't need to change the source to use it.
-
cargo-mutants should be reasonably fast even on large Rust trees. The overall run time is, roughly, the product of the number of viable mutations multiplied by the time to run the test suite for each mutation. Typically, one
cargo mutants
run will give you all the information it can find about missing test coverage in the tree, and you don't need to run it again as you iterate on tests, so it's relatively OK if it takes a while.(There is currently very little overhead beyond the cost to do an incremental build and run the tests for each mutant, but that can still take a while for large trees that produce many mutants especially if their test suite takes a while.)
-
cargo-mutants should run correctly on any Rust source trees that are built and tested by Cargo, that will build and run their tests in a copy of the tree, and that have hermetic tests.
-
cargo-mutants shouldn't crash or hang, even if it generates mutants that cause the software under test to crash or hang.
-
The results should be reproducible, assuming the build and test suite is deterministic.
-
cargo-mutants should avoid generating unviable mutants that don't compile, because that wastes time. However, when it's uncertain whether the mutant will build, it's worth trying things that might find interesting results even if they might fail to build. (It does currently generate some unviable mutants, but typically not too many, and they don't have a large effect on runtime in most trees.)
-
Realistically, cargo-mutants may generate some mutants that aren't caught by tests but also aren't interesting, or aren't feasible to test. In those cases it should be easy to permanently dismiss them (e.g. by adding a
#[mutants::skip]
attribute or a config file.)
Interestingness
Showing interesting results mean:
-
cargo-mutants should tell you about places where the code could be wrong and the test suite wouldn't catch it. If it doesn't find any interesting results on typical trees, there's no point. Aspirationally, it will even find useful results in code with high line coverage, when there is code that is reached by a test, but no test depends on its behavior.
-
In superbly-tested projects cargo-mutants may find nothing to say, but hey, at least it was easy to run, and hopefully the assurance that the tests really do seem to be good is useful data.
-
Most, ideally all, findings should indicate something that really should be tested more, or that may already be buggy, or that's at least worth looking at.
-
It should be easy to understand what the output is telling you about a potential bug that wouldn't be caught. (This seems true today.) It might take some thought to work out why the existing tests don't cover it, or how to check it, but at least you know where to begin.
-
As much as possible cargo-mutants should avoid generating trivial mutants, where the mutated code is effectively equivalent to the original code, and so it's not interesting that the test suite doesn't catch the change.
-
For trees that are thoroughly tested, you can use
cargo mutants
in CI to check that they remain so.
How is mutation testing different to coverage measurement?
Coverage measurements tell you which lines of code (or other units) are reached while running a test. They don't tell you whether the test really checks anything about the behavior of the code.
For example, a function that writes a file and returns a Result
might be
covered by a test that checks the return value, but not by a test that checks
that the file was actually written. cargo-mutants will try mutating the function
to simply return Ok(())
and report that this was not caught by any tests.
Historically, rust coverage measurements have required manual setup of several
OS and toolchain-dependent tools, although this is improving. Because
cargo-mutants
just runs cargo
it has no OS-specific or tight toolchain
integrations, and so is simple to install and run on any Rust source tree.
cargo-mutants also needs no special tools to view or interpret the results.
Coverage tools also in some cases produce output that is hard to interpret, with lines sometimes shown as covered or not due to toolchain quirks that aren't easy to map to direct changes to the test suite. cargo-mutants produces a direct list of changes that are not caught by the test suite, which can be quickly reviewed and prioritized.
One drawback of mutation testing is that it runs the whole test suite once per generated mutant, so it can be slow on large trees with slow test suites. There are some techniques to speed up cargo-mutants, including running multiple tests in parallel.
How is mutation testing different to fuzzing?
Fuzzing is a technique for finding bugs by feeding pseudo-random inputs to a program, and is particularly useful on programs that parse complex or untrusted inputs such as binary file formats or network protocols.
Mutation testing makes algorithmically-generated changes to a copy of the program source, and measures whether the test suite catches the change.
The two techniques are complementary. Although some bugs might be found by either technique, fuzzing will tend to find bugs that are triggered by complex or unusual inputs, whereas mutation testing will tend to point out logic that might be correct but that's not tested.
Limitations, caveats, known bugs, and future enhancements
Cases where cargo-mutants can't help
cargo-mutants can only help if the test suite is hermetic: if the tests are flaky or non-deterministic, or depend on external state, it will draw the wrong conclusions about whether the tests caught a bug.
If you rely on testing the program's behavior by manual testing, or by an
integration test not run by cargo test
, then cargo-mutants can't know this,
and will only tell you about gaps in the in-tree tests. It may still be helpful
to run mutation tests on only some selected modules that do have in-tree tests.
Running cargo-mutants on your code won't, by itself, make your code better. It only helps suggest places you might want to improve your tests, and that might indirectly find bugs, or prevent future bugs. Sometimes the results will point out real current bugs. But it's on you to follow up. (However, it's really easy to run, so you might as well look!)
cargo-mutants typically can't do much to help with crates that primarily generate code using macros or build scripts, because it can't "see" the code that's generated. (You can still run it, but it's may generate very few mutants.)
Stability
cargo-mutants behavior, output formats, command-line syntax, json output formats, etc, may change from one release to the next.
Limitations and known bugs
cargo-mutants currently only supports mutation testing of Rust code that builds
using cargo
and where the tests are run using cargo test
. Support for other tools such as Bazel or Nextest could in principle be added.
cargo-mutants sees the AST of the tree but doesn't fully "understand" the types, so sometimes generates unviable mutants or misses some opportunities to generate interesting mutants.
cargo-mutants reads CARGO_ENCODED_RUSTFLAGS
and RUSTFLAGS
environment variables, and sets CARGO_ENCODED_RUSTFLAGS
. It does not read .cargo/config.toml
files, and so any rust flags set there will be ignored.
cargo-mutants does not yet understand conditional compilation, such as
#[cfg(target_os = "linux")]
. It will report functions for other platforms as
missed, when it should know to skip them.
Support for other build tools
cargo-mutants currently only works with Cargo, but could in principle be extended to work with other build tools such as Bazel.
cargo-mutants contains two main categories of code, which are mostly independent:
-
Code for reading Rust source code, parsing it, and mutating it: this is not specific to Cargo.
-
Code for finding the modules to mutate and their source files, finding the tree to copy, adjusting paths after it is copied, and finally running builds and tests. This is very Cargo-specific, but should not be too hard to generalize.
The main precondition for supporting Bazel is a realistc test case: preferably an open source Rust tree built with Bazel, or at least a contributor with a Bazel-based Rust tree who is willing to help test and debug and to produce some test cases.
(See https://github.com/sourcefrog/cargo-mutants/issues/77 for more discussion.)
Caution on side effects
cargo-mutants builds and runs code with machine-generated modifications. This is generally fine, but if the code under test has side effects such as writing or deleting files, running it with mutations might conceivably have unexpected effects, such as deleting the wrong files, in the same way that a bug might.
If you're concerned about this, run cargo-mutants in a container or virtual machine.
cargo-mutants never modifies the original source tree, other than writing a
mutants.out
directory, and that can be sent elsewhere with the --output
option. All mutations are applied and tested in a copy of the source tree.
How to help
Experience reports in GitHub Discussions or issues are very welcome:
- Did it find a bug or important coverage gap?
- Did it fail to build and test your tree? (Some cases that aren't supported yet are already listed in this doc or the bug tracker.)
It's especially helpful if you can either point to an open source tree that will reproduce the problem (or success) or at least describe how to reproduce it.
If you are interested in contributing a patch, please read CONTRIBUTING.md.
cargo-mutants changelog
23.12.0
An exciting step forward: cargo-mutants can now generate mutations smaller than a whole function. To start with, several binary operators are mutated.
-
New: Mutate
==
to!=
and vice versa. -
New: Mutate
&&
to||
and vice versa, and mutate both of them to==
and!=
. -
New: Mutate
<
,<=
,>
,>=
. -
Changed: If no mutants are generated then
cargo mutants
now exits successfully, showing a warning. (Previously it would exit with an error.) This works better with--in-diff
in CI, where it's normal that some changes may not have any mutants. -
Changed: Include column numbers in text listings of mutants and output to disambiguate smaller-than-function mutants, for example if there are several operators that can be changed on one line. This also applies to the names used for regex matching, so may break some regexps that match the entire line (sorry). The new option
--line-col=false
turns them both off in--list
output. -
Changed: In the mutants.json format, replaced the
function
,line
, andreturn_type
fields with afunction
submessage (including the name and return type) and aspan
indicating the entire replaced region, to better handle smaller-than-function mutants. Also, thefunction
includes the line-column span of the entire function.
23.11.2
-
Changed: If
--file
or--exclude
are set on the command line, then they replace the corresponding config file options. Similarly, if--re
is given then theexamine_re
config key is ignored, and if--exclude-re
is given thenexclude_regex
is ignored. (Previously the values were combined.) This makes it easier to use the command line to test files or mutants that are normally not tested. -
Improved: By default, files matching gitignore patterns (including in parent directories, per-user configuration, and
info/exclude
) are excluded from copying to temporary build directories. This should improve performance in some large trees with many files that are not part of the build. This behavior can be turned off with--gitignore=false
. -
Improved: Run
cargo metadata
with--no-deps
, so that it doesn't download and compute dependency information, which can save time in some situations. -
Added: Alternative aliases for command line options, so you don't need to remember if it's "regex" or "re":
--regex
,--examine-re
,--examine-regex
(all for names to include) and--exclude-regex
. -
Added: Accept
--manifest-path
as an alternative to-d
, for consistency with other cargo commands.
23.11.1
- New
--in-diff FILE
option tests only mutants that are in the diff from the given file. This is useful to avoid testing mutants from code that has not changed, either locally or in CI.
23.11.0
-
Changed:
cargo mutants
now tries to match the behavior ofcargo test
when run within a workspace. If run in a package directory, it tests only that package. If run in a workspace that is not a package (a "virtual workspace"), it tests the configured default packages, or otherwise all packages. This can all be overridden with the--package
or--workspace
options. -
New: generate key-value map values from types like
BTreeMap<String, Vec<u8>>
. -
Changed: Send trace messages to stderr rather stdout, in part so that it won't pollute json output.
23.10.0
- The baseline test (with no mutants) now tests only the packages in which mutants will be generated, subject to any file or regex filters. This should both make baseline tests faster, and allow testing workspaces in which some packages have non-hermetic tests.
23.9.1
-
Mutate the known collection types
BinaryHeap
,BTreeSet
,HashSet
,LinkedList
, andVecDeque
to generate empty and one-element collections usingT::new()
andT::from_iter(..)
. -
Mutate known container types like
Arc
,Box
,Cell
,Mutex
,Rc
,RefCell
intoT::new(a)
. -
Mutate unknown types that look like containers or collections
T<A>
orT<'a, A>'
and try to construct them from anA
withT::from_iter
,T::new
, andT::from
. -
Minimum Rust version updated to 1.70.
-
Mutate
Cow<'_, T>
intoOwned
andBorrowed
variants. -
Mutate functions returning
&[T]
and&mut [T]
to return leaked vecs of values. -
Mutate
(A, B, C, ...)
into the product of all replacements fora, b, c, ...
-
The combination of options
--list --diff --json
is now supported, and emits adiff
key in the JSON. -
Mutate
-> impl Iterator<Item = A>
to produce empty and one-element iterators of the item type.
23.9.0
-
Fixed a bug causing an assertion failure when cargo-mutants was run from a subdirectory of a workspace. Thanks to Adam Chalmers!
-
Generate
HttpResponse::Ok().finish()
as a mutation of an ActixHttpResponse
.
23.6.0
-
Generate
Box::leak(Box::new(...))
as a mutation of functions returning&mut
. -
Add a concept of mutant "genre", which is included in the json listing of mutants. The only genre today is
FnValue
, in which a function body is replaced by a value. This will in future allow filtering by genre. -
Recurse into return types, so that for example
Result<bool>
can generateOk(true)
andOk(false)
, andSome<T>
generatesNone
and every generated value ofT
. Similarly forBox<T>
,Vec<T>
,Rc<T>
,Arc<T>
. -
Generate specific values for integers:
[0, 1]
for unsigned integers,[0, 1, -1]
for signed integers;[1]
for NonZero unsigned integers and[1, -1]
for NonZero signed integers. -
Generate specific values for floats:
[0.0, 1.0, -1.0]
. -
Generate (fixed-length) array values, like
[0; 256], [1; 256]
using every recursively generated value for the element type.
23.5.0
"Pickled crab"
Released 2023-05-27
-
cargo mutants
can now successfully test packages that transitively depend on a different version of themselves, such asitertools
. Previously, cargo-mutants used the cargo--package
option, which is ambiguous in this case, and now it uses--manifest-path
instead. -
Mutate functions returning
&'_ str
(whether a lifetime is named or not) to return"xyzzy"
and""
. -
Switch to CalVer numbering.
1.2.3
Released 2023-05-05
-
Mutate functions returning
String
toString::new()
rather than"".into()
: same result but a bit more idiomatic. -
New
--leak-dirs
option, for debugging cargo-mutants. -
Update to syn 2.0, adding support for new Rust syntax.
-
Minimum supported Rust version increased to 1.65 due to changes in dependencies.
-
New
--error
option, to cause functions returningResult
to be mutated to return the specified error. -
New
--no-config
option, to disable reading.cargo/mutants.toml
.
1.2.2
Released 2023-04-01
-
Don't mutate
unsafe
fns. -
Don't mutate functions that never return (i.e.
-> !
). -
Minimum supported Rust version increased to 1.64 due to changes in dependencies.
-
Some command-line options can now also be configured through environment variables:
CARGO_MUTANTS_JOBS
,CARGO_MUTANTS_TRACE_LEVEL
. -
New command line option
--minimum-test-timeout
and config file variableminimum_test_timeout
join existing environment variableCARGO_MUTANTS_MINIMUM_TEST_TIMEOUT
, to allow boosting the minimum, especially for test environments with poor or uneven throughput. -
Changed: Renamed fields in
outcomes.json
fromcargo_result
toprocess_status
and fromcommand
toargv
. -
Warn if no mutants were generated or if all mutants were unviable.
1.2.1
Released 2023-01-05
-
Converted most of the docs to a book available at https://mutants.rs/.
-
Fixed: Correctly find submodules that don't use mmod.rs
naming, e.g. when descending from
src/foo.rsto
src/foo/bar.rs. Also handle module names that are raw identifiers using
r#`. (Thanks to @kpreid for the report.)
1.2.0
Thankful mutants!
-
Fixed: Files that are excluded by filters are also excluded from
--list-files
. -
Fixed:
--exclude-re
and--re
can match against the return type as shown in--list
. -
New: A
.cargo/mutants.toml
file can be used to configure standard filters and cargo args for a project.
1.1.1
Released 2022-10-31
Spooky mutants!
-
Fixed support for the Mold linker, or for other options passed via
RUSTFLAGS
orCARGO_ENCODED_RUSTFLAGS
. (See the instructions in README.md). -
Source trees are walked by following
mod
statements rather than globbing the directory. This is more correct if there are files that are not referenced bymod
statements. Once attributes on modules are stable in Rust (https://github.com/rust-lang/rust/issues/54727) this opens a path to skip mods using attributes.
1.1.0
Released 2022-10-30
Fearless concurrency!
-
cargo-mutants can now run multiple cargo build and test tasks in parallel, to make better use of machine resources and find mutants faster, controlled by
--jobs
. -
The minimum Rust version to build cargo-mutants is now 1.63.0. It can still be used to test code under older toolchains.
1.0.3
Released 2022-09-29
-
cargo-mutants is now finds no uncaught mutants in itself! Various tests were added and improved, particularly around handling timeouts.
-
New:
--re
and--exclude-re
options to filter by mutant name, including the path. The regexps match against the strings printed by--list
.
1.0.2
Released 2022-09-24
-
New:
cargo mutants --completions SHELL
to generate shell completions usingclap_complete
. -
Changed:
cargo-mutants
no longer builds in the source directory, and no longer copies thetarget/
directory to the scratch directory. Sincecargo-mutants
now setsRUSTFLAGS
to avoid false failures from warnings, it is unlikely to match the existing build products in the source directorytarget/
, and in fact building there is just likely to cause rebuilds in the source. The behavior now is as if--no-copy-target
was always passed. That option is still accepted, but it has no effect. -
Changed:
cargo-mutants
finds all possible mutations before doing the baseline test, so that you can see earlier how many there will be. -
New: Set
INSTA_UPDATE=no
so that tests that use the Insta library don't write updates back into the source directory, and so don't falsely pass.
1.0.1
Released 2022-09-12
-
Fixed: Don't try to mutate functions within test targets, e.g. within
tests/**/*.rs
. -
New:
missed.txt
,caught.txt
,timeout.txt
andunviable.txt
files are written in to the output directory to make results easier to review later. -
New:
--output
creates the specified directory if it does not exist. -
Internal: Switched from Argh to Clap for command-line parsing. There may be some small changes in CLI behavior and help formatting.
1.0.0
Released 2022-08-21
A 1.0 release to celebrate that with the addition of workspace handling, cargo-mutants gives useful results on many Rust projects.
-
New: Supports workspaces containing multiple packages. Mutants are generated for all relevant targets in all packages, and mutants are subject to the tests of their own package.
cargo mutants --list-files --json
andcargo mutants --list --json
now includes package names for each file or mutant. -
Improved: Generate mutations in
cdylib
,rlib
, and ever other*lib
target. For example, this correctly exercises Wasm projects. -
Improved: Write
mutants.out/outcomes.json
after the source-tree build and baseline tests so that it can be observed earlier on. -
Improved:
mutants.out/outcomes.json
includes the commands run.
0.2.11
Released 2022-08-20
-
New
--exclude
command line option to exclude source files from mutants generation, matching a glob. -
New:
CARGO_MUTANTS_MINIMUM_TEST_TIMEOUT
sets a minimum timeout for cargo tests, in seconds. This can be used to allow more time on slow CI builders. If unset the default is still 20s. -
Added: A new
mutants.out/debug.log
with internal debugging information. -
Improved: The time for check, build, and test is now shown separately in progress bars and output, to give a better indication of which is taking more time in the tree under test. Also, times are show in seconds with one decimal place, and they are styled more consistently.
-
Improved: More consistent use of 'unviable' and other terms for outcomes in the UI.
0.2.10
Released 2022-08-07
cargo-mutants 0.2.10 comes with improved docs, and the new -C
option can be used to pass options like --release
or --all-features
to cargo
.
-
Added:
--cargo-arg
(or-C
for short) allows passing arguments to cargo commands (check, build, and test), for example to set--release
or--features
. -
Improved: Works properly if run from a subdirectory of a crate, or if
-d
points to a subdirectory of a crate. -
Improved: Various docs.
-
Improved: Relative dependencies within the source tree are left as relative paths, and will be built within the scratch directory. Relative dependencies outside the source tree are still rewritten as absolute paths.
0.2.9
Released 2022-07-30
-
Faster:
cargo mutants
no longer runscargo check
before building, in cases where the build products are wanted or tests will be run. This saves a significant amount of work in build phases; in some treescargo mutants
is now 30% faster. (In trees where most of the time is spent running tests the effect will be less.) -
Fixed: Open log files in append mode to fix messages from other processes occasionally being partly overwritten.
-
Improved:
cargo mutants
should now give useful results in packages that use#![deny(unused)]
or other mechanisms to reject warnings. Mutated functions often ignore some parameters, which would previously be rejected by this configuration without proving anything interesting about test coverage. Now,--cap-lints=allow
is passed inRUSTFLAGS
while building mutants, so that they're not falsely rejected and the tests can be exercised. -
Improved: The build dir name includes the root package name.
-
Improved: The progress bar shows more information.
-
Improved: The final message shows how many mutants were tested and how long it took.
0.2.8
Released 2022-07-18
-
New: Summarize the overall number of mutants generated, caught, missed, etc, at the end.
-
Fixed: Works properly with crates that have relative
path
dependencies inCargo.toml
or.cargo/config.toml
, by rewriting them to absolute paths in the scratch directory.
0.2.7
Released 2022-07-11
-
New: You can skip functions by adding
#[cfg_attr(test, mutants::skip)
, in which case themutants
crate can be only adev-dependency
. -
Improved: Don't generate pointless mutations of functions with an empty body (ignoring comments.)
-
Improved: Remove extra whitespace from the display of function names and return types: the new formatting is closer to the spacing used in idiomatic Rust.
-
Improved: Show the last line of compiler/test output while running builds, so that it's more clear where time is being spent.
-
Docs: Instructions on how to check for missed mutants from CI.
0.2.6
Released 2022-04-17
-
Improved: Find source files by looking at
cargo metadata
output, rather than assuming they're insrc/**/*.rs
. This makescargo mutants
work properly on trees where it previously failed to find the source. -
New
--version
option. -
New: Write a
lock.json
into themutants.out
directory including the start timestamp, cargo-mutants version, hostname and username. Take a lock on this file whilecargo mutants
is running, so that it doesn't crash or get confused if two tasks try to write to the same directory at the same time. -
New: Restored a
--list-files
option. -
Changed: Error if no mutants are generated, which probably indicates a bug or configuration error(?)
0.2.5
Released 2022-04-14
-
New
--file
command line option to mutate only functions in source files matching a glob. -
Improved: Don't attempt to mutate functions called
new
or implementations ofDefault
. cargo-mutants can not yet generate good mutations for these so they are generally false positives. -
Improved: Better display of
<impl Foo for Bar>::foo
and similar type paths. -
New:
--output
directory to writemutants.out
somewhere other than the source directory.
0.2.4
Released 2022-03-26
-
Fix: Ignore errors setting file mtimes during copies, which can cause failures on Windows if some files are readonly.
-
Fix: Log file names now include only the source file relative path, the line number, and a counter, so they are shorter, and shouldn't cause problems on filesystems with length limits.
-
Change: version-control directories like
.git
are not copied with the source tree: they should have no effect on the build, so copying them is just a waste. -
Changed/improved json logs in
mutants.out
:-
Show durations as fractional seconds.
-
Outcomes include a "summary" field.
-
0.2.3
Released 2022-03-23
-
Switch from Indicatif to Nutmeg to draw progress bars and output. This fixes a bug where terminal output line-wraps badly, and adds a projection for the total estimated time to completion.
-
Change: Mutants are now tested in random order by default, so that repeated runs are more likely to surface interesting new findings early, rather than repeating previous results. The previous behavior of testing mutants in the deterministic order they're encountered in the tree can be restored with
--no-shuffle
.
0.2.2
Released 2022-02-16
-
The progress bar now shows which mutant is being tested out of how many total.
-
The automatic timeout is now set to the minimum of 20 seconds, or 5x the time of the tests in a baseline tree, to reduce the incidence of false timeouts on machines with variable throughput.
-
Ctrl-c (or
SIGINT
) interrupts the program during copying the tree. Previously it was not handled until the copy was complete. -
New
--no-copy-target
option.
0.2.1
Released 2022-02-10
- Arguments to
cargo test
can be passed on the command line after--
. This allows, for example, skipping doctests or setting the number of test threads. https://github.com/sourcefrog/cargo-mutants/issues/15
0.2.0
Released 2022-02-06
-
A new
--timeout SECS
option to limit the runtime of anycargo test
invocation, so that mutations that cause tests to hang don't causecargo mutants
to hang.A default timeout is set based on the time to run tests in an unmutated tree. There is no timeout by default on the unmutated tree.
On Unix, the
cargo
subprocesses run in a new process group. As a consequence ctrl-c is explicitly caught and propagated to the child processes. -
Show a progress bar while looking for mutation opportunities, and show the total number found.
-
Show how many mutation opportunities were found, before testing begins.
-
New
--shuffle
option tests mutants in random order. -
By default, the output now only lists mutants that were missed or that timed out. Mutants that were caught, and mutants that did not build, can be printed with
--caught
and--unviable
respectively.
0.1.0
Released 2021-11-30
-
Logs and other information are written into
mutants.out
in the source directory, rather thantarget/mutants
. -
New
--all-logs
option prints all Cargo output to stdout, which is verbose but useful for example in CI, by making all the output directly available in captured stdout. -
The output distinguishes check or build failures (probably due to an unviable mutant) from test failures (probably due to lacking coverage.)
-
A new file
mutants.out/mutants.json
lists all the generated mutants. -
Show function return types in some places, to make it easier to understand whether the mutants were useful or viable.
-
Run
cargo check --tests
andcargo build --tests
in the source directory to freshen the build and download any dependencies, before copying it to a scratch directory. -
New
--check
option runscargo check
on generated mutants to see if they are viable, without actually running the tests. This is useful in tuning cargo-mutants to generate better mutants. -
New
--no-times
output hides times (and tree sizes) from stdout, mostly to make the output deterministic and easier to match in tests. -
Mutate methods too!
0.0.4
Released 2021-11-10
-
Fixed
cargo install cargo-mutants
(sometimes?) failing due to thederive
feature not getting set on theserde
dependency. -
Show progress while copying the tree.
-
Respect the
$CARGO
environment variable so that the same toolchain is used to run tests as was used to invokecargo mutants
. Concretely,cargo +nightly mutants
should work correctly.
0.0.3
Released 2021-11-06
-
Skip functions or modules marked
#[test]
,#[cfg(test)]
or#[mutants::skip]
. -
Early steps towards type-guided mutations:
- Generate mutations of
true
andfalse
for functions that returnbool
- Empty and arbitrary strings for functions returning
String
. - Return
Ok(Default::default())
for functions that returnResult<_, _>
.
- Generate mutations of
-
Rename
--list-mutants
to just--list
. -
New
--list --json
. -
Colored output makes test names and mutations easier to read (for me at least.)
-
Return distinct exit codes for different situations including that uncaught mutations were found.
0.0.2
- Functions that should not be mutated can be marked with
#[mutants::skip]
from themutants
helper crate.
0.0.1
First release.