An exciting step forward: cargo-mutants can now generate mutations smaller than a whole function. To start with, several binary operators are mutated.
!=and vice versa.
||and vice versa, and mutate both of them to
Changed: If no mutants are generated then
cargo mutantsnow exits successfully, showing a warning. (Previously it would exit with an error.) This works better with
--in-diffin CI, where it's normal that some changes may not have any mutants.
Changed: Include column numbers in text listings of mutants and output to disambiguate smaller-than-function mutants, for example if there are several operators that can be changed on one line. This also applies to the names used for regex matching, so may break some regexps that match the entire line (sorry). The new option
--line-col=falseturns them both off in
Changed: In the mutants.json format, replaced the
return_typefields with a
functionsubmessage (including the name and return type) and a
spanindicating the entire replaced region, to better handle smaller-than-function mutants. Also, the
functionincludes the line-column span of the entire function.
--excludeare set on the command line, then they replace the corresponding config file options. Similarly, if
--reis given then the
examine_reconfig key is ignored, and if
--exclude-reis given then
exclude_regexis ignored. (Previously the values were combined.) This makes it easier to use the command line to test files or mutants that are normally not tested.
Improved: By default, files matching gitignore patterns (including in parent directories, per-user configuration, and
info/exclude) are excluded from copying to temporary build directories. This should improve performance in some large trees with many files that are not part of the build. This behavior can be turned off with
--no-deps, so that it doesn't download and compute dependency information, which can save time in some situations.
Added: Alternative aliases for command line options, so you don't need to remember if it's "regex" or "re":
--examine-regex(all for names to include) and
--manifest-pathas an alternative to
-d, for consistency with other cargo commands.
--in-diff FILEoption tests only mutants that are in the diff from the given file. This is useful to avoid testing mutants from code that has not changed, either locally or in CI.
cargo mutantsnow tries to match the behavior of
cargo testwhen run within a workspace. If run in a package directory, it tests only that package. If run in a workspace that is not a package (a "virtual workspace"), it tests the configured default packages, or otherwise all packages. This can all be overridden with the
New: generate key-value map values from types like
Changed: Send trace messages to stderr rather stdout, in part so that it won't pollute json output.
- The baseline test (with no mutants) now tests only the packages in which mutants will be generated, subject to any file or regex filters. This should both make baseline tests faster, and allow testing workspaces in which some packages have non-hermetic tests.
Mutate the known collection types
VecDequeto generate empty and one-element collections using
Mutate known container types like
Mutate unknown types that look like containers or collections
T<'a, A>'and try to construct them from an
Minimum Rust version updated to 1.70.
Mutate functions returning
&mut [T]to return leaked vecs of values.
(A, B, C, ...)into the product of all replacements for
a, b, c, ...
The combination of options
--list --diff --jsonis now supported, and emits a
diffkey in the JSON.
-> impl Iterator<Item = A>to produce empty and one-element iterators of the item type.
Fixed a bug causing an assertion failure when cargo-mutants was run from a subdirectory of a workspace. Thanks to Adam Chalmers!
HttpResponse::Ok().finish()as a mutation of an Actix
Box::leak(Box::new(...))as a mutation of functions returning
Add a concept of mutant "genre", which is included in the json listing of mutants. The only genre today is
FnValue, in which a function body is replaced by a value. This will in future allow filtering by genre.
Recurse into return types, so that for example
Noneand every generated value of
T. Similarly for
Generate specific values for integers:
[0, 1]for unsigned integers,
[0, 1, -1]for signed integers;
for NonZero unsigned integers and
[1, -1]for NonZero signed integers.
Generate specific values for floats:
[0.0, 1.0, -1.0].
Generate (fixed-length) array values, like
[0; 256], [1; 256]using every recursively generated value for the element type.
cargo mutantscan now successfully test packages that transitively depend on a different version of themselves, such as
itertools. Previously, cargo-mutants used the cargo
--packageoption, which is ambiguous in this case, and now it uses
Mutate functions returning
&'_ str(whether a lifetime is named or not) to return
Switch to CalVer numbering.
Mutate functions returning
"".into(): same result but a bit more idiomatic.
--leak-dirsoption, for debugging cargo-mutants.
Update to syn 2.0, adding support for new Rust syntax.
Minimum supported Rust version increased to 1.65 due to changes in dependencies.
--erroroption, to cause functions returning
Resultto be mutated to return the specified error.
--no-configoption, to disable reading
Don't mutate functions that never return (i.e.
Minimum supported Rust version increased to 1.64 due to changes in dependencies.
Some command-line options can now also be configured through environment variables:
New command line option
--minimum-test-timeoutand config file variable
minimum_test_timeoutjoin existing environment variable
CARGO_MUTANTS_MINIMUM_TEST_TIMEOUT, to allow boosting the minimum, especially for test environments with poor or uneven throughput.
Changed: Renamed fields in
Warn if no mutants were generated or if all mutants were unviable.
Converted most of the docs to a book available at https://mutants.rs/.
Fixed: Correctly find submodules that don't use mmod.rs
naming, e.g. when descending fromsrc/foo.rs
. Also handle module names that are raw identifiers usingr#`. (Thanks to @kpreid for the report.)
Fixed: Files that are excluded by filters are also excluded from
--recan match against the return type as shown in
.cargo/mutants.tomlfile can be used to configure standard filters and cargo args for a project.
Fixed support for the Mold linker, or for other options passed via
CARGO_ENCODED_RUSTFLAGS. (See the instructions in README.md).
Source trees are walked by following
modstatements rather than globbing the directory. This is more correct if there are files that are not referenced by
modstatements. Once attributes on modules are stable in Rust (https://github.com/rust-lang/rust/issues/54727) this opens a path to skip mods using attributes.
cargo-mutants can now run multiple cargo build and test tasks in parallel, to make better use of machine resources and find mutants faster, controlled by
The minimum Rust version to build cargo-mutants is now 1.63.0. It can still be used to test code under older toolchains.
cargo-mutants is now finds no uncaught mutants in itself! Various tests were added and improved, particularly around handling timeouts.
--exclude-reoptions to filter by mutant name, including the path. The regexps match against the strings printed by
cargo mutants --completions SHELLto generate shell completions using
cargo-mutantsno longer builds in the source directory, and no longer copies the
target/directory to the scratch directory. Since
RUSTFLAGSto avoid false failures from warnings, it is unlikely to match the existing build products in the source directory
target/, and in fact building there is just likely to cause rebuilds in the source. The behavior now is as if
--no-copy-targetwas always passed. That option is still accepted, but it has no effect.
cargo-mutantsfinds all possible mutations before doing the baseline test, so that you can see earlier how many there will be.
INSTA_UPDATE=noso that tests that use the Insta library don't write updates back into the source directory, and so don't falsely pass.
Fixed: Don't try to mutate functions within test targets, e.g. within
unviable.txtfiles are written in to the output directory to make results easier to review later.
--outputcreates the specified directory if it does not exist.
Internal: Switched from Argh to Clap for command-line parsing. There may be some small changes in CLI behavior and help formatting.
A 1.0 release to celebrate that with the addition of workspace handling, cargo-mutants gives useful results on many Rust projects.
New: Supports workspaces containing multiple packages. Mutants are generated for all relevant targets in all packages, and mutants are subject to the tests of their own package.
cargo mutants --list-files --jsonand
cargo mutants --list --jsonnow includes package names for each file or mutant.
Improved: Generate mutations in
rlib, and ever other
*libtarget. For example, this correctly exercises Wasm projects.
mutants.out/outcomes.jsonafter the source-tree build and baseline tests so that it can be observed earlier on.
mutants.out/outcomes.jsonincludes the commands run.
--excludecommand line option to exclude source files from mutants generation, matching a glob.
CARGO_MUTANTS_MINIMUM_TEST_TIMEOUTsets a minimum timeout for cargo tests, in seconds. This can be used to allow more time on slow CI builders. If unset the default is still 20s.
Added: A new
mutants.out/debug.logwith internal debugging information.
Improved: The time for check, build, and test is now shown separately in progress bars and output, to give a better indication of which is taking more time in the tree under test. Also, times are show in seconds with one decimal place, and they are styled more consistently.
Improved: More consistent use of 'unviable' and other terms for outcomes in the UI.
cargo-mutants 0.2.10 comes with improved docs, and the new
-C option can be used to pass options like
-Cfor short) allows passing arguments to cargo commands (check, build, and test), for example to set
Improved: Works properly if run from a subdirectory of a crate, or if
-dpoints to a subdirectory of a crate.
Improved: Various docs.
Improved: Relative dependencies within the source tree are left as relative paths, and will be built within the scratch directory. Relative dependencies outside the source tree are still rewritten as absolute paths.
cargo mutantsno longer runs
cargo checkbefore building, in cases where the build products are wanted or tests will be run. This saves a significant amount of work in build phases; in some trees
cargo mutantsis now 30% faster. (In trees where most of the time is spent running tests the effect will be less.)
Fixed: Open log files in append mode to fix messages from other processes occasionally being partly overwritten.
cargo mutantsshould now give useful results in packages that use
#![deny(unused)]or other mechanisms to reject warnings. Mutated functions often ignore some parameters, which would previously be rejected by this configuration without proving anything interesting about test coverage. Now,
--cap-lints=allowis passed in
RUSTFLAGSwhile building mutants, so that they're not falsely rejected and the tests can be exercised.
Improved: The build dir name includes the root package name.
Improved: The progress bar shows more information.
Improved: The final message shows how many mutants were tested and how long it took.
New: Summarize the overall number of mutants generated, caught, missed, etc, at the end.
Fixed: Works properly with crates that have relative
.cargo/config.toml, by rewriting them to absolute paths in the scratch directory.
New: You can skip functions by adding
#[cfg_attr(test, mutants::skip), in which case the
mutantscrate can be only a
Improved: Don't generate pointless mutations of functions with an empty body (ignoring comments.)
Improved: Remove extra whitespace from the display of function names and return types: the new formatting is closer to the spacing used in idiomatic Rust.
Improved: Show the last line of compiler/test output while running builds, so that it's more clear where time is being spent.
Docs: Instructions on how to check for missed mutants from CI.
Improved: Find source files by looking at
cargo metadataoutput, rather than assuming they're in
src/**/*.rs. This makes
cargo mutantswork properly on trees where it previously failed to find the source.
New: Write a
mutants.outdirectory including the start timestamp, cargo-mutants version, hostname and username. Take a lock on this file while
cargo mutantsis running, so that it doesn't crash or get confused if two tasks try to write to the same directory at the same time.
New: Restored a
Changed: Error if no mutants are generated, which probably indicates a bug or configuration error(?)
--filecommand line option to mutate only functions in source files matching a glob.
Improved: Don't attempt to mutate functions called
newor implementations of
Default. cargo-mutants can not yet generate good mutations for these so they are generally false positives.
Improved: Better display of
<impl Foo for Bar>::fooand similar type paths.
--outputdirectory to write
mutants.outsomewhere other than the source directory.
Fix: Ignore errors setting file mtimes during copies, which can cause failures on Windows if some files are readonly.
Fix: Log file names now include only the source file relative path, the line number, and a counter, so they are shorter, and shouldn't cause problems on filesystems with length limits.
Change: version-control directories like
.gitare not copied with the source tree: they should have no effect on the build, so copying them is just a waste.
Changed/improved json logs in
Show durations as fractional seconds.
Outcomes include a "summary" field.
Switch from Indicatif to Nutmeg to draw progress bars and output. This fixes a bug where terminal output line-wraps badly, and adds a projection for the total estimated time to completion.
Change: Mutants are now tested in random order by default, so that repeated runs are more likely to surface interesting new findings early, rather than repeating previous results. The previous behavior of testing mutants in the deterministic order they're encountered in the tree can be restored with
The progress bar now shows which mutant is being tested out of how many total.
The automatic timeout is now set to the minimum of 20 seconds, or 5x the time of the tests in a baseline tree, to reduce the incidence of false timeouts on machines with variable throughput.
SIGINT) interrupts the program during copying the tree. Previously it was not handled until the copy was complete.
- Arguments to
cargo testcan be passed on the command line after
--. This allows, for example, skipping doctests or setting the number of test threads. https://github.com/sourcefrog/cargo-mutants/issues/15
--timeout SECSoption to limit the runtime of any
cargo testinvocation, so that mutations that cause tests to hang don't cause
cargo mutantsto hang.
A default timeout is set based on the time to run tests in an unmutated tree. There is no timeout by default on the unmutated tree.
On Unix, the
cargosubprocesses run in a new process group. As a consequence ctrl-c is explicitly caught and propagated to the child processes.
Show a progress bar while looking for mutation opportunities, and show the total number found.
Show how many mutation opportunities were found, before testing begins.
--shuffleoption tests mutants in random order.
By default, the output now only lists mutants that were missed or that timed out. Mutants that were caught, and mutants that did not build, can be printed with
Logs and other information are written into
mutants.outin the source directory, rather than
--all-logsoption prints all Cargo output to stdout, which is verbose but useful for example in CI, by making all the output directly available in captured stdout.
The output distinguishes check or build failures (probably due to an unviable mutant) from test failures (probably due to lacking coverage.)
A new file
mutants.out/mutants.jsonlists all the generated mutants.
Show function return types in some places, to make it easier to understand whether the mutants were useful or viable.
cargo check --testsand
cargo build --testsin the source directory to freshen the build and download any dependencies, before copying it to a scratch directory.
cargo checkon generated mutants to see if they are viable, without actually running the tests. This is useful in tuning cargo-mutants to generate better mutants.
--no-timesoutput hides times (and tree sizes) from stdout, mostly to make the output deterministic and easier to match in tests.
Mutate methods too!
cargo install cargo-mutants(sometimes?) failing due to the
derivefeature not getting set on the
Show progress while copying the tree.
$CARGOenvironment variable so that the same toolchain is used to run tests as was used to invoke
cargo mutants. Concretely,
cargo +nightly mutantsshould work correctly.
Skip functions or modules marked
Early steps towards type-guided mutations:
- Generate mutations of
falsefor functions that return
- Empty and arbitrary strings for functions returning
Ok(Default::default())for functions that return
- Generate mutations of
Colored output makes test names and mutations easier to read (for me at least.)
Return distinct exit codes for different situations including that uncaught mutations were found.
- Functions that should not be mutated can be marked with