Parallelism

After the initial test of the unmutated tree, cargo-mutants can run multiple builds and tests of the tree in parallel on a single machine. Separately, you can shard work across multiple machines.

Caution: cargo build and cargo test internally spawn many threads and processes and can be very resource hungry. Don't set --jobs too high, or your machine may thrash, run out of memory, or overheat.

Background

Even though cargo builds, rustc, and Rust's test framework launch multiple processes or threads, they typically spend some time waiting for straggler tasks, during which time some CPU cores are idle. For example, a cargo build commonly ends up waiting for a single-threaded linker for several seconds.

Running one or more build or test tasks in parallel can use up this otherwise wasted capacity. This can give significant performance improvements, depending on the tree under test and the hardware resources available.

Timeouts

Because tests may be slower with high parallelism, or may exhibit more variability in execution time, you may see some spurious timeouts, and you may need to set --timeout manually to allow enough safety margin. (User feedback on this is welcome.)

Non-hermetic tests

If your test suite is non-hermetic -- for example, if it talks to an external database -- then running multiple jobs in parallel may cause test flakes. cargo-mutants is just running multiple copies of cargo test simultaneously: if that doesn't work in your tree, then you can't use this option.

Choosing a job count

You should set the number of jobs very conservatively, starting at -j2 or -j3.

Higher settings are only likely to be helpful on very large machines, perhaps with >100 cores and >256GB RAM.

Unlike with make, setting -j proportionally to the number of cores is unlikely to work out well, because so the Rust build and test tools already parallelize very aggressively.

The best setting will depend on many factors including the behavior of your program's test suite, the amount of memory on your system, and your system's behavior under high load. Ultimately you'll need to experiment to find the best setting.

To tune the number of jobs, you can watch htop or some similar program while the tests are running, to see whether cores are fully utilized or whether the system is running out of memory. On laptop or desktop machines you might also want to watch the temperature of the CPU.

As well as using more CPU and RAM, higher -j settings will also use more disk space in your temporary directory: Rust target directories can commonly be 2GB or more, and there will be one per parallel job, plus whatever temp files your test suite might create.

Interaction with --test-threads

The Rust test framework exposes a --test-threads option controlling how many threads run inside a test binary. cargo-mutants doesn't set this, but you can set it from the command line, along with other parameters to the test binary. You might need to set this if your test suite is non-hermetic with regard to global process state.

Limiting the number of threads inside a single test binary would tend to make that binary less resource-hungry, and so might allow you to set a higher -j option.

Reducing the number of test threads to increase -j seems unlikely to help performance in most trees.