Sharding

In addition to running multiple jobs locally, cargo-mutants can also run jobs on multiple machines, to get an overall result faster by using more CPU cores.

Each job tests a subset of mutants, selected by a shard. Shards are described as k/n, where n is the number of shards and k is the index of the shard, from 0 to n-1.

There is no runtime coordination between shards: they each independently discover the available mutants and then select a subset based on the --shard option.

If any shard fails then that would indicate that some mutants were missed, or there was some other problem.

Consistency across shards

CAUTION: All shards must be run with the same arguments, and the same sharding denominator n, or the results will be meaningless, as they won't agree on how to divide the work.

Sharding can be combined with filters or shuffling, as long as the filters are set consistently in all shards. Sharding can also combine with --in-diff, again as long as all shards see the same diff.

Setting up sharding

Your CI system or other tooling is responsible for launching multiple shards, and for collecting the results. You're responsible for choosing the number of shards (see below).

For example, in GitHub Actions, you could use a matrix job to run multiple shards:

# Example of using a GitHub Actions matrix to shard the mutants run into 8 parts.

# See https://github.com/sourcefrog/cargo-mutants/blob/main/.github/workflows/tests.yml for a full example.

# Only run this on PRs or main branch commits that could affect the results,
# so we don't waste time on doc-only changes. Adjust these paths and branch names
# to suit your project.
on:
  pull_request:
    paths:
      - ".cargo/*.toml"
      - ".github/workflows/tests.yml"
      - "Cargo.*"
      - "mutants_attrs/**"
      - "src/**"
      - "testdata/**"
      - "tests/**"
  push:
    branches:
      - main
    # Actions doesn't support YAML references, so it's repeated here
    paths:
      - ".cargo/*.toml"
      - ".github/workflows/tests.yml"
      - "Cargo.*"
      - "mutants_attrs/**"
      - "src/**"
      - "testdata/**"
      - "tests/**"

jobs:
  # Before testing mutants, run the build and tests on all platforms.
  # You probably already have CI configuration like this, so don't duplicate it,
  # merge cargo-mutants into your existing workflow.
  test:
    strategy:
      matrix:
        os: [macOS-latest, ubuntu-latest, windows-latest]
        version: [stable, nightly]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@master
        with:
          toolchain: ${{ matrix.version }}
          components: rustfmt
      - uses: swatinem/rust-cache@v2
      - name: rustfmt
        run: cargo fmt --all -- --check
      - name: Build
        run: cargo build --all-targets
      - name: Test
        run: cargo test --workspace
  cargo-mutants:
    runs-on: ubuntu-latest
    # Often you'll want to only run this after the build is known to pass its basic tests,
    # to avoid wasting time, and to allow using --baseline=skip.
    needs: [test]
    strategy:
      fail-fast: false # Collect all mutants even if some are missed
      matrix:
        shard: [0, 1, 2, 3, 4, 5, 6, 7]
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@master
      - uses: Swatinem/rust-cache@v2
      - uses: taiki-e/install-action@v2
        name: Install cargo-mutants using install-action
        with:
          tool: cargo-mutants
      # Set an appropriate timeout for your tree here.
      # The denominator of the shard count must be the number of shards.
      - name: Mutants
        run: |
          cargo mutants --no-shuffle -vV --shard ${{ matrix.shard }}/8 --baseline=skip --timeout 300 --in-place
      - name: Archive mutants.out
        uses: actions/upload-artifact@v4
        if: always()
        with:
          path: mutants.out
          name: mutants-shard${{matrix.shard}}.out

Note that the number of shards is set to match the /8 in the --shard argument.

Skipping the baseline

Sharding works with --baseline=skip, to avoid the cost of running the baseline on every shard. But, if you do this, then you must ensure that the tests suite is passing in the baseline, for example by checking it in a previous CI step.

Performance of sharding

Each mutant does some constant upfront work:

  • Any CI setup including starting the machine, getting a checkout, installing a Rust toolchain, and installing cargo-mutants
  • An initial clean build of the code under test
  • A baseline run of the unmutated code (unless this is skipped)

Then, for each mutant in its shard, it does an incremental build and runs all the tests.

Each shard runs the same number of mutants, +/-1. Typically this will mean they each take roughly the same amount of time, although it's possible that some shards are unlucky in drawing mutants that happen to take longer to test.

A rough model for the overall execution time for all of the shards, allowing for this work occurring in parallel, is

SHARD_STARTUP + (CLEAN_BUILD + TEST) + (N_TOTAL_MUTANTS / N_SHARDS) * (INCREMENTAL_BUILD + TEST)

The total cost in CPU seconds can be modelled as:

N_SHARDS * (SHARD_STARTUP + CLEAN_BUILD + TEST) + N_MUTANTS * (INCREMENTAL_BUILD + TEST)

As a result, if you use many shards the cost of the initial build will dominate, and the overall time will converge towards the time for a clean build, a baseline test, and the test of one mutant.

Choosing a number of shards

Because there's some constant overhead for every shard there will be diminishing returns and increasing ineffiency if you use too many shards. (In the extreme cases where there are more shards than mutants, some of them will find they have nothing to do and immediately exit.)

As a rule of thumb, you should probably choose n such that each worker runs at least 10 mutants, and possibly much more. 8 to 32 shards might be a good place to start.

The optimal setting probably depends on how long your tree takes to build from zero and incrementally, how long the tests take to run, and the performance of your CI system.

If your CI system offers a choice of VM sizes you might experiment with using smaller or larger VMs and more or less shards: the optimal setting probably also depends on your tree's ability to exploit larger machines.

You should also think about cost and capacity constraints in your CI system, and the risk of starving out other users.

cargo-mutants has no internal scaling constraints to prevent you from setting n very large, if cost, efficiency and CI capacity are not a concern.

Sampling mutants

An option like --shard 1/100 can be used to run 1% of all the generated mutants for testing cargo-mutants, to get a sense of whether it works or to see how it performs on some tree.