~/handys11
Browse the docs

dotnet/Tooling

Mutation testing with Stryker.NET

intermediate

TL;DR

dotnet tool install -g dotnet-stryker
cd tests/MyProject.Tests
dotnet stryker                    # mutate the project under test, run the suite
dotnet stryker --open-report      # inspect survived mutants in the HTML report
dotnet stryker --since:main       # CI-friendly: only mutate changed code
// stryker-config.json, next to the test project
{
  "stryker-config": {
    "project": "MyProject.csproj",
    "mutation-level": "Standard",
    "thresholds": { "high": 80, "low": 60, "break": 50 }
  }
}

How it works

Stryker mutates your source code with small, deliberate bugs — mutants — then reruns your test suite against each one. >= becomes >, + becomes -, a string literal gets emptied, a boolean flips. If a test starts failing, the mutant is killed: something in your suite noticed the behavior change. If every test still passes, the mutant survived — and survivors are the actual output you care about. A survived mutant is a concrete, reproducible piece of code whose behavior no test pins down.

Mutation score is killed / total valid mutants, and it’s a meaningfully stronger signal than line coverage. Coverage only asks whether a line executed during a test run; it says nothing about whether the test would have noticed if that line did something else entirely. A line inside an if with no matching assertion downstream is 100% covered and 0% tested — mutation testing is what catches that gap.

Mutant schemata: why it isn’t mutants × slower

Recompiling and rerunning the whole suite once per mutant would be brutal for anything past a toy project. Stryker.NET avoids the recompilation cost with mutant schemata: every mutant for a run is woven into one compiled assembly behind conditional branches, and an environment variable picked at runtime selects which single mutant, if any, is “switched on” for a given test execution. You compile once; the suite still effectively runs once per mutant, but you only pay for that multiplication in test time, not build time.

Coverage analysis trims the remaining cost further. Stryker first runs your suite once with coverage instrumentation to learn which tests exercise which lines. When it comes time to test a given mutant, it only reruns the tests that can actually reach that line — tests that can’t reach the mutant are skipped outright instead of run-and-ignored.

Gotchas

  • It’s still expensive — scope it. Mutating and re-testing a whole codebase on every run doesn’t scale past a small project. Use --since:main in CI to mutate only what a PR actually changed, and project or file filters locally when iterating on one area.
  • Equivalent mutants are a ceiling, not a bug. Some mutants change the code with no observable difference in behavior (dead branches, over-defensive checks) and can never be killed no matter how thorough your tests are. Don’t chase 100% — a healthy mutation score is well short of it.
  • An aggressive break threshold makes CI flaky as code moves. Start the break threshold low, let the score climb as you write tests against survivors, and ratchet the threshold up to match — not the other way around.
  • Not every survivor deserves a new test. Mutants surviving inside logging calls or ToString() formatting are usually noise, not gaps. Configure mutation levels or explicit ignores for that code instead of writing tests nobody wants to maintain.
  • It audits assertions, not inputs. A survived mutant usually means an existing test exercises the code but doesn’t assert on the outcome — the fix is often a missing assertion, not a missing test case.

Further reading