Repository navigation
Description
@edelauna — this is a follow-up to #6144 with (1) an independently verified minimal reproduction on Stryker 10.0.0 (both with and without testFiles), (2) a controlled experiment that isolates the defect, and (3) a secondary finding in the vitest-runner plugin that #6144 does not cover. It is deliberately not a duplicate: the original report (9.6.1) is reproduced here on 10.0.0, and the new evidence narrows the root cause further.
Environment
@stryker-mutator/[email protected],@stryker-mutator/[email protected]- vitest 4.1.9
- Node v25.9.0 on Windows. The defect is pure control-flow timing; expected to reproduce identically on Linux CI.
What happens
When the config sets testFiles, static mutants (top-level values evaluated at module import time) are reported as Survived even when the tests provably kill them. The mutant is never activated during the mutant run: the instrumented top-level code executes with activeMutant === undefined.
Minimal reproduction (verified on Stryker 10.0.0)
Standalone project, 4 files. Install:
pnpm add -D @stryker-mutator/[email protected] @stryker-mutator/[email protected] [email protected]src/value.ts:
export const featureEnabled = false;src/value.spec.ts:
import { featureEnabled } from './value';
import { expect, it } from 'vitest';
it('featureEnabled defaults to false', () => {
expect(featureEnabled).toBe(false);
});stryker.config.mjs:
export default {
plugins: ['@stryker-mutator/vitest-runner'],
testRunner: 'vitest',
testFiles: ['src/value.spec.ts'], // <- presence of this option triggers the bug
mutate: ['src/value.ts'],
thresholds: { break: null },
};npx stryker runActual output — the mutant is not killed (false Survived), even though the single test asserts the exact value:
WARN MutantTestPlanner Detected 1 static mutants (100% of total) that are estimated to take 100% of the time running the tests!
...
[Survived] BooleanLiteral
src/value.ts:1:31
File | total | covered | # killed | # timeout | # survived | # no cov | # errors |
All files | 0.00 | 0.00 | 0 | 0 | 1 | 0 | 0 |
value.ts | 0.00 | 0.00 | 0 | 0 | 1 | 0 | 0 |
mutation.json: BooleanLiteral, status: "Survived", static: true, replacement: "true".
Control run — identical project with the testFiles line removed:
✓ featureEnabled defaults to false (killed 1)
File | total | covered | # killed | # timeout | # survived | # no cov | # errors |
All files | 100.00 | 100.00 | 1 | 0 | 0 | 0 | 0 |
value.ts | 100.00 | 100.00 | 1 | 0 | 0 | 0 | 0 |
The only difference between the two runs is the testFiles option. The same false-Survived shape also reproduces with top-level object literals containing nested booleans (config-map / registry idiom): all such static mutants are reported Survived while the tests kill them.
Root cause chain (10.0.0 dist)
-
@stryker-mutator/core,dist/src/mutants/mutant-test-planner.js,createMutantRunPlan():const canHotSwap = !!testFilter && isStatic === false; // computed, but NOT used for activation ... mutantActivation: testFilter ? 'runtime' : 'static', // line 148
For a static mutant the planner takes the "run all tests" branch with
testFilter = this.globalTestFilter— a non-empty array built fromproject.testFileswhentestFilesis configured — and computesmutantActivation: 'runtime'. Presence of a test filter is conflated with the mutant being a runtime mutant.isStaticis already in scope (and is used correctly by thecanHotSwapcomputed on the line just above). -
@stryker-mutator/vitest-runner,dist/src/stryker-setup.js: formutantActivation !== 'static', activation happens only inbeforeAll:beforeAll(() => { ns.activeMutant = inject('activeMutant'); });
beforeAllruns after the spec module imports — i.e. after the instrumented top-level code has already been evaluated. The wrapper readsns.activeMutantat import time, getsundefined, evaluates the original value, and the tests trivially pass → falseSurvived. Only the'static'branch (top-level assignment in the setup file, which runs before the imports) can activate a static mutant.Net: static mutant +
testFilesconfigured ⇒'runtime'activation ⇒ mutant never active ⇒ falseSurvived. WithouttestFiles,globalTestFilterisundefined, the planner happens to pick'static', and the bug is hidden (as the control run above shows).Note: the instrumenter's mutant wrapper contains a fallback to
process.env.__STRYKER_ACTIVE_MUTANT__, but a grep of the entire@stryker-mutator/[email protected]dist finds zero references to that variable — core never sets it, so the fallback is dead code as far as core is concerned.
Additional evidence (from a real-world repo, reproducible on request)
- The dry run detects the mutants as 100% static (empty perTest coverage, static coverage present); they live in top-level object literals.
- Manual mutation proof: applying each mutation by hand and running the spec fails 2 of 11 tests per mutant — the tests DO kill them.
- Sandbox probe: a probe spec run inside Stryker's own sandbox, activated via the
__STRYKER_ACTIVE_MUTANT__env-var fallback built into the instrumenter wrapper, shows the mutations applying correctly. Sandbox, instrumentation, and tests are all healthy; only the activation path is broken. - Worker instrumentation: instrumenting the worker-side setup file shows
inject('mutantActivation')returning"runtime"for the 100% static mutants, andns.activeMutantbeing inserted into the namespace only inbeforeAll— after the import-timemutantCoveragewrites — soactiveMutantisundefinedexactly when the instrumented top-level code evaluates. - Controlled experiment (decisive): identical run, but with the setup file forced to assign
ns.activeMutantat top level (static activation) unconditionally — the only change. Result: all mutantsKilled(status: Killed, static: true). Activation timing is the sole defect; there is no test-coverage gap and no instrumentation defect.
Secondary finding: reloadEnvironment capability declared but not implemented
- Core sends
reloadEnvironment: truein the run options for static mutants (mutant-test-planner.js:reloadEnvironment: !canHotSwap, preserved throughReloadEnvironmentDecoratorwhen the capability is advertised). VitestTestRunner.capabilities()returns{ reloadEnvironment: true }.- But
VitestTestRunner.mutantRun()never readsoptions.reloadEnvironment: the single long-lived vitest instance and its worker/module state are never reset between runs.
In the experiments above the module registry was re-evaluated per run, so this did not cause false results, but the contract the decorator relies on is broken: either implement the reload in mutantRun(), or drop the capability so the decorator falls back to recover().
Impact
Any project using the vitest runner with testFiles gets false Survived for every top-level constant / registry / config-map mutation — exactly the code shape produced by adding feature-flag, experiment, or schema entries. For us this breaks a mutation-diff CI gate on every PR that adds such an entry, despite the tests killing the mutants. The ignoreStatic workaround is not an option for a gate: it silently skips the entire static-mutant class (the 10.0.0 planner even warns about it).
Proposed fixes
-
Planner (primary, as proposed in Static mutants are activated too late when --testFiles produces a test filter #6144) — use the already-computed capability:
mutantActivation: canHotSwap ? 'runtime' : 'static',
(equivalently
isStatic ? 'static' : 'runtime'— both yieldruntimeonly for non-static mutants with a test filter). -
Plugin (defensive): top-level activation in
stryker-setup.jsis safe for all mutants — runtime mutants live in test bodies, so assigningns.activeMutantearlier is harmless, while static mutants require it. The same pattern likely exists in the other test-runner plugins' setup files, since they share the same planner. -
Reload capability: implement it in
mutantRun(), or drop it fromcapabilities().
Activity
Mheaus commentedon Nov 12, 2026
About the secondary finding: the Vitest runner starts a new worker for each mutant run, so the modules are imported again every time, even if the runner does not reload anything itself. This is why the modules were re-evaluated in your experiments. I measured it with a small bench: 50 runs on one Vitest instance use 50 different workers, also with
isolate: falseand with the vm pools (https://github.com/Mheaus/vitest-worker-reuse-bench).For the main defect, the two open fixes are #6145 and #6293. I compared them on a real project in #6144.