Skip to content

Static mutants falsely reported as Survived with testFiles on Stryker 10.0.0 (corroborates #6144); vitest-runner reloadEnvironment capability declared but not implemented #6209

Open

Description

@easonLiangWorldedtech

@edelauna — this is a follow-up to #6144 with (1) an independently verified minimal reproduction on Stryker 10.0.0 (both with and without testFiles), (2) a controlled experiment that isolates the defect, and (3) a secondary finding in the vitest-runner plugin that #6144 does not cover. It is deliberately not a duplicate: the original report (9.6.1) is reproduced here on 10.0.0, and the new evidence narrows the root cause further.

Environment

  • @stryker-mutator/[email protected], @stryker-mutator/[email protected]
  • vitest 4.1.9
  • Node v25.9.0 on Windows. The defect is pure control-flow timing; expected to reproduce identically on Linux CI.

What happens

When the config sets testFiles, static mutants (top-level values evaluated at module import time) are reported as Survived even when the tests provably kill them. The mutant is never activated during the mutant run: the instrumented top-level code executes with activeMutant === undefined.

Minimal reproduction (verified on Stryker 10.0.0)

Standalone project, 4 files. Install:

pnpm add -D @stryker-mutator/[email protected] @stryker-mutator/[email protected] [email protected]

src/value.ts:

export const featureEnabled = false;

src/value.spec.ts:

import { featureEnabled } from './value';
import { expect, it } from 'vitest';

it('featureEnabled defaults to false', () => {
  expect(featureEnabled).toBe(false);
});

stryker.config.mjs:

export default {
  plugins: ['@stryker-mutator/vitest-runner'],
  testRunner: 'vitest',
  testFiles: ['src/value.spec.ts'], // <- presence of this option triggers the bug
  mutate: ['src/value.ts'],
  thresholds: { break: null },
};
npx stryker run

Actual output — the mutant is not killed (false Survived), even though the single test asserts the exact value:

WARN MutantTestPlanner Detected 1 static mutants (100% of total) that are estimated to take 100% of the time running the tests!
...
[Survived] BooleanLiteral
src/value.ts:1:31
File      |  total | covered | # killed | # timeout | # survived | # no cov | # errors |
All files |   0.00 |    0.00 |        0 |         0 |          1 |        0 |        0 |
 value.ts |   0.00 |    0.00 |        0 |         0 |          1 |        0 |        0 |

mutation.json: BooleanLiteral, status: "Survived", static: true, replacement: "true".

Control run — identical project with the testFiles line removed:

✓ featureEnabled defaults to false (killed 1)
File      |  total | covered | # killed | # timeout | # survived | # no cov | # errors |
All files | 100.00 |  100.00 |        1 |         0 |          0 |        0 |        0 |
 value.ts | 100.00 |  100.00 |        1 |         0 |          0 |        0 |        0 |

The only difference between the two runs is the testFiles option. The same false-Survived shape also reproduces with top-level object literals containing nested booleans (config-map / registry idiom): all such static mutants are reported Survived while the tests kill them.

Root cause chain (10.0.0 dist)

  1. @stryker-mutator/core, dist/src/mutants/mutant-test-planner.js, createMutantRunPlan():

    const canHotSwap = !!testFilter && isStatic === false;  // computed, but NOT used for activation
    ...
    mutantActivation: testFilter ? 'runtime' : 'static',    // line 148

    For a static mutant the planner takes the "run all tests" branch with testFilter = this.globalTestFilter — a non-empty array built from project.testFiles when testFiles is configured — and computes mutantActivation: 'runtime'. Presence of a test filter is conflated with the mutant being a runtime mutant. isStatic is already in scope (and is used correctly by the canHotSwap computed on the line just above).

  2. @stryker-mutator/vitest-runner, dist/src/stryker-setup.js: for mutantActivation !== 'static', activation happens only in beforeAll:

    beforeAll(() => { ns.activeMutant = inject('activeMutant'); });

    beforeAll runs after the spec module imports — i.e. after the instrumented top-level code has already been evaluated. The wrapper reads ns.activeMutant at import time, gets undefined, evaluates the original value, and the tests trivially pass → false Survived. Only the 'static' branch (top-level assignment in the setup file, which runs before the imports) can activate a static mutant.

    Net: static mutant + testFiles configured ⇒ 'runtime' activation ⇒ mutant never active ⇒ false Survived. Without testFiles, globalTestFilter is undefined, the planner happens to pick 'static', and the bug is hidden (as the control run above shows).

    Note: the instrumenter's mutant wrapper contains a fallback to process.env.__STRYKER_ACTIVE_MUTANT__, but a grep of the entire @stryker-mutator/[email protected] dist finds zero references to that variable — core never sets it, so the fallback is dead code as far as core is concerned.

Additional evidence (from a real-world repo, reproducible on request)

  1. The dry run detects the mutants as 100% static (empty perTest coverage, static coverage present); they live in top-level object literals.
  2. Manual mutation proof: applying each mutation by hand and running the spec fails 2 of 11 tests per mutant — the tests DO kill them.
  3. Sandbox probe: a probe spec run inside Stryker's own sandbox, activated via the __STRYKER_ACTIVE_MUTANT__ env-var fallback built into the instrumenter wrapper, shows the mutations applying correctly. Sandbox, instrumentation, and tests are all healthy; only the activation path is broken.
  4. Worker instrumentation: instrumenting the worker-side setup file shows inject('mutantActivation') returning "runtime" for the 100% static mutants, and ns.activeMutant being inserted into the namespace only in beforeAll — after the import-time mutantCoverage writes — so activeMutant is undefined exactly when the instrumented top-level code evaluates.
  5. Controlled experiment (decisive): identical run, but with the setup file forced to assign ns.activeMutant at top level (static activation) unconditionally — the only change. Result: all mutants Killed (status: Killed, static: true). Activation timing is the sole defect; there is no test-coverage gap and no instrumentation defect.

Secondary finding: reloadEnvironment capability declared but not implemented

  • Core sends reloadEnvironment: true in the run options for static mutants (mutant-test-planner.js: reloadEnvironment: !canHotSwap, preserved through ReloadEnvironmentDecorator when the capability is advertised).
  • VitestTestRunner.capabilities() returns { reloadEnvironment: true }.
  • But VitestTestRunner.mutantRun() never reads options.reloadEnvironment: the single long-lived vitest instance and its worker/module state are never reset between runs.

In the experiments above the module registry was re-evaluated per run, so this did not cause false results, but the contract the decorator relies on is broken: either implement the reload in mutantRun(), or drop the capability so the decorator falls back to recover().

Impact

Any project using the vitest runner with testFiles gets false Survived for every top-level constant / registry / config-map mutation — exactly the code shape produced by adding feature-flag, experiment, or schema entries. For us this breaks a mutation-diff CI gate on every PR that adds such an entry, despite the tests killing the mutants. The ignoreStatic workaround is not an option for a gate: it silently skips the entire static-mutant class (the 10.0.0 planner even warns about it).

Proposed fixes

  1. Planner (primary, as proposed in Static mutants are activated too late when --testFiles produces a test filter #6144) — use the already-computed capability:

    mutantActivation: canHotSwap ? 'runtime' : 'static',

    (equivalently isStatic ? 'static' : 'runtime' — both yield runtime only for non-static mutants with a test filter).

  2. Plugin (defensive): top-level activation in stryker-setup.js is safe for all mutants — runtime mutants live in test bodies, so assigning ns.activeMutant earlier is harmless, while static mutants require it. The same pattern likely exists in the other test-runner plugins' setup files, since they share the same planner.

  3. Reload capability: implement it in mutantRun(), or drop it from capabilities().

Activity

  1. Mheaus commented on Nov 12, 2026

    @Mheaus

    About the secondary finding: the Vitest runner starts a new worker for each mutant run, so the modules are imported again every time, even if the runner does not reload anything itself. This is why the modules were re-evaluated in your experiments. I measured it with a small bench: 50 runs on one Vitest instance use 50 different workers, also with isolate: false and with the vm pools (https://github.com/Mheaus/vitest-worker-reuse-bench).

    For the main defect, the two open fixes are #6145 and #6293. I compared them on a real project in #6144.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions