You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 416eb17
Browse filesBrowse the repository at this point in the historyBrowse files
ci: pick mutmut's process isolation per platform (ADR-1122)
#906 set process_isolation = "forkserver" for every platform. On macOS that
aborts before any mutant runs: the stats run, now in a forked child, imports
sentence_transformers, whose torch.backends.mps.is_available() ends in SIGABRT
(Child exited with code -6). Local runs on macOS, the autopilot's included, could
no longer run the gate.
mutation_check.sh now rewrites process_isolation from ci_mutation_scope.py:
forkserver on Linux (fork hangs in torch autograd there), fork on macOS (measured
again: 11 mutants of #878, 76 s, 0 timeouts). Any other platform stops the gate
instead of guessing.
Copy file name to clipboardExpand all lines: docs/adr/ADR-1122-a-pull-request-must-kill-every-mutant-in-the-functions-it-changes.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -27,13 +27,13 @@ Prior art: Google mutates only the lines a change touches and reports the result
27
27
1.`scripts/ci_unit.sh` ends with `scripts/ci_mutation_diff.py`, so the `unit` job (and its coverage run) carries the mutation gate. `ci.json` keeps its five keys. Outside a pull request (`GITHUB_BASE_REF` empty) the step does nothing.
28
28
2. The script fetches the base branch, reads `git diff -U0 FETCH_HEAD...HEAD` (the merge base, so a base branch that moves on adds nothing) for `mcp_server/**/*.py`, and keeps the top-level functions and methods whose lines the PR changed, decorators and trailing blank lines included. A deletion of a whole top-level definition selects nothing; a deletion inside a function selects it. mutmut cannot mutate lines, so the granularity is the function, the finest unit it names. The deletion of a whole method selects nothing in its class, the deletion of a nested function selects its enclosing function. A changed function whose decorators mutmut 3.8.1 cannot mutate (anything but `staticmethod` and `classmethod`) produces no mutant at all, so it is reported as a `::warning::` and not gated.
29
29
3. Per changed file, the tests are the unit test files that import the module (`tests_py/scripts`, `integration`, `benchmarks` excluded). Imports are read with `ast`, so a parenthesized multiline import counts. A changed function whose module no test imports fails the job.
30
-
4. `mutation_check.sh` runs once per file with `MUTATION_ONLY_PREFIXES` and passes one `<prefix>*` name per changed function to `mutmut run`, so only the mutants of the changed functions run (`scripts/ci_mutation_scope.py`). mutmut 3.8.1 asserts when no mutant matches those names; after a failed run the gate passes only if mutmut's stats file exists (the stats run succeeded) and the `.meta` file of every mutated file exists and holds no mutant in scope; any other failure fails the job, and so does a failure of the helper that builds the names (it never falls back to running every mutant). `mutation_equivalents.py` drops survivors and registry entries outside those prefixes. What remains is judged as before: every survivor is a registered equivalent (`memory/mutation-equivalents.json`, pinned to its diff) or the job fails. A mutant in scope that no test killed (`no tests`, `timeout`, `suspicious`, `not checked`, `segfault`) fails the job too, and mutmut runs with `--max-children 1` because the tests share one database that every test empties. mutmut forks each mutant from a fork server that never ran a test (`process_isolation = "forkserver"` in `pyproject.toml`): forked from the process that ran the suite, a mutant hangs in torch on Linux and is reported as a timeout. The mutation sandbox copies `benchmarks` because unit tests import it. No wildcard in the equivalents registry, no per-file suppression.
30
+
4. `mutation_check.sh` runs once per file with `MUTATION_ONLY_PREFIXES` and passes one `<prefix>*` name per changed function to `mutmut run`, so only the mutants of the changed functions run (`scripts/ci_mutation_scope.py`). mutmut 3.8.1 asserts when no mutant matches those names; after a failed run the gate passes only if mutmut's stats file exists (the stats run succeeded) and the `.meta` file of every mutated file exists and holds no mutant in scope; any other failure fails the job, and so does a failure of the helper that builds the names (it never falls back to running every mutant). `mutation_equivalents.py` drops survivors and registry entries outside those prefixes. What remains is judged as before: every survivor is a registered equivalent (`memory/mutation-equivalents.json`, pinned to its diff) or the job fails. A mutant in scope that no test killed (`no tests`, `timeout`, `suspicious`, `not checked`, `segfault`) fails the job too, and mutmut runs with `--max-children 1` because the tests share one database that every test empties. `mutation_check.sh` sets mutmut's `process_isolation` per platform from `scripts/ci_mutation_scope.py`: `forkserver` on Linux, where a mutant forked from the process that ran the suite hangs in torch and is reported as a timeout; `fork` on macOS, where `forkserver` runs the stats in a forked child that aborts in torch MPS. Any other platform stops the gate, since no isolation was measured there. The mutation sandbox copies `benchmarks` because unit tests import it. No wildcard in the equivalents registry, no per-file suppression.
31
31
5. Survivors in functions the PR did not change are not this PR's failure; a PR that rewrites such a function owns its mutants.
32
32
33
33
## Consequences
34
34
35
35
- Metric: surviving unregistered mutants in changed functions at merge: 0 (not measured today).
36
36
- Measured end to end on a real diff: touching `_callee_basename` passes in 44 s (its 5 survivors are registered equivalents); touching `_walk_for_calls` fails with 4 unregistered survivors.
37
37
- The `unit` job grows with the mutants of the changed functions, not with the size of their files. Before the names were passed, mutmut ran every mutant of the file and the gate judged the changed functions afterwards: #865 (`backend_resolution.py`, 262 mutants) was cancelled at the six-hour job limit after 141 of them, and #877 ran past two hours. Measured on #877's diff (2026-11-06, local macOS arm64, PostgreSQL 18.6 from `launcher_pg_assets`): 36 mutants run instead of 1,164 (`remember.py` 26 of 741, `domain_mapping.py` about 10 of 423), and the gate reaches its verdict in 21 min 15 s, including the stats and clean-test runs. At the measured 0.07 mutants/s of `remember.py`, its 741 mutants would take about 2 h 56 min (extrapolated, not run). No mutant hit mutmut's timeout on macOS (`⏰ 0`), so no wall-clock cap is added; no cap is applied, so no change goes unchecked.
38
-
- On the Linux runner, mutmut's default `fork` isolation made the timeouts the verdict: #877's 26 `_resolve_domain` mutants were all `⏰`, and #878 was cancelled at the six-hour limit with 141 `⏰` among 272 `recall.py` mutants and 29 of 29 in `telemetry.py`. Reproduced on 2026-11-06 in an Ubuntu 26.04 aarch64 container set up as `ci.json` "unit" (Python 3.12, PostgreSQL 18.6.1), on the 11 `publish_record__mutmut_1*` mutants of #878: with `fork`, 5 of the first 7 timed out, the child idle in `loss.backward()` (`titans_memory.py`, `py-spy dump`); with `forkserver`, 0 of 11 timed out and the run took 71 s. The same 11 on macOS with `fork`: 0 timed out; one verdict differs between the platforms (`mutmut_17`, killed on Linux, survived on macOS).
38
+
- On the Linux runner, mutmut's default `fork` isolation made the timeouts the verdict: #877's 26 `_resolve_domain` mutants were all `⏰`, and #878 was cancelled at the six-hour limit with 141 `⏰` among 272 `recall.py` mutants and 29 of 29 in `telemetry.py`. Reproduced on 2026-11-06 in an Ubuntu 26.04 aarch64 container set up as `ci.json` "unit" (Python 3.12, PostgreSQL 18.6.1), on the 11 `publish_record__mutmut_1*` mutants of #878: with `fork`, 5 of the first 7 timed out, the child idle in `loss.backward()` (`titans_memory.py`, `py-spy dump`); with `forkserver`, 0 of 11 timed out and the run took 71 s. The same 11 on macOS with `fork`: 0 timed out; one verdict differs between the platforms (`mutmut_17`, killed on Linux, survived on macOS). On macOS, `forkserver` aborts before any mutant runs: the stats run, now in a forked child, imports `sentence_transformers`, whose `torch.backends.mps.is_available()` ends in SIGABRT (`Child exited with code -6`); with `fork` there, the gate run on the same 11 mutants takes 76 s with 0 timeouts.
39
39
- Module-level code (constants, imports) is not mutated by mutmut and is not gated.
assert n1 and n2 and n3 and n4, "pyproject [tool.mutmut] must define only_mutate, source_paths, pytest_add_cli_args_test_selection and process_isolation"
0 commit comments