Skip to content

Commit 416eb17

Browse files
committed
ci: pick mutmut's process isolation per platform (ADR-1122)
#906 set process_isolation = "forkserver" for every platform. On macOS that aborts before any mutant runs: the stats run, now in a forked child, imports sentence_transformers, whose torch.backends.mps.is_available() ends in SIGABRT (Child exited with code -6). Local runs on macOS, the autopilot's included, could no longer run the gate. mutation_check.sh now rewrites process_isolation from ci_mutation_scope.py: forkserver on Linux (fork hangs in torch autograd there), fork on macOS (measured again: 11 mutants of #878, 76 s, 0 timeouts). Any other platform stops the gate instead of guessing.
1 parent d325ebb commit 416eb17

8 files changed

Lines changed: 68 additions & 19 deletions

‎CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,9 @@ adheres to [Semantic Versioning](https://semver.org/).
1717

1818
### Changed
1919

20+
- The mutation gate sets mutmut's process isolation per platform (ADR-1122):
21+
`forkserver` on Linux, `fork` on macOS, where `forkserver` aborted the stats run in
22+
torch MPS before any mutant ran. Another platform stops the gate.
2023
- mutmut forks each mutant from a fork server (`process_isolation = "forkserver"`,
2124
ADR-1122). Forked from the process that had run the suite, mutants hung in torch on
2225
the Linux runner and timed out: #878 was cancelled at the six-hour job limit.

‎docs/adr/ADR-1122-a-pull-request-must-kill-every-mutant-in-the-functions-it-changes.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,13 +27,13 @@ Prior art: Google mutates only the lines a change touches and reports the result
2727
1. `scripts/ci_unit.sh` ends with `scripts/ci_mutation_diff.py`, so the `unit` job (and its coverage run) carries the mutation gate. `ci.json` keeps its five keys. Outside a pull request (`GITHUB_BASE_REF` empty) the step does nothing.
2828
2. The script fetches the base branch, reads `git diff -U0 FETCH_HEAD...HEAD` (the merge base, so a base branch that moves on adds nothing) for `mcp_server/**/*.py`, and keeps the top-level functions and methods whose lines the PR changed, decorators and trailing blank lines included. A deletion of a whole top-level definition selects nothing; a deletion inside a function selects it. mutmut cannot mutate lines, so the granularity is the function, the finest unit it names. The deletion of a whole method selects nothing in its class, the deletion of a nested function selects its enclosing function. A changed function whose decorators mutmut 3.8.1 cannot mutate (anything but `staticmethod` and `classmethod`) produces no mutant at all, so it is reported as a `::warning::` and not gated.
2929
3. Per changed file, the tests are the unit test files that import the module (`tests_py/scripts`, `integration`, `benchmarks` excluded). Imports are read with `ast`, so a parenthesized multiline import counts. A changed function whose module no test imports fails the job.
30-
4. `mutation_check.sh` runs once per file with `MUTATION_ONLY_PREFIXES` and passes one `<prefix>*` name per changed function to `mutmut run`, so only the mutants of the changed functions run (`scripts/ci_mutation_scope.py`). mutmut 3.8.1 asserts when no mutant matches those names; after a failed run the gate passes only if mutmut's stats file exists (the stats run succeeded) and the `.meta` file of every mutated file exists and holds no mutant in scope; any other failure fails the job, and so does a failure of the helper that builds the names (it never falls back to running every mutant). `mutation_equivalents.py` drops survivors and registry entries outside those prefixes. What remains is judged as before: every survivor is a registered equivalent (`memory/mutation-equivalents.json`, pinned to its diff) or the job fails. A mutant in scope that no test killed (`no tests`, `timeout`, `suspicious`, `not checked`, `segfault`) fails the job too, and mutmut runs with `--max-children 1` because the tests share one database that every test empties. mutmut forks each mutant from a fork server that never ran a test (`process_isolation = "forkserver"` in `pyproject.toml`): forked from the process that ran the suite, a mutant hangs in torch on Linux and is reported as a timeout. The mutation sandbox copies `benchmarks` because unit tests import it. No wildcard in the equivalents registry, no per-file suppression.
30+
4. `mutation_check.sh` runs once per file with `MUTATION_ONLY_PREFIXES` and passes one `<prefix>*` name per changed function to `mutmut run`, so only the mutants of the changed functions run (`scripts/ci_mutation_scope.py`). mutmut 3.8.1 asserts when no mutant matches those names; after a failed run the gate passes only if mutmut's stats file exists (the stats run succeeded) and the `.meta` file of every mutated file exists and holds no mutant in scope; any other failure fails the job, and so does a failure of the helper that builds the names (it never falls back to running every mutant). `mutation_equivalents.py` drops survivors and registry entries outside those prefixes. What remains is judged as before: every survivor is a registered equivalent (`memory/mutation-equivalents.json`, pinned to its diff) or the job fails. A mutant in scope that no test killed (`no tests`, `timeout`, `suspicious`, `not checked`, `segfault`) fails the job too, and mutmut runs with `--max-children 1` because the tests share one database that every test empties. `mutation_check.sh` sets mutmut's `process_isolation` per platform from `scripts/ci_mutation_scope.py`: `forkserver` on Linux, where a mutant forked from the process that ran the suite hangs in torch and is reported as a timeout; `fork` on macOS, where `forkserver` runs the stats in a forked child that aborts in torch MPS. Any other platform stops the gate, since no isolation was measured there. The mutation sandbox copies `benchmarks` because unit tests import it. No wildcard in the equivalents registry, no per-file suppression.
3131
5. Survivors in functions the PR did not change are not this PR's failure; a PR that rewrites such a function owns its mutants.
3232

3333
## Consequences
3434

3535
- Metric: surviving unregistered mutants in changed functions at merge: 0 (not measured today).
3636
- Measured end to end on a real diff: touching `_callee_basename` passes in 44 s (its 5 survivors are registered equivalents); touching `_walk_for_calls` fails with 4 unregistered survivors.
3737
- The `unit` job grows with the mutants of the changed functions, not with the size of their files. Before the names were passed, mutmut ran every mutant of the file and the gate judged the changed functions afterwards: #865 (`backend_resolution.py`, 262 mutants) was cancelled at the six-hour job limit after 141 of them, and #877 ran past two hours. Measured on #877's diff (2026-11-06, local macOS arm64, PostgreSQL 18.6 from `launcher_pg_assets`): 36 mutants run instead of 1,164 (`remember.py` 26 of 741, `domain_mapping.py` about 10 of 423), and the gate reaches its verdict in 21 min 15 s, including the stats and clean-test runs. At the measured 0.07 mutants/s of `remember.py`, its 741 mutants would take about 2 h 56 min (extrapolated, not run). No mutant hit mutmut's timeout on macOS (`⏰ 0`), so no wall-clock cap is added; no cap is applied, so no change goes unchecked.
38-
- On the Linux runner, mutmut's default `fork` isolation made the timeouts the verdict: #877's 26 `_resolve_domain` mutants were all `⏰`, and #878 was cancelled at the six-hour limit with 141 `⏰` among 272 `recall.py` mutants and 29 of 29 in `telemetry.py`. Reproduced on 2026-11-06 in an Ubuntu 26.04 aarch64 container set up as `ci.json` "unit" (Python 3.12, PostgreSQL 18.6.1), on the 11 `publish_record__mutmut_1*` mutants of #878: with `fork`, 5 of the first 7 timed out, the child idle in `loss.backward()` (`titans_memory.py`, `py-spy dump`); with `forkserver`, 0 of 11 timed out and the run took 71 s. The same 11 on macOS with `fork`: 0 timed out; one verdict differs between the platforms (`mutmut_17`, killed on Linux, survived on macOS).
38+
- On the Linux runner, mutmut's default `fork` isolation made the timeouts the verdict: #877's 26 `_resolve_domain` mutants were all `⏰`, and #878 was cancelled at the six-hour limit with 141 `⏰` among 272 `recall.py` mutants and 29 of 29 in `telemetry.py`. Reproduced on 2026-11-06 in an Ubuntu 26.04 aarch64 container set up as `ci.json` "unit" (Python 3.12, PostgreSQL 18.6.1), on the 11 `publish_record__mutmut_1*` mutants of #878: with `fork`, 5 of the first 7 timed out, the child idle in `loss.backward()` (`titans_memory.py`, `py-spy dump`); with `forkserver`, 0 of 11 timed out and the run took 71 s. The same 11 on macOS with `fork`: 0 timed out; one verdict differs between the platforms (`mutmut_17`, killed on Linux, survived on macOS). On macOS, `forkserver` aborts before any mutant runs: the stats run, now in a forked child, imports `sentence_transformers`, whose `torch.backends.mps.is_available()` ends in SIGABRT (`Child exited with code -6`); with `fork` there, the gate run on the same 11 mutants takes 76 s with 0 timeouts.
3939
- Module-level code (constants, imports) is not mutated by mutmut and is not gated.

‎pyproject.toml‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -823,8 +823,7 @@ also_copy = [
823823
]
824824
pytest_add_cli_args_test_selection = ["tests_py/shared/test_json_native.py"]
825825
use_setproctitle = false
826-
# source: mutmut 3.8.1 configuration.py ProcessIsolation.FORKSERVER, "for test setups
827-
# that are not fork-safe (gevent, grpc, torch)"; ADR-1122. With the default "fork",
828-
# each mutant is forked from the process that already ran the suite, and on Linux it
829-
# hangs in torch autograd (titans_memory.py) until mutmut times it out (#877, #878).
826+
# source: ADR-1122. scripts/mutation_check.sh rewrites this line per platform from
827+
# scripts/ci_mutation_scope.py ISOLATION_BY_SYSTEM ("forkserver" on Linux, where "fork"
828+
# hangs in torch autograd; "fork" on macOS, where "forkserver" aborts in torch MPS).
830829
process_isolation = "forkserver"

‎scripts/ci_mutation_scope.py‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
#!/usr/bin/env python3
22
"""Run only the mutants of the changed functions; tell an empty scope apart (ADR-1122).
33
4+
It also names the mutmut process isolation that works on this platform.
5+
46
`mutmut run` with no name runs every mutant of the file, and the gate used to judge
57
only the changed functions afterwards: #865 ran 262 mutants of one file for six hours
68
to judge a few. Passing `<prefix>*` per changed function to `mutmut run` runs only
@@ -16,6 +18,7 @@
1618

1719
import json
1820
import os
21+
import platform
1922
import sys
2023
from pathlib import Path
2124

@@ -27,13 +30,27 @@
2730
MUTANTS_KEY = "exit_code_by_key"
2831
# source: scripts/ci_mutation_diff.py SCOPE_ENV, the variable both scripts read.
2932
SCOPE_ENV = "MUTATION_ONLY_PREFIXES"
33+
# source: ADR-1122 (measured 2026-11-06 on 11 mutants of #878, PostgreSQL 18.6).
34+
# Linux: with "fork" each mutant, forked from the process that ran the suite, hangs in
35+
# torch autograd until it times out (5 of the first 7); "forkserver" gives 0 of 11.
36+
# macOS: "forkserver" also runs the stats in a forked child, where importing
37+
# sentence_transformers calls torch.backends.mps.is_available() and aborts (SIGABRT);
38+
# "fork" gives 0 timeouts in 11.
39+
ISOLATION_BY_SYSTEM = {"Linux": "forkserver", "Darwin": "fork"}
3040

3141

3242
def name_globs(prefixes: list[str]) -> list[str]:
3343
"""The `mutmut run` arguments selecting every mutant of each changed function."""
3444
return [f"{prefix}*" for prefix in prefixes]
3545

3646

47+
def process_isolation(system: str) -> str:
48+
"""mutmut's `process_isolation` for this platform; none for an unmeasured one."""
49+
if system not in ISOLATION_BY_SYSTEM:
50+
raise SystemExit(f"error: no mutmut process isolation measured for {system!r}")
51+
return ISOLATION_BY_SYSTEM[system]
52+
53+
3754
def in_scope_count(
3855
mutants_dir: Path, prefixes: list[str], sources: list[str]
3956
) -> int | None:
@@ -57,12 +74,15 @@ def main(argv: list[str]) -> int:
5774
if argv[:1] == ["--globs"]:
5875
print("\n".join(name_globs(prefixes)))
5976
return 0
77+
if argv[:1] == ["--isolation"]:
78+
print(process_isolation(platform.system()))
79+
return 0
6080
# mutation_check.sh forwards its `-- <source>...` tail as given.
6181
sources = [arg for arg in argv[2:] if arg != "--"]
6282
if argv[:1] != ["--empty-scope"] or not sources or not prefixes:
6383
print(
6484
"usage: MUTATION_ONLY_PREFIXES=... ci_mutation_scope.py "
65-
"--globs | --empty-scope <mutants dir> <source>...",
85+
"--globs | --isolation | --empty-scope <mutants dir> <source>...",
6686
file=sys.stderr,
6787
)
6888
return 2

‎scripts/mutation_check.sh‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,9 +28,12 @@ cleanup() {
2828
}
2929
trap cleanup EXIT
3030

31+
# source: ADR-1122. An assignment keeps the helper's exit status, so `set -e` stops
32+
# on a platform with no measured isolation.
33+
ISOLATION="$(python3 "$ROOT/scripts/ci_mutation_scope.py" --isolation)"
3134
# source: ADR-0770
32-
python3 - "$PY" "${TEST_ARR[@]}" -- "$@" <<'PYEOF'
33-
import re, sys
35+
ISOLATION="$ISOLATION" python3 - "$PY" "${TEST_ARR[@]}" -- "$@" <<'PYEOF'
36+
import os, re, sys
3437
args = sys.argv[2:]
3538
sep = args.index("--")
3639
tests, sources = args[:sep], args[sep + 1:]
@@ -45,7 +48,9 @@ src, n2 = re.subn(r'^pytest_add_cli_args_test_selection = .*$',
4548
"pytest_add_cli_args_test_selection = " + fmt(tests), src, count=1, flags=re.M)
4649
src, n3 = re.subn(r'^source_paths = .*$',
4750
"source_paths = " + fmt(sorted(set(existing) | set(roots))), src, count=1, flags=re.M)
48-
assert n1 and n2 and n3, "pyproject [tool.mutmut] must define only_mutate, source_paths and pytest_add_cli_args_test_selection"
51+
src, n4 = re.subn(r'^process_isolation = .*$',
52+
f'process_isolation = "{os.environ["ISOLATION"]}"', src, count=1, flags=re.M)
53+
assert n1 and n2 and n3 and n4, "pyproject [tool.mutmut] must define only_mutate, source_paths, pytest_add_cli_args_test_selection and process_isolation"
4954
open(path, "w").write(src)
5055
PYEOF
5156

‎tests_py/scripts/test_ci_mutation_scope.py‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010

1111
import json
1212
import os
13+
import platform
1314
import shutil
1415
import subprocess
1516
import sys
@@ -37,6 +38,7 @@
3738
if [ "$1" = run ]; then
3839
shift
3940
printf '%s\\n' "$@" > "$FAKE_ARGS"
41+
grep '^process_isolation' pyproject.toml > "$FAKE_ARGS.isolation"
4042
mkdir -p mutants/mcp_server/core
4143
[ -n "$FAKE_STATS" ] && echo '{}' > mutants/mutmut-stats.json
4244
meta=mutants/mcp_server/core/a.py.meta
@@ -49,6 +51,7 @@
4951
source_paths = ["mcp_server"]
5052
only_mutate = ["x"]
5153
pytest_add_cli_args_test_selection = ["x"]
54+
process_isolation = "forkserver"
5255
"""
5356

5457

@@ -179,3 +182,22 @@ def test_a_failing_glob_helper_stops_the_gate_before_mutmut(bash, tmp_path):
179182
done, ran = _run(bash, root, PREFIX, {})
180183
assert done.returncode == 5
181184
assert ran is None
185+
186+
187+
@pytest.mark.parametrize("bash", BASHES, ids=lambda path: f"bash={path}")
188+
def test_mutmut_sees_the_isolation_of_this_platform(bash, tmp_path):
189+
done, _ = _gate(bash, tmp_path, PREFIX)
190+
assert done.returncode == 0, done.stderr
191+
seen = (tmp_path / "args.txt.isolation").read_text(encoding="utf-8")
192+
expected = scope.process_isolation(platform.system())
193+
assert seen == f'process_isolation = "{expected}"\n'
194+
195+
196+
def test_each_measured_platform_has_its_isolation():
197+
assert scope.process_isolation("Linux") == "forkserver"
198+
assert scope.process_isolation("Darwin") == "fork"
199+
200+
201+
def test_an_unmeasured_platform_stops_the_gate():
202+
with pytest.raises(SystemExit, match="no mutmut process isolation measured"):
203+
scope.process_isolation("Windows")
Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
1-
"""mutmut forks each mutant from a fork server, not from the suite (ADR-1122).
1+
"""mutmut's process isolation is chosen per platform by the gate (ADR-1122).
22
3-
With mutmut's default ``fork`` isolation every mutant is forked from the process that
4-
already ran the suite; on the Linux runner it then hangs in torch autograd until
5-
mutmut times it out, and a timeout fails the gate (#877, #878).
3+
mutation_check.sh rewrites `process_isolation` from ci_mutation_scope.py, so the line
4+
must exist in pyproject.toml and hold a value mutmut 3.8.1 accepts. With "fork" on
5+
Linux mutants hung in torch autograd until they timed out (#877, #878); with
6+
"forkserver" on macOS the stats run aborts in torch MPS.
67
"""
78

89
from __future__ import annotations
@@ -15,7 +16,6 @@
1516
PYPROJECT = Path(__file__).resolve().parents[2] / "pyproject.toml"
1617

1718

18-
def test_mutmut_forks_mutants_from_a_clean_fork_server():
19+
def test_pyproject_declares_the_line_the_gate_rewrites():
1920
config = tomllib.loads(PYPROJECT.read_text(encoding="utf-8"))["tool"]["mutmut"]
20-
isolation = ProcessIsolation(config["process_isolation"])
21-
assert isolation is ProcessIsolation.FORKSERVER
21+
assert ProcessIsolation(config["process_isolation"]) is ProcessIsolation.FORKSERVER

0 commit comments

Comments
 (0)