ci: pick mutmut's process isolation per platform (ADR-1122) by cdeust · Pull Request #911 · cdeust/Cortex

cdeust · 2026-11-06T12:34:35Z

Problem: a regression from #906 on macOS

#906 set process_isolation = "forkserver" for every platform. On macOS this mode aborts before any mutant runs. The stats run now executes in a forked child; there, importing sentence_transformers calls torch.backends.mps.is_available(), which ends in SIGABRT:

Current thread … torch/backends/mps/__init__.py, line 31 in is_available
  … transformers/integrations/sdpa_attention.py … sentence_transformers/base/model.py
ChildProcessError: Child exited with code -6

The result is that no local run on macOS can execute the gate any more, including the autopilot's checks (GITHUB_BASE_REF=main python scripts/ci_mutation_diff.py). The Linux CI is not affected.

Fix

mutation_check.sh already rewrites [tool.mutmut] before each run. It now also rewrites process_isolation, taking the value from scripts/ci_mutation_scope.py ISOLATION_BY_SYSTEM:

Platform	Isolation	Measured on the same 11 mutants of #878, PostgreSQL 18.6
Linux (Ubuntu 26.04 container, set up like CI)	`forkserver`	`fork`: 5 timeouts in the first 7 (child idle in torch `backward()`); `forkserver`: 0 of 11
macOS arm64	`fork`	`forkserver`: SIGABRT before any mutant runs; `fork`: 0 timeouts in 11, gate run 76 s (9 killed, 2 survived)

Any other platform stops the gate with an error. There is no silent fallback to a mode nobody has measured.

Tests

test_mutmut_sees_the_isolation_of_this_platform (on every bash on the host): the pyproject.toml that mutmut reads carries this platform's value.
test_each_measured_platform_has_its_isolation and test_an_unmeasured_platform_stops_the_gate.
test_mutation_isolation.py: the line the gate rewrites exists in pyproject.toml and is valid for mutmut.
22 tests pass. ruff check and ruff format --check pass on the touched files, and so does scripts/check_craftsmanship.py.

ADR-1122 (wiki page and mirror) and the CHANGELOG are updated.

#906 set process_isolation = "forkserver" for every platform. On macOS that aborts before any mutant runs: the stats run, now in a forked child, imports sentence_transformers, whose torch.backends.mps.is_available() ends in SIGABRT (Child exited with code -6). Local runs on macOS, the autopilot's included, could no longer run the gate. mutation_check.sh now rewrites process_isolation from ci_mutation_scope.py: forkserver on Linux (fork hangs in torch autograd there), fork on macOS (measured again: 11 mutants of #878, 76 s, 0 timeouts). Any other platform stops the gate instead of guessing.

chatgpt-codex-connector · 2026-11-06T12:34:49Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-06T12:50:20.551291Z	`1b4c3a0`	New commits
🔒 Security Review	✅ Completed 2026-11-06T12:51:12.766534Z	`1b4c3a0`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

claude · 2026-11-06T12:35:21Z

Claude finished @cdeust's task in 2m 33s —— View job

PR #911 review (1 of 3)

Read the diff and the gate script
Ran five reviews: code quality, security, test coverage, documentation, performance
Checked each finding against the code
Posted the review: ci: pick mutmut's process isolation per platform (ADR-1122) #911 (comment)

No Blockers. The review leaves three notes for after merge, all about tests. One concerns the mutmut import in test_mutation_isolation.py: it was flagged as a blocker, but the line is unchanged from #906, so it was not counted as one for this diff.

I could not run pytest, ruff or the craftsmanship gate in this job (permission denied), so the review comes from reading the code.

Verdict: approve

claude · 2026-11-06T12:37:56Z

Review 1 of 3: PR #911 (ADR-1119 contract)

I ran five reviews: code quality, security, test coverage, documentation and performance. No Blockers.

What I traced:

Linux CI is unchanged. ISOLATION_BY_SYSTEM["Linux"] is "forkserver", so mutation_check.sh writes back the same pyproject.toml line ci: fork mutmut mutants from a fork server, not from the suite's process (ADR-1122) #906 set.
Other platforms stop the gate. ISOLATION="$(… --isolation)" is a plain assignment, so set -e stops on the helper's SystemExit. The EXIT trap is installed first, so pyproject.toml is restored. There is no fallback.
No silent miss. The n4 assert fails loudly if the process_isolation line is missing.
No injection. The value can only come from the fixed dict, and every mutmut run goes through this script.
Docs agree with the code. The ADR and its wiki mirror are identical, the CHANGELOG matches, and # source: is present on the new constant.

I could not run pytest, ruff or the craftsmanship gate in this job (permission denied), so these findings come from reading the code. The Pipeline run was still in progress when I wrote this.

Notes, no action before merge

test_a_failing_glob_helper_stops_the_gate_before_mutmut (tests_py/scripts/test_ci_mutation_scope.py:177): the stub now fails at the new --isolation call, not at --globs. A failing --globs alone is no longer tested.
test_mutmut_sees_the_isolation_of_this_platform: the sandbox PYPROJECT already says "forkserver", so on Linux the test passes even if the rewrite does nothing. Seeding a placeholder value would prove the rewrite.
tests_py/scripts/test_mutation_isolation.py:14 imports mutmut. This is unchanged from ci: fork mutmut mutants from a fork server, not from the suite's process (ADR-1122) #906, but mutmut is only in the dev dependency group, not the dev extra that the test suite installs. Worth confirming that collection works there.

View job run

Verdict: approve

…-per-platform # Conflicts: # CHANGELOG.md

claude · 2026-11-06T12:49:26Z

Claude finished @cdeust's task in 1m 42s —— View job

PR #911 review (2 of 3)

Check the Blockers from review 1: there were none
Find the commits pushed after 416eb17: one, 1b4c3a0, which merges origin/main (ci: sync the dev group in the workflows that run ci_unit.sh (ADR-1122) #910)
Run five reviews on that delta: code quality, security, test coverage, documentation, performance
Post the review: ci: pick mutmut's process isolation per platform (ADR-1122) #911 (comment)

No Blockers. The only new commit is a merge from main. It keeps both CHANGELOG entries, and #910's --group dev sync works with this PR's per-platform isolation: Linux gets forkserver. It also clears review 1's question about the mutmut import being collected. The review leaves three notes for after merge:

The ci: fork mutmut mutants from a fork server, not from the suite's process (ADR-1122) #906 CHANGELOG entry now needs "on Linux".
The unused test-suite action lacks --group dev.
memory-exact-review can run the gate up to three times per run.

I could not run pytest, ruff or the craftsmanship gate in this job (permission denied), so the review comes from reading the code.

Verdict: approve

claude · 2026-11-06T12:51:09Z

Review 2 of 3: PR #911 (ADR-1119 contract)

No Blockers. Review 1 had none either.

What changed since 416eb17. One commit, 1b4c3a0. It merges origin/main, which brings in #910 (--group dev added to memory-exact-review and supersession-delete-parity, plus test_mutation_workflows_install_mutmut.py). No file this PR changes was edited in that commit. I reviewed only this delta, using five reviews: code quality, security, test coverage, documentation and performance.

What I checked:

CHANGELOG merge. Both Unreleased entries are kept: this PR's per-platform isolation and ci: sync the dev group in the workflows that run ci_unit.sh (ADR-1122) #910's dev-group sync. No conflict markers remain.
How the two changes fit together. Both workflows run on ubuntu-latest. ci_unit.sh calls ci_mutation_diff.py, which calls mutation_check.sh. On Linux, ISOLATION_BY_SYSTEM gives forkserver, so the new --isolation step cannot stop these jobs. With ci: sync the dev group in the workflows that run ci_unit.sh (ADR-1122) #910 they now install mutmut and actually run the gate, where before they failed with "Failed to spawn: mutmut".
Review 1's third note (the mutmut import in test_mutation_isolation.py). The hosted unit job installs the dev group by default (plain uv sync). The two workflows from ci: sync the dev group in the workflows that run ci_unit.sh (ADR-1122) #910 now pass --group dev. The test collects in both.
Security. --group dev is a fixed string. It reaches uv sync --locked through UV_SET, still resolves from uv.lock, and permissions: is unchanged.

As in review 1, I could not run pytest or ruff in this job, so these findings come from reading the code.

Notes, no action before merge

CHANGELOG.md: the earlier ci: fork mutmut mutants from a fork server, not from the suite's process (ADR-1122) #906 entry ("mutmut forks each mutant from a fork server") is now true only on Linux. Adding "on Linux" would match the new entry above it.
.github/actions/test-suite/action.yml:188 syncs --extra dev without --group dev, so a full pytest run through it would fail to import mutmut in test_mutation_isolation.py. No workflow uses this action today, and the ci: sync the dev group in the workflows that run ci_unit.sh (ADR-1122) #910 guard does not check it.
memory-exact-review.yml calls ci_unit.sh three times, so with mutmut installed it can run the diff-scoped gate up to three times per run.

Verdict: approve

Merge remote-tracking branch 'origin/main' into ci/mutation-isolation…

1b4c3a0

…-per-platform # Conflicts: # CHANGELOG.md

github-actions Bot merged commit 6fd760e into main Nov 6, 2026
18 checks passed

claude Bot mentioned this pull request Nov 6, 2026

ci: an added blank or comment line selects no function for the mutation gate (ADR-1122) #912

Merged

This was referenced Nov 6, 2026

feat(telemetry): carry a per-stage latency breakdown in every telemetry sample (#722, slice 1) #878

Merged

fix(storage): no default PostgreSQL URL in code; auto without a URL is SQLite (#704, slice 2) #865

Closed

Navigation Menu

Repository navigation

ci: pick mutmut's process isolation per platform (ADR-1122) - #911

ci: pick mutmut's process isolation per platform (ADR-1122)#911
github-actions[bot] merged 2 commits into
mainfrom
ci/mutation-isolation-per-platform

cdeust commented Nov 6, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026 •

edited

Loading

Uh oh!

claude Bot commented Nov 6, 2026 •

edited

Loading

Uh oh!

claude Bot commented Nov 6, 2026

Uh oh!

claude Bot commented Nov 6, 2026 •

edited

Loading

Uh oh!

claude Bot commented Nov 6, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

cdeust commented Nov 6, 2026

Problem: a regression from #906 on macOS

Fix

Tests

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

claude Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR #911 review (1 of 3)

Uh oh!

claude Bot commented Nov 6, 2026

Review 1 of 3: PR #911 (ADR-1119 contract)

Uh oh!

claude Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR #911 review (2 of 3)

Uh oh!

claude Bot commented Nov 6, 2026

Review 2 of 3: PR #911 (ADR-1119 contract)

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

chatgpt-codex-connector Bot commented Nov 6, 2026 •

edited

Loading

claude Bot commented Nov 6, 2026 •

edited

Loading

claude Bot commented Nov 6, 2026 •

edited

Loading