Skip to content

ci: fork mutmut mutants from a fork server, not from the suite's process (ADR-1122) - #906

Merged
github-actions[bot] merged 1 commit into
mainfrom
ci/mutation-timeout-calibration
Nov 6, 2026
Merged

ci: fork mutmut mutants from a fork server, not from the suite's process (ADR-1122)#906
github-actions[bot] merged 1 commit into
mainfrom
ci/mutation-timeout-calibration

Conversation

@cdeust

@cdeust cdeust commented Nov 6, 2026

Copy link
Copy Markdown
Owner

Problem

On the Linux runner the mutation gate reports timeouts instead of verdicts:

ADR-1122 counts a timeout as a mutant not killed, so the gate fails after burning hours.

Root cause

mutmut 3.8.1 defaults to process_isolation = "fork": each mutant is forked from the main process, which has already run the suite (the stats and clean-test runs). The forked child inherits torch's state and, on Linux, stays idle in loss.backward() (mcp_server/core/titans_memory.py) until mutmut kills it.

Evidence

Same 11 mutants (telemetry.x_publish_record__mutmut_1*, #878 head 64ae942), PostgreSQL 18.6:

Environment Isolation ⏰
macOS arm64 fork 0 / 11
Ubuntu 26.04 aarch64, set up as ci.json "unit" (Python 3.12, extras dev/sqlite/codebase/postgresql) fork 5 / first 7
same container forkserver 0 / 11, 71 s

During the hang, py-spy dump of the child shows the main thread in TitansMemory.update → loss.backward(), with no CPU used. Between the platforms, one verdict differs: mutmut_17 is killed on Linux and survives on macOS.

Fix

process_isolation = "forkserver" in [tool.mutmut]. mutmut 3.8.1 documents this mode "for test setups that are not fork-safe (gevent, grpc, torch)": mutants are forked from a dedicated server that has never run a test. There is no timeout calibration, because the limits were not too short; the children were deadlocked.

  • pyproject.toml: the setting, with its source.
  • tests_py/scripts/test_mutation_isolation.py: locks the setting; it fails if fork comes back.
  • ADR-1122 (wiki page + mirror): item 4 and Consequences. This corrects the earlier claim "⏰ 0", which was measured on macOS only.
  • CHANGELOG under Changed.

Checks

@cdeust
ci: fork mutmut mutants from a fork server, not from the suite's proc…
381e2ed 
…ess (ADR-1122)

mutmut's default fork isolation forks every mutant from the process that already
ran the suite. On the Linux runner the child then hangs in torch autograd
(titans_memory.py loss.backward) until mutmut times it out, and a timeout fails
the gate: #877's 26 _resolve_domain mutants all timed out, and #878 was cancelled
at the six-hour job limit.

Reproduced in an Ubuntu 26.04 container set up as ci.json "unit" on 11 mutants
of #878: fork, 5 of the first 7 timed out (py-spy: child idle in backward());
forkserver, 0 of 11 timed out in 71 s. mutmut 3.8.1 documents forkserver for
test setups that are not fork-safe (gevent, grpc, torch).
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 6, 2026 •
edited
Loading

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-06T08:03:25.317195Z 381e2ed PR opened
🔒 Security Review ✅ Completed 2026-11-06T08:04:26.938905Z 381e2ed PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@claude

claude Bot commented Nov 6, 2026 •
edited
Loading

Copy link
Copy Markdown

Claude finished @cdeust's task in 4m 16s —— View job

Review of PR #906 (1 of 3)

  • Read the diff
  • Check that mutmut is importable wherever tests_py/scripts runs
  • Run five reviewer agents (quality, docs, performance, security, tests)
  • Post the verdict

No Blockers. The full review is in this comment.

  • One agent reported the module-level mutmut import as a CI Blocker, but that doesn't hold:
    • The unit job in ci.json installs uv's default dev group, which includes mutmut.
    • The only action that skips that group, .github/actions/test-suite, isn't called by any workflow.
  • That import does still break a plain local pytest for anyone using the install command in CLAUDE.md. It's recorded as a Note.
  • I couldn't run pytest here (permission denied), and web access was blocked, so the mutmut 3.8.1 quote and its fork default weren't checked against upstream.

Verdict: approve

@claude

claude Bot commented Nov 6, 2026

Copy link
Copy Markdown

Review 1 of 3: PR #906

I ran five reviews: code quality, documentation accuracy, performance, security and test coverage. No Blockers.

  • Code: process_isolation = "forkserver" has its # source: comment. The new test fails if the key is removed (KeyError), set back to fork (assertion fails) or misspelled (ValueError).
  • CI: the gating unit job (ci.json "unit") runs uv sync without --no-default-groups, so it installs the dev group and the mutmut import works there.
    • .github/actions/test-suite does skip that group, but no workflow calls it. The only mention is a comment at claude.yml:142.
    • memory-exact-review.yml and supersession-delete-parity.yml pass explicit test paths, so they never collect tests_py/scripts.
  • Docs: the ADR-1122 page and its wiki mirror match word for word. loss.backward() in TitansMemory.update is at mcp_server/core/titans_memory.py:163. PostgreSQL 18.6 (macOS) and 18.6.1 (Linux) agree with the CHANGELOG.
  • Security: no findings. No new dependency. The forkserver socket is in a private temp directory and reachable only by the runner's own user.

Notes, no action before merge

  • tests_py/scripts/test_mutation_isolation.py:13: mutmut is only in [dependency-groups] dev, so the install command in CLAUDE.md and CONTRIBUTING.md (--no-default-groups --extra dev) and the claude.yml agent environment don't have it. In those environments a plain pytest stops at collection. assert config["process_isolation"] == "forkserver" would remove the import.
  • Each forkserver child imports pytest, conftest and torch again for every mutant, while mutmut sets its timeout from test times measured in a process that already has them loaded. The evidence that this fits is one aarch64 sample (11 mutants in 71 s); the x86_64 hosted runner has not been measured.
  • Not checked against upstream (no web access here): the mutmut 3.8.1 quote "not fork-safe (gevent, grpc, torch)" and the claim that fork is the default. The next CI runs of fix(remember): keep an explicit domain instead of remapping it by substring (#721, slice 1) #877 and feat(telemetry): carry a per-stage latency breakdown in every telemetry sample (#722, slice 1) #878 will confirm the fix.

Verdict: approve

@github-actions
github-actions Bot merged commit d325ebb into main Nov 6, 2026
18 checks passed
This was referenced Nov 6, 2026
Merged
Merged
github-actions Bot pushed a commit that referenced this pull request Nov 6, 2026
@cdeust
ci: pick mutmut's process isolation per platform (ADR-1122) (#911)
6fd760e 
#906 set process_isolation = "forkserver" for every platform. On macOS that
aborts before any mutant runs: the stats run, now in a forked child, imports
sentence_transformers, whose torch.backends.mps.is_available() ends in SIGABRT
(Child exited with code -6). Local runs on macOS, the autopilot's included, could
no longer run the gate.

mutation_check.sh now rewrites process_isolation from ci_mutation_scope.py:
forkserver on Linux (fork hangs in torch autograd there), fork on macOS (measured
again: 11 mutants of #878, 76 s, 0 timeouts). Any other platform stops the gate
instead of guessing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@cdeust