ci: run only the mutants of the changed functions in the mutation gate (ADR-1122) by cdeust · Pull Request #898 · cdeust/Cortex

cdeust · 2026-11-05T23:32:13Z

Summary

The mutation gate (ADR-1122) ran mutmut run with no name, so mutmut ran every mutant of each changed file and the gate judged only the changed functions afterwards (MUTATION_ONLY_PREFIXES). The unit job of a PR that touches a large module therefore outlived the six-hour job limit:

fix(storage): no default PostgreSQL URL in code; auto without a URL is SQLite (#704, slice 2) #865 (backend_resolution.py, 262 mutants): cancelled at 6 h after 141 mutants, run 42044899519.
fix(remember): keep an explicit domain instead of remapping it by substring (#721, slice 1) #877 and feat(telemetry): carry a per-stage latency breakdown in every telemetry sample (#722, slice 1) #878: unit still running more than 2 h after they started.

scripts/mutation_check.sh now passes one <prefix>* name per changed function to mutmut run (scripts/ci_mutation_scope.py --globs), so only the mutants of the changed functions run. Without MUTATION_ONLY_PREFIXES (a manual run), every mutant runs as before.

mutmut 3.8.1 asserts when no mutant matches the names, for example when a changed function holds nothing to mutate. After a failed run, the gate passes only when mutmut's own files show that case: mutants/mutmut-stats.json exists (written only after the stats run succeeds) and the .meta files hold no mutant in scope. Any other failure fails the job. The check reads files, not error text.

Checked in the mutmut 3.8.1 source:

tests_for_mutant_names matches a wildcard against function names, finds none for <prefix>*, and hands an empty set to the clean run.
run_tests([]) runs the whole selection, so the clean-test safeguard keeps its full scope.
The Mutant results block printed when names are given lists 🙁 <name>. Names start with mcp_server, so the 🙁 [0-9]+ tally still reads the progress line.

ADR-1122 is amended in the canonical page and its mirror. No wall-clock cap is added: no mutant hit mutmut's CPU timeout in the measurement, and the ADR keeps "no cap is applied, so no change goes unchecked".

Refs #865

Type of change

Bug fix (non-breaking change which fixes an issue)

Test plan

Benchmark on a real diff (fix(remember): keep an explicit domain instead of remapping it by substring (#721, slice 1) #877, fix/721 merged with main, local macOS arm64, PostgreSQL 18.6 from launcher_pg_assets, ci_unit.sh without the pytest line):
- With this change: 36 mutants run instead of 1,164 (remember.py: 26 of 741; domain_mapping.py: about 10 of 423). The gate reaches its verdict in 21 min 15 s, stats and clean-test runs included.
- Without this change: estimated at about 2 h 56 min for the 741 mutants of remember.py, from the measured 0.07 mutants/s. This is an extrapolation, not a run.
- Verdict on fix(remember): keep an explicit domain instead of remapping it by substring (#721, slice 1) #877: 25 unregistered survivors in its changed functions (24 in _resolve_domain, 1 in resolve_explicit_domain). This is a real test gap that fix(remember): keep an explicit domain instead of remapping it by substring (#721, slice 1) #877 owns, not a defect of this PR.
New tests_py/scripts/test_ci_mutation_scope.py (13 tests) drives the real mutation_check.sh against a fake uv, under /bin/bash 3.2 and Homebrew bash. It covers:
- the scoped names;
- the run without a scope;
- the empty scope, which passes;
- a failure before the stats file, which still fails;
- a failure with mutants in scope, which still fails.
  With main's mutation_check.sh, the scoped-names and empty-scope tests fail (4 failures, both bashes).
pytest tests_py/scripts: 1,832 passed. The 41 errors all come from test_launcher_platform_coverage.py, where uv venv hits a broken x86_64 managed interpreter on this host ("Bad CPU type"). They are unrelated to this diff.
python scripts/check_craftsmanship.py: OK. ruff check and ruff format --check pass on the new files.
Mutation survival check: the tests catch dropping the names, passing an empty scope as a pass when stats are missing, and swallowing a failure with mutants in scope.

Audit notes

No independent review in this pass. The advisor review before coding pointed at the clean-test selection, the pipefail capture and the tally parsing, all checked above.
Out of scope: the verdict-comment binding defect (docs(audit): inventory the transcript capture producers (#716, slice 1) #874, fix(hooks): read the post-commit cooldown as UTF-8 and copy the plugin trees into the mutation sandbox (#692, class 3) #888) and the grouping of the autopilot's fixup! commits.

Coding-standards compliance

§2.2 Layer direction: scripts only, no mcp_server import in production code.
§4.1 / §4.2 / §4.4 File, function and parameter limits hold.
§8 Every new constant has a # source: line.
§9 No dead code, no TODO.

…e (ADR-1122) mutation_check.sh ran `mutmut run` with no name, so mutmut ran every mutant of each changed file and the gate judged the changed functions afterwards. #865 (backend_resolution.py, 262 mutants) was cancelled at the six-hour job limit after 141; #877 ran past two hours. The script now passes one `<prefix>*` name per changed function (scripts/ci_mutation_scope.py --globs). mutmut 3.8.1 asserts when no mutant matches; after a failed run the gate passes only when mutmut's stats file exists and its .meta files hold no mutant in scope, any other failure fails the job. Without MUTATION_ONLY_PREFIXES, every mutant runs as before. Measured on #877's diff (local macOS arm64, PostgreSQL 18.6): 36 mutants instead of 1,164, verdict in 21 min 15 s. No mutant hit mutmut's CPU timeout, so no wall-clock cap is added.

chatgpt-codex-connector · 2026-11-05T23:32:25Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-06T00:03:19.829019Z	`50349fa`	New commits
🔒 Security Review	✅ Completed 2026-11-06T00:25:33.582521Z	`50349fa`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

claude · 2026-11-05T23:32:51Z

Claude finished @cdeust's task in 3m 23s —— View job

Review of PR #898: done

Read the diff and how ci_mutation_diff.py builds the name prefixes
Ran the five reviews: code quality, documentation, performance, security, test coverage
Posted the review: ci: run only the mutants of the changed functions in the mutation gate (ADR-1122) #898 (comment)

Result: no Blockers and three Notes. Verdict: approve.

I could not run pytest because the command was not approved in this environment, so the findings come from reading the code, not from a test run.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 31790c8a30

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-11-05T23:35:06Z

+  if [ -n "${MUTATION_ONLY_PREFIXES:-}" ] \
+    && python3 "$ROOT/scripts/ci_mutation_scope.py" --empty-scope "$ROOT/mutants"; then
+    exit 0


Require fresh mutation artifacts before accepting a failed run

If mutants/ already contains artifacts from a previous manual mutmut run, a scoped invocation can incorrectly pass without running mutation tests. The script only deletes that directory on exit, so an old stats file and metadata with no mutants matching the current prefixes satisfy --empty-scope even when the current command fails before mutmut starts. I reproduced this with pre-existing stats and a fake uv exiting 42: the gate returned 0. Clear the previous artifacts before starting, or otherwise verify that the stats and metadata belong to the current invocation before converting its failure into success.

Useful? React with 👍 / 👎.

claude · 2026-11-05T23:36:13Z

Review 1 of 3: PR #898

I ran five reviews: code quality, documentation, performance, security and test coverage. No Blockers. I could not run the test suite in this environment, so these findings come from reading the code, not from a run.

What I traced:

Scoping works. ci_mutation_diff.py skips a file that has no prefixes, so a PR job always sets MUTATION_ONLY_PREFIXES. Each prefix ends in __mutmut_, so the glob <prefix>* cannot reach a neighbouring function.
The empty-scope pass cannot hide a real failure. It passes only when no mutant is in scope:
- an empty or blank scope exits 2;
- a malformed .meta raises an error, so the job keeps mutmut's exit code;
- a missing stats file returns 1.
The shell handling is correct. PIPESTATUS is read inside set +e, and ${NAMES[@]+…} is safe on bash 3.2.
The survivor counts still agree. TALLY and LISTED count the same survivors, because mutants that did not run show as "not checked". --untested filters those by scope.
Limits hold. The longest file is 153 lines, every function is under 40 lines, and every constant has a # source: line.

Notes, no action before merge

scripts/mutation_check.sh:57-58: if --globs fails inside < <(…), set -e does not stop the script. NAMES stays empty and the job quietly runs every mutant (the verdict stays correct). GLOBS="$(…)" would fail loudly instead.
scripts/ci_mutation_scope.py:62-66: --empty-scope passes any mutmut failure after the stats run when no mutant is in scope, including when the target file's .meta is missing (for example if mutmut's naming changes). The comment at mutation_check.sh:66 says "only that case". Requiring the target file's .meta would close this.
tests_py/scripts/test_ci_mutation_scope.py: the fake failures use rc=1, so a test cannot tell exit "$RUN_RC" apart from a hard-coded exit 1. Also, the ADR's measurement date (2026-11-06) is later than the ADR's own created date.

Verdict: approve

… meta for an empty scope Review 1 of #898 notes: - A failure of `ci_mutation_scope.py --globs` inside `< <(...)` did not stop the script: NAMES stayed empty and every mutant ran. The names are now read from an assignment, so `set -e` stops the gate. - `--empty-scope` passed when the mutated file's `.meta` was missing. It now takes the mutated files and concludes only when each `.meta` exists; an empty file list is a usage error. - The shell tests used rc=1 for the fake failures; they now use 3, so the script must forward mutmut's own exit code.

claude · 2026-11-06T00:03:22Z

Claude finished @cdeust's task in 2m 12s —— View job

Review 2 of 3: PR #898, done

Check the Blockers from review 1 (there were none) and find the commits pushed after 31790c8 (only 50349fa)
Read 50349fa
Run the five reviews on 50349fa: code quality, documentation, performance, security, test coverage
Post the review: ci: run only the mutants of the changed functions in the mutation gate (ADR-1122) #898 (comment)

Result: no Blockers. The three notes from review 1 are fixed and each fix has a test. There are three new Notes, the first about code already on main. Codex's comment on stale mutants/ files cannot happen in CI. I reviewed by reading the code; I did not run the tests.

Verdict: approve

claude · 2026-11-06T00:05:36Z

Review 2 of 3: PR #898

Blockers from review 1: there were none. This review covers only 50349fa, the one commit pushed after 31790c8. I ran five reviews on it: code quality, documentation, performance, security and test coverage. No Blockers. I traced the code by reading it; I did not run the tests.

The three notes from review 1 are fixed:

The glob helper now fails loudly. At mutation_check.sh:58, GLOBS="$(…)" is a top-level assignment, so set -e stops the script with the helper's exit code on bash 3.2 and 5. test_a_failing_glob_helper_stops_the_gate_before_mutmut checks both the exit code (5) and that mutmut never ran.
--empty-scope now needs each target's .meta. After the shift, "$@" is -- <file>. argv[2:] with -- filtered out gives [file], and mutants/<file>.meta matches mutmut's meta_path. A missing stats file or a missing target meta returns None, so the job keeps mutmut's exit code. An empty source list is a usage error. Reverting any of these makes a test fail.
The tests now catch a hard-coded exit code. Fake failures exit with 3, so the tests check that the script forwards mutmut's own code.

Codex's comment on stale mutants/ files cannot happen in CI:

Each job starts from a fresh checkout, and mutants/ is gitignored.
The EXIT trap deletes mutants/ after each file.
The check now needs the current target's own .meta, and an earlier file's run never writes that.

Notes, no action before merge

scripts/mutation_check.sh:32 (already on main, not in this commit): "${TEST_ARR[@]}" -- "$@" receives -- <file>, so the inline Python gets -- -- <file>. sources then becomes ["--", "<file>"], and "--" ends up in only_mutate and source_paths. Filtering -- there, the way ci_mutation_scope.py:61 does, would fix it.
scripts/mutation_check.sh:67: the comment says "only that case is not a failure", but the code passes any failed run whose files show an empty scope. The ADR describes this correctly; only the comment claims more.
scripts/mutation_check.sh:59: when MUTATION_ONLY_PREFIXES holds only newlines (a manual run only), NAMES=("") is passed to mutmut. It still fails, because --empty-scope returns 2, but skipping empty lines would be cleaner.

Verdict: approve

cdeust added 2 commits November 6, 2026 00:31

test(ci): keep the mutation scope test helper within four parameters

31790c8

chatgpt-codex-connector Bot reviewed Nov 5, 2026

View reviewed changes

github-actions Bot merged commit f9f8074 into main Nov 6, 2026
18 checks passed

claude Bot mentioned this pull request Nov 6, 2026

feat(telemetry): carry a per-stage latency breakdown in every telemetry sample (#722, slice 1) #878

Merged

3 tasks done

cdeust mentioned this pull request Nov 6, 2026

fix(storage): no default PostgreSQL URL in code; auto without a URL is SQLite (#704, slice 2) #865

Closed

5 of 6 tasks

Navigation Menu

Repository navigation

ci: run only the mutants of the changed functions in the mutation gate (ADR-1122) - #898

ci: run only the mutants of the changed functions in the mutation gate (ADR-1122)#898
github-actions[bot] merged 3 commits into
mainfrom
ci/mutation-gate-budget

cdeust commented Nov 5, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 5, 2026 •

edited

Loading

Uh oh!

claude Bot commented Nov 5, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Nov 5, 2026

Uh oh!

claude Bot commented Nov 5, 2026

Uh oh!

claude Bot commented Nov 6, 2026 •

edited

Loading

Uh oh!

claude Bot commented Nov 6, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

cdeust commented Nov 5, 2026

Summary

Type of change

Test plan

Audit notes

Coding-standards compliance

Uh oh!

chatgpt-codex-connector Bot commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

claude Bot commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review of PR #898: done

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot commented Nov 5, 2026

Review 1 of 3: PR #898

Uh oh!

claude Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review 2 of 3: PR #898, done

Uh oh!

claude Bot commented Nov 6, 2026

Review 2 of 3: PR #898

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

chatgpt-codex-connector Bot commented Nov 5, 2026 •

edited

Loading

claude Bot commented Nov 5, 2026 •

edited

Loading

claude Bot commented Nov 6, 2026 •

edited

Loading