Skip to content

fix(test-impact): started roles, fork PRs, base tooling, all test legs - #2959

Merged
DeusData merged 4 commits into
mainfrom
fix/test-impact-gate
Nov 12, 2026
Merged

DeusData merged 4 commits into
mainfrom
fix/test-impact-gate

Conversation

@DeusData

Copy link
Copy Markdown
Owner

What this changes

Three fixes to the test-impact gate (smart CI, layer 2), in three commits:

  1. Engine: a role started by another role inherits its spawners.
    __cbm_rss_tree_hold is started only by the __cbm_rss_tree_root role
    handler, never by a test, so every change that reached its handler vetoed
    the whole selection (RUNNER_REACHED). That veto decided 14 of the last 39
    engine answers. A handler that names another role's literal now starts
    that role: the started role is lifted with its starter's spawners. A role
    that is both named by a test and started by another role now selects both
    sets of spawners (before: only the direct ones, an under-selection). The
    role note now carries the call chain, like the main-reach note.
  2. selection.py: platform-suite classification and a portable view.
    platform-suites lists the suites whose test file holds SKIP_PLATFORM(
    or an #if/#ifdef on an OS or architecture macro (the two markers
    scripts/check-no-test-skips.sh names), at the base and at HEAD, plus
    suites the runner registers under such a conditional: 80 suites today.
    filter --whole-suites writes the view where those suites run whole.
  3. CI wiring.
    • select-tests now runs for fork PRs: (vars.TEST_IMPACT_MODE || 'gate') == 'gate'.
      Fork PRs get no repository variables, so all 24 fork runs in the sample
      skipped the job. No pull_request_target, no new permissions, no secrets.
    • select-tests builds and runs the merge base's engine,
      test-impact-select.sh and selection.py against the PR checkout
      (--repo), so a PR can no longer narrow its own tests by changing the
      selection tooling. This matters more now that fork PRs narrow.
    • The arm64 and macOS test-unix legs and the Windows leg now run the
      selection's portable view (platform-neutral suites as selected, every
      platform-specific suite whole). Linux x86_64 runs the selection as
      before. This reverses the 2026-10-16 x86-only decision (maintainer
      decision, 2026-11-11).

CI scope (please review before merge)

  • Workflows and jobs touched: pr.yml (select-tests: condition,
    steps, new portable output; test: new test_selection_portable input)
    and _test.yml (new boolean input test_selection_portable; test-unix
    fetches the selection on every leg when the portable view exists;
    test-windows gains a fetch step). Dry run and release do not pass a
    selection and are unchanged. TSan, MSan, diag, LSan, smoke, package
    wrappers and UI legs are unchanged and still run every test.
  • Gating: select-tests stays outside ci-ok (a failure means every
    test runs). The test legs stay gating; what changes is how much of the
    suite list they run on arm64, macOS and Windows.
  • Cost: select-tests (ubuntu, ~10-15 job-min: engine build plus
    selection) now also runs on fork PRs. Savings from the portable view are
    small today: in the 10 sampled selections the platform-specific suites
    are 76-94% of each leg's suite time and the selections keep 47-176 of
    ~190 suites, so the modelled saving is a median of 0.2-0.6 minutes per
    non-x86 leg (best case 1.4-7.8). See "Known limits".
  • Flake surface: narrowed suite waves change which suites share a
    runner on arm64, macOS and Windows; the selection download is a new step
    on those legs (an artifact download failure fails the leg, as it already
    can on x86). The harness selection path itself is exercised on every leg
    by the existing contract step 2c.
  • Transition: until this lands, the merge base has no --repo, so this
    PR's own select-tests fails and every test runs. That is the designed
    fail-open path.
  • Off switches: TEST_IMPACT_MODE=shadow (same-repo and Dependabot
    PRs); for fork PRs, change the 'gate' default in pr.yml on main; to
    keep narrowing on x86 only, pass test_selection_portable: false.

Known limits (not changed here)

  • With the role veto gone, 4 of 6 replayed merged PRs now hit a second
    RUNNER_REACHED that the role veto had masked: the runner's main reaches
    the memory core through its startup path, and the memory core is
    "reached" because the parse-gap rule cascades. A gap's text is matched
    against the short name of every reached node, so gaps fire on generic
    names (idx, n, index, used; 312 distinct firings in one replay),
    and a file-scope gap seeds its whole file. In the sampled replays it was
    src/foundation/compat_thread.c:109 (__attribute__((used)), fired on
    used), which seeds cbm_mutex_init, called by the memory core. An
    experiment that ignores attribute words only moves the way in
    (main -> tf_setup_cache_sentinel -> ... -> platform.cbm_app_local_dir,
    504-1224 firings). Follow-up (decided): gaps will match only reached
    function, variable and macro names, with no whole-file seeding for a
    file-scope gap, validated by a replay miss audit before it gates.
  • Ancestor-bundle pollution (main's own commits in A..HEAD) was not changed:
    all 5 polluted verdicts in the sample came from main commits that changed
    runner inputs (a vendored grammar; the harness and Makefile), which the
    ancestor's coverage map does not know. Keeping run_all there is required.
    Follow-up (decided): cut the coverage producer's latency so the exact
    merge-base bundle exists (it was missing in 51% of the sampled runs).

Proof

  • Engine: new test_impact_engine_role_started_by_a_role_selects_its_spawners
    is RED on main (run_all, "the change reaches the test-runner role
    __fx_hold_role, which no test names") and GREEN with the fix; suites
    test_impact_engine (22), test_impact, test_impact_artifact,
    test_impact_runner_filter and runner_isolation (99) pass under ASan+UBSan.
  • Real input: replaying merged PR 0d835c5 (M^1..M): main's engine vetoes
    on __cbm_rss_tree_hold; the fixed engine no longer does (it then stops
    at the main-reach veto described under "Known limits").
  • Fork PRs: all 7 engine answers in the 40-run sample that would have
    narrowed came from fork runs whose select-tests was skipped; with this
    change they reach the test legs.
  • selection.py: the portable-view cases pass with this change and fail on
    main (--whole-suites unknown).
  • clang-format (Homebrew LLVM) clean on the touched C hunks; memory-core
    linter passes; shellcheck clean on the touched scripts.

The test runner's __cbm_rss_tree_root role starts a __cbm_rss_tree_hold
child, and no test names __cbm_rss_tree_hold itself: the test that spawns
the root does. The role lift excludes every handler from its literal scan
(a handler naming its own literal is no spawner), so a change reaching the
hold handler found no test naming it and vetoed the whole selection with
RUNNER_REACHED. In the last 40 PR runs that veto decided 14 of the 39
engine answers, every one on __cbm_rss_tree_hold.

A handler that names another role's literal starts that role. Reaching the
started role now reaches its starters too: their literal scan selects their
spawners, and the started role counts as named when a starter does. A role
that a test names directly and that another role also starts now selects
both sets of spawners; before, only the direct ones ran. Runner files that
define no suite are read for starters as well, since handlers live there.

The RUNNER_REACHED note for a role now carries the call chain from the
handler to the changed code, as the main-reach note does, and names the
role it starts when the veto comes from a starter.

Test: test_impact_engine_role_started_by_a_role_selects_its_spawners
(before the fix: run_all, "the change reaches the test-runner role
__fx_hold_role, which no test names").

Signed-off-by: Martin Vogel <[email protected]>
selection.py learns two things the CI needs to narrow test legs other than
Linux x86_64, where the coverage map is recorded:

- `platform-suites --repo DIR --rev REV...` lists the platform-specific
  runner suites: those whose defining test file holds a SKIP_PLATFORM( call
  or an #if/#ifdef on an OS or architecture macro (the two markers
  scripts/check-no-test-skips.sh names for a genuinely platform-specific
  test), and those tests/test_main.c registers under such a conditional.
  Every revision given is read and the results are united, so a change
  cannot declassify a suite by dropping its conditionals. Any git failure
  exits 1.
- `filter --whole-suites FILE` writes the view for such a leg: every listed
  suite runs whole there, its per-test tokens collapsed to the suite; a
  listed suite the answer did not select is added and marked optional (a
  build that does not register it drops it with a notice, as a full run
  never runs it); the answer's own required suites stay required. An
  unusable list runs everything.

Nothing consumes the new view yet. Contract cases in
tests/test_test_impact_selection.sh.

Signed-off-by: Martin Vogel <[email protected]>
Three changes to how PR CI consumes the test-impact selection (user
decisions R150 and R151, 2026-11-11).

Fork PRs. select-tests ran only when vars.TEST_IMPACT_MODE was `gate`, and
GitHub passes no repository variables to a pull_request run from a fork:
all 24 fork runs in the last 40 PR runs skipped the job, maintainer-started
ones included. The condition is now (vars.TEST_IMPACT_MODE || 'gate') ==
'gate'. Setting the variable to anything else (e.g. `shadow`) still turns
the gate off for same-repository and Dependabot PRs; for fork PRs the off
switch is the default in the expression on main. No pull_request_target,
no new permissions, no secrets.

Base tooling. select-tests built and ran the PR checkout's own engine,
test-impact-select.sh and selection.py, so a change to them could narrow
its own tests. The job now checks out the merge base (HEAD^1 of the merge
ref) into a worktree, builds that engine and runs that copy of the script
with --repo pointing at the PR checkout. test-impact-select.sh separates
the tooling (its own checkout) from the repository under test (--repo).
Until this lands, a PR's merge base has no --repo: the job then fails and,
as before, every test runs.

All legs. Only the Linux x86_64 test leg consumed a selection. A narrowed
answer now also produces portable/ (selection.py --whole-suites with the
platform-specific suites of the base and HEAD), and the arm64 and macOS
test-unix legs and the Windows leg run that view: platform-neutral suites
as selected, every platform-specific suite whole. No portable view, no
narrowing there. Sanitizer, TSan, MSan, diag and LSan lanes still run
every test. The shard harness already slices the selected list, so the
sharded arm64 leg and the shard-completeness proof need no change.

docs/SMART_CI.md and README.md follow; tests/test_lane_wiring_contract.sh
pins the new wiring.

Signed-off-by: Martin Vogel <[email protected]>
@DeusData
DeusData merged commit c90b1d9 into main Nov 12, 2026
41 of 42 checks passed
DeusData added a commit that referenced this pull request Nov 12, 2026
Brings in #2957 (sql became a self-maintained fork), #2959 and #2962.
Both sides added a self-maintained grammar fork, so every conflict keeps
both additions:

- MANIFEST.md summary: 164 grammars = 139 vendored-from-upstream + 20
  self-maintained (sql moved over on main, vba added here) + 5
  registry-disagreement; the ABI tally stays 9/78/77 (sql stayed ABI 15).
  The notes and the forks table carry both the sql and the vba rows.
- THIRD_PARTY.md: thirteen further self-maintained forks including sql and
  vba; main's sql paragraph, then this branch's tree-sitter-vba section.
- tests/test_parse_coverage.c: both test blocks and both sets of
  RUN_TEST registrations.
- scripts/vendored-checksums.txt: regenerated with
  security-vendored.sh --update; against main only the MANIFEST.md line
  and the five vba files differ.

Signed-off-by: Martin Vogel <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant