Repository navigation
fix(test-impact): started roles, fork PRs, base tooling, all test legs - #2959
Merged
Merged
Conversation
The test runner's __cbm_rss_tree_root role starts a __cbm_rss_tree_hold child, and no test names __cbm_rss_tree_hold itself: the test that spawns the root does. The role lift excludes every handler from its literal scan (a handler naming its own literal is no spawner), so a change reaching the hold handler found no test naming it and vetoed the whole selection with RUNNER_REACHED. In the last 40 PR runs that veto decided 14 of the 39 engine answers, every one on __cbm_rss_tree_hold. A handler that names another role's literal starts that role. Reaching the started role now reaches its starters too: their literal scan selects their spawners, and the started role counts as named when a starter does. A role that a test names directly and that another role also starts now selects both sets of spawners; before, only the direct ones ran. Runner files that define no suite are read for starters as well, since handlers live there. The RUNNER_REACHED note for a role now carries the call chain from the handler to the changed code, as the main-reach note does, and names the role it starts when the veto comes from a starter. Test: test_impact_engine_role_started_by_a_role_selects_its_spawners (before the fix: run_all, "the change reaches the test-runner role __fx_hold_role, which no test names"). Signed-off-by: Martin Vogel <[email protected]>
selection.py learns two things the CI needs to narrow test legs other than Linux x86_64, where the coverage map is recorded: - `platform-suites --repo DIR --rev REV...` lists the platform-specific runner suites: those whose defining test file holds a SKIP_PLATFORM( call or an #if/#ifdef on an OS or architecture macro (the two markers scripts/check-no-test-skips.sh names for a genuinely platform-specific test), and those tests/test_main.c registers under such a conditional. Every revision given is read and the results are united, so a change cannot declassify a suite by dropping its conditionals. Any git failure exits 1. - `filter --whole-suites FILE` writes the view for such a leg: every listed suite runs whole there, its per-test tokens collapsed to the suite; a listed suite the answer did not select is added and marked optional (a build that does not register it drops it with a notice, as a full run never runs it); the answer's own required suites stay required. An unusable list runs everything. Nothing consumes the new view yet. Contract cases in tests/test_test_impact_selection.sh. Signed-off-by: Martin Vogel <[email protected]>
Three changes to how PR CI consumes the test-impact selection (user decisions R150 and R151, 2026-11-11). Fork PRs. select-tests ran only when vars.TEST_IMPACT_MODE was `gate`, and GitHub passes no repository variables to a pull_request run from a fork: all 24 fork runs in the last 40 PR runs skipped the job, maintainer-started ones included. The condition is now (vars.TEST_IMPACT_MODE || 'gate') == 'gate'. Setting the variable to anything else (e.g. `shadow`) still turns the gate off for same-repository and Dependabot PRs; for fork PRs the off switch is the default in the expression on main. No pull_request_target, no new permissions, no secrets. Base tooling. select-tests built and ran the PR checkout's own engine, test-impact-select.sh and selection.py, so a change to them could narrow its own tests. The job now checks out the merge base (HEAD^1 of the merge ref) into a worktree, builds that engine and runs that copy of the script with --repo pointing at the PR checkout. test-impact-select.sh separates the tooling (its own checkout) from the repository under test (--repo). Until this lands, a PR's merge base has no --repo: the job then fails and, as before, every test runs. All legs. Only the Linux x86_64 test leg consumed a selection. A narrowed answer now also produces portable/ (selection.py --whole-suites with the platform-specific suites of the base and HEAD), and the arm64 and macOS test-unix legs and the Windows leg run that view: platform-neutral suites as selected, every platform-specific suite whole. No portable view, no narrowing there. Sanitizer, TSan, MSan, diag and LSan lanes still run every test. The shard harness already slices the selected list, so the sharded arm64 leg and the shard-completeness proof need no change. docs/SMART_CI.md and README.md follow; tests/test_lane_wiring_contract.sh pins the new wiring. Signed-off-by: Martin Vogel <[email protected]>
DeusData
added a commit
that referenced
this pull request
Nov 12, 2026
Brings in #2957 (sql became a self-maintained fork), #2959 and #2962. Both sides added a self-maintained grammar fork, so every conflict keeps both additions: - MANIFEST.md summary: 164 grammars = 139 vendored-from-upstream + 20 self-maintained (sql moved over on main, vba added here) + 5 registry-disagreement; the ABI tally stays 9/78/77 (sql stayed ABI 15). The notes and the forks table carry both the sql and the vba rows. - THIRD_PARTY.md: thirteen further self-maintained forks including sql and vba; main's sql paragraph, then this branch's tree-sitter-vba section. - tests/test_parse_coverage.c: both test blocks and both sets of RUN_TEST registrations. - scripts/vendored-checksums.txt: regenerated with security-vendored.sh --update; against main only the MANIFEST.md line and the five vba files differ. Signed-off-by: Martin Vogel <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Three fixes to the test-impact gate (smart CI, layer 2), in three commits:
__cbm_rss_tree_holdis started only by the__cbm_rss_tree_rootrolehandler, never by a test, so every change that reached its handler vetoed
the whole selection (RUNNER_REACHED). That veto decided 14 of the last 39
engine answers. A handler that names another role's literal now starts
that role: the started role is lifted with its starter's spawners. A role
that is both named by a test and started by another role now selects both
sets of spawners (before: only the direct ones, an under-selection). The
role note now carries the call chain, like the main-reach note.
platform-suiteslists the suites whose test file holdsSKIP_PLATFORM(or an
#if/#ifdefon an OS or architecture macro (the two markersscripts/check-no-test-skips.shnames), at the base and at HEAD, plussuites the runner registers under such a conditional: 80 suites today.
filter --whole-suiteswrites the view where those suites run whole.select-testsnow runs for fork PRs:(vars.TEST_IMPACT_MODE || 'gate') == 'gate'.Fork PRs get no repository variables, so all 24 fork runs in the sample
skipped the job. No
pull_request_target, no new permissions, no secrets.select-testsbuilds and runs the merge base's engine,test-impact-select.shandselection.pyagainst the PR checkout(
--repo), so a PR can no longer narrow its own tests by changing theselection tooling. This matters more now that fork PRs narrow.
test-unixlegs and the Windows leg now run theselection's portable view (platform-neutral suites as selected, every
platform-specific suite whole). Linux x86_64 runs the selection as
before. This reverses the 2026-10-16 x86-only decision (maintainer
decision, 2026-11-11).
CI scope (please review before merge)
pr.yml(select-tests: condition,steps, new
portableoutput;test: newtest_selection_portableinput)and
_test.yml(new boolean inputtest_selection_portable;test-unixfetches the selection on every leg when the portable view exists;
test-windowsgains a fetch step). Dry run and release do not pass aselection and are unchanged. TSan, MSan, diag, LSan, smoke, package
wrappers and UI legs are unchanged and still run every test.
select-testsstays outsideci-ok(a failure means everytest runs). The test legs stay gating; what changes is how much of the
suite list they run on arm64, macOS and Windows.
select-tests(ubuntu, ~10-15 job-min: engine build plusselection) now also runs on fork PRs. Savings from the portable view are
small today: in the 10 sampled selections the platform-specific suites
are 76-94% of each leg's suite time and the selections keep 47-176 of
~190 suites, so the modelled saving is a median of 0.2-0.6 minutes per
non-x86 leg (best case 1.4-7.8). See "Known limits".
runner on arm64, macOS and Windows; the selection download is a new step
on those legs (an artifact download failure fails the leg, as it already
can on x86). The harness selection path itself is exercised on every leg
by the existing contract step 2c.
--repo, so thisPR's own
select-testsfails and every test runs. That is the designedfail-open path.
TEST_IMPACT_MODE=shadow(same-repo and DependabotPRs); for fork PRs, change the
'gate'default inpr.ymlon main; tokeep narrowing on x86 only, pass
test_selection_portable: false.Known limits (not changed here)
RUNNER_REACHED that the role veto had masked: the runner's main reaches
the memory core through its startup path, and the memory core is
"reached" because the parse-gap rule cascades. A gap's text is matched
against the short name of every reached node, so gaps fire on generic
names (
idx,n,index,used; 312 distinct firings in one replay),and a file-scope gap seeds its whole file. In the sampled replays it was
src/foundation/compat_thread.c:109(__attribute__((used)), fired onused), which seedscbm_mutex_init, called by the memory core. Anexperiment that ignores attribute words only moves the way in
(
main -> tf_setup_cache_sentinel -> ... -> platform.cbm_app_local_dir,504-1224 firings). Follow-up (decided): gaps will match only reached
function, variable and macro names, with no whole-file seeding for a
file-scope gap, validated by a replay miss audit before it gates.
all 5 polluted verdicts in the sample came from main commits that changed
runner inputs (a vendored grammar; the harness and Makefile), which the
ancestor's coverage map does not know. Keeping run_all there is required.
Follow-up (decided): cut the coverage producer's latency so the exact
merge-base bundle exists (it was missing in 51% of the sampled runs).
Proof
test_impact_engine_role_started_by_a_role_selects_its_spawnersis RED on main (run_all, "the change reaches the test-runner role
__fx_hold_role, which no test names") and GREEN with the fix; suites
test_impact_engine (22), test_impact, test_impact_artifact,
test_impact_runner_filter and runner_isolation (99) pass under ASan+UBSan.
on
__cbm_rss_tree_hold; the fixed engine no longer does (it then stopsat the main-reach veto described under "Known limits").
narrowed came from fork runs whose
select-testswas skipped; with thischange they reach the test legs.
main (
--whole-suitesunknown).linter passes; shellcheck clean on the touched scripts.