Skip to content

Commit c90b1d9

Browse files
authored
Merge pull request #2959 from DeusData/fix/test-impact-gate
fix(test-impact): started roles, fork PRs, base tooling, all test legs
2 parents b3ec08d + 74affb1 commit c90b1d9

10 files changed

Lines changed: 629 additions & 100 deletions

File tree

‎.github/workflows/_test.yml‎

Lines changed: 25 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,13 @@ on:
2525
type: string
2626
default: all
2727
test_selection:
28-
description: 'PR test selection (smart CI): the artifact the select-tests job uploaded; the Linux x86_64 test-unix leg runs only its tests, the arm64, macOS and Windows legs run every test. Empty = every test (dry run, release, and any PR the selection does not narrow)'
28+
description: 'PR test selection (smart CI): the artifact the select-tests job uploaded; the Linux x86_64 test-unix leg runs only its tests. Empty = every test (dry run, release, and any PR the selection does not narrow)'
2929
type: string
3030
default: ''
31+
test_selection_portable:
32+
description: 'The test_selection artifact also holds portable/: the view of the legs the coverage is not recorded on (arm64, macOS, Windows), where every platform-specific suite runs whole; those legs run it. false = they run every test'
33+
type: boolean
34+
default: false
3135

3236
permissions:
3337
contents: read
@@ -226,11 +230,14 @@ jobs:
226230
ccache-test-${{ matrix.os }}-${{ matrix.cc }}-
227231
228232
- name: Fetch the test selection
229-
# Only the Linux x86_64 leg narrows (user decision 2026-10-16): the
230-
# coverage that justifies an omission comes from the Linux x86_64
231-
# producer, so arm64, macOS and Windows code under #ifdef has static
232-
# evidence only and those legs run every test.
233-
if: ${{ inputs.test_selection != '' && runner.os == 'Linux' && runner.arch == 'X64' }}
233+
# The coverage that justifies an omission is recorded on Linux x86_64:
234+
# that leg runs the selection as the engine answered it. arm64 and
235+
# macOS run its portable view, where every platform-specific suite
236+
# (scripts/test-impact/selection.py platform-suites) runs whole, since
237+
# their #ifdef code has static evidence only (user decision 2026-11-11,
238+
# R151; until then these legs ran every test). No portable view: every
239+
# test.
240+
if: ${{ inputs.test_selection != '' && ((runner.os == 'Linux' && runner.arch == 'X64') || inputs.test_selection_portable) }}
234241
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
235242
with:
236243
name: ${{ inputs.test_selection }}
@@ -242,7 +249,7 @@ jobs:
242249
CBM_TEST_SHARD: ${{ matrix.shard }}
243250
CBM_TEST_LEG: ${{ matrix.os }}-${{ matrix.cc }}
244251
CBM_SKIP_PERF: ${{ inputs.skip_perf && '1' || '' }}
245-
CBM_TEST_SELECTION_DIR: ${{ inputs.test_selection != '' && runner.os == 'Linux' && runner.arch == 'X64' && format('{0}/test-selection', runner.temp) || '' }}
252+
CBM_TEST_SELECTION_DIR: ${{ inputs.test_selection != '' && runner.os == 'Linux' && runner.arch == 'X64' && format('{0}/test-selection', runner.temp) || inputs.test_selection != '' && inputs.test_selection_portable && format('{0}/test-selection/portable', runner.temp) || '' }}
246253
CCACHE_DIR: ${{ github.workspace }}/.ccache
247254
CCACHE_MAXSIZE: 1500M
248255

@@ -635,6 +642,15 @@ jobs:
635642
ccache-test-${{ matrix.os }}-${{ matrix.msystem }}-${{ github.ref }}-
636643
ccache-test-${{ matrix.os }}-${{ matrix.msystem }}-
637644
645+
# The portable view of the test selection (see test-unix): every
646+
# platform-specific suite runs whole here.
647+
- name: Fetch the test selection
648+
if: ${{ inputs.test_selection != '' && inputs.test_selection_portable }}
649+
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
650+
with:
651+
name: ${{ inputs.test_selection }}
652+
path: ${{ runner.temp }}/test-selection
653+
638654
- name: Test
639655
shell: msys2 {0}
640656
# AddressSanitizer is unavailable on native ARM64 Windows (LLVM ships no
@@ -662,8 +678,8 @@ jobs:
662678
CBM_TEST_SHARD: ${{ matrix.shard }}
663679
CBM_TEST_LEG: ${{ matrix.os }}-${{ matrix.msystem }}
664680
CBM_SKIP_PERF: ${{ inputs.skip_perf && '1' || '' }}
665-
# Windows legs run every test: see the Linux x86_64-only fetch in test-unix.
666-
CBM_TEST_SELECTION_DIR: ''
681+
# The portable view, as on test-unix's arm64 and macOS legs.
682+
CBM_TEST_SELECTION_DIR: ${{ inputs.test_selection != '' && inputs.test_selection_portable && format('{0}/test-selection/portable', runner.temp) || '' }}
667683
CCACHE_DIR: ${{ github.workspace }}/.ccache
668684
CCACHE_MAXSIZE: 1500M
669685

‎.github/workflows/pr.yml‎

Lines changed: 23 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -42,40 +42,53 @@ jobs:
4242
lanes: ${{ needs.changes.outputs.lanes }}
4343

4444
# ── Smart CI, PR CI only: run only the tests this change can reach.
45-
# Active only while the repository variable TEST_IMPACT_MODE is `gate`;
46-
# unset (shadow) every test runs and test-impact-shadow only reports.
45+
# On by default, for every PR: fork PRs get no `vars`, so the expression
46+
# supplies `gate` itself (the test-impact-shadow job never needed vars).
47+
# Off switches: the repository variable TEST_IMPACT_MODE set to anything
48+
# but `gate` (e.g. `shadow`) turns it off for same-repo and Dependabot
49+
# PRs; the default below, changed on main, turns it off for fork PRs too
50+
# (each PR's merge ref picks up main's pr.yml on its next sync).
4751
# test-impact-select.sh --gate narrows only on a well-formed answer: a
4852
# failure of this job, a run-all answer or a lane that selects nothing
4953
# leaves `test` without a selection, and every test runs. ci-ok does not
5054
# need this job: when it breaks, PRs cost more minutes, never coverage.
51-
# Dry runs and releases never pass a selection.
55+
# The engine, the selection scripts and selection.py come from the merge
56+
# base, never from the PR: a change cannot narrow its own tests by
57+
# changing what selects them. Dry runs and releases never pass a selection.
5258
select-tests:
5359
needs: [changes]
54-
if: ${{ vars.TEST_IMPACT_MODE == 'gate' }}
60+
if: ${{ (vars.TEST_IMPACT_MODE || 'gate') == 'gate' }}
5561
runs-on: ubuntu-24.04
5662
timeout-minutes: 30
5763
permissions:
5864
contents: read
5965
actions: read # the merge base's team-artifact bundle (test-impact-artifact.yml)
6066
outputs:
6167
selection: ${{ steps.select.outputs.selection }}
68+
portable: ${{ steps.select.outputs.portable }}
6269
steps:
6370
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
6471
with:
6572
# Full history: the engine certifies the history it diffs and refuses
6673
# a shallow clone, so a depth-2 checkout always answered `all`.
6774
fetch-depth: 0
6875
persist-credentials: false
69-
- name: Build this checkout's engine
76+
- name: Check out the merge base's selection tooling
77+
run: git worktree add --detach "$RUNNER_TEMP/base-tooling" HEAD^1
78+
- name: Build the merge base's engine
7079
timeout-minutes: 20
7180
run: |
7281
APT_RETRIES=3; [ -f /etc/apt/apt-mirrors.txt ] && APT_RETRIES=0; sudo apt-get -o Acquire::Retries=$APT_RETRIES -o Acquire::http::Timeout=30 -o Acquire::https::Timeout=30 --error-on=any update && sudo apt-get -o Acquire::Retries=$APT_RETRIES -o Acquire::http::Timeout=30 -o Acquire::https::Timeout=30 install -y zlib1g-dev
82+
cd "$RUNNER_TEMP/base-tooling"
7383
scripts/build.sh CC=gcc CXX=g++
7484
- name: Select the tests
7585
id: select
7686
env:
7787
GH_TOKEN: ${{ github.token }}
78-
run: scripts/ci/test-impact-select.sh --binary build/c/codebase-memory-mcp --out "$RUNNER_TEMP/test-impact" --gate "$RUNNER_TEMP/test-selection"
88+
run: |
89+
"$RUNNER_TEMP/base-tooling/scripts/ci/test-impact-select.sh" --repo "$GITHUB_WORKSPACE" \
90+
--binary "$RUNNER_TEMP/base-tooling/build/c/codebase-memory-mcp" \
91+
--out "$RUNNER_TEMP/test-impact" --gate "$RUNNER_TEMP/test-selection"
7992
- name: Upload the selection
8093
if: ${{ steps.select.outputs.selection == 'narrowed' }}
8194
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
@@ -103,8 +116,11 @@ jobs:
103116
lanes: ${{ needs.changes.outputs.lanes }}
104117
# Smart CI: only a successful select-tests run that narrowed passes a
105118
# selection; skipped (shadow), failed, run-all and nothing-selected
106-
# answers all run every test.
119+
# answers all run every test. The Linux x86_64 leg runs it as answered;
120+
# arm64, macOS and Windows run its portable view (platform-specific
121+
# suites whole) when select-tests produced one, else every test.
107122
test_selection: ${{ needs.select-tests.result == 'success' && needs.select-tests.outputs.selection == 'narrowed' && 'test-selection' || '' }}
123+
test_selection_portable: ${{ needs.select-tests.result == 'success' && needs.select-tests.outputs.selection == 'narrowed' && needs.select-tests.outputs.portable == 'narrowed' }}
108124
# Perf assertions are timing-sensitive on shared runners; they stay in
109125
# dry runs and releases where a flaky red does not block a merge.
110126
skip_perf: true

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -369,7 +369,7 @@ Use the graph to explain which tests a change reaches, and use that evidence to
369369

370370
**Narrowing currently supports C/C++ family code with this repository's test conventions or declared equivalents.** It is not general test selection for every language CBM indexes. Missing, stale, incompatible, or incomplete evidence broadens the selection or returns `run_all`; static reachability without admitted coverage never omits a suite. Begin with report-only shadow mode and validate your harness before enabling gating.
371371

372-
In this repository, test narrowing is wired to the Linux x86_64 test leg when `TEST_IMPACT_MODE=gate`; release and dry-run workflows retain full execution. See the [Smart CI guide](docs/SMART_CI.md) for supported conventions, coverage admission, setup, version requirements, and fallback behavior.
372+
In this repository, test narrowing is on for every PR unless `TEST_IMPACT_MODE` says otherwise: the Linux x86_64 test leg runs the selection, and the arm64, macOS and Windows legs run it with every platform-specific suite whole; release and dry-run workflows retain full execution. See the [Smart CI guide](docs/SMART_CI.md) for supported conventions, coverage admission, setup, version requirements, and fallback behavior.
373373

374374
### Edge types (selected)
375375
- `CALLS`: a callable is invoked at the source site

‎docs/SMART_CI.md‎

Lines changed: 63 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,11 @@ change can reach. This page explains how this repository does it and how
55
another repository on GitHub Actions could do the same with the released
66
`codebase-memory-mcp` binary.
77

8-
Status: experimental. Every answer carries `"experimental": true`. The
9-
selection narrows real test runs only while the repository variable
10-
`TEST_IMPACT_MODE` is `gate`; otherwise it runs in report-only (shadow) mode.
8+
Status: experimental. Every answer carries `"experimental": true`. In this
9+
repository the selection narrows real test runs by default, fork PRs
10+
included; the repository variable `TEST_IMPACT_MODE` set to anything but
11+
`gate` turns it off for same-repository PRs (fork PRs get no variables: see
12+
(d)), and it then runs in report-only (shadow) mode.
1113

1214
> **Read the scope section before you start.** The lane layer is a pattern
1315
> any repository can copy. The test layer narrows only C/C++ code whose tests
@@ -27,7 +29,7 @@ cannot be affected. When it is unsure, it runs everything:
2729
- no coverage map, or a coverage map that is not admitted (wrong commit, more
2830
than 7 days old, made for another platform or harness);
2931
- a change that reaches the test runner's `main`, or a test-runner role
30-
(below) that no test names;
32+
(below) that no test names and that no role with known spawners starts;
3133
- a role table entry the engine cannot read, or a macro naming a role's
3234
literal that it cannot follow;
3335
- any failure of the selection job itself.
@@ -40,7 +42,8 @@ Each of these makes the run cost more CI minutes. None of them loses coverage.
4042
changed files ──► Layer 1: lane selector ──► which CI jobs run
4143
(scripts/ci/select-lanes.sh, your own rules)
4244
│
43-
▼ (the Linux x86_64 test leg only, in this repo)
45+
▼ (every test leg in this repo; off Linux x86_64,
46+
│ platform-specific suites run whole)
4447
merge base + PR ─► Layer 2: test-impact engine ─► which suites / tests run
4548
(codebase-memory-mcp test-impact select
4649
+ team-artifact bundle from main)
@@ -251,7 +254,7 @@ everything (`builtin:test-impact-config`).
251254

252255
```yaml
253256
select-tests:
254-
if: ${{ vars.TEST_IMPACT_MODE == 'gate' }}
257+
if: ${{ (vars.TEST_IMPACT_MODE || 'gate') == 'gate' }}
255258
runs-on: ubuntu-24.04
256259
permissions:
257260
contents: read
@@ -263,12 +266,17 @@ everything (`builtin:test-impact-config`).
263266
with:
264267
fetch-depth: 0
265268
persist-credentials: false
269+
# The selection scripts come from the merge base, never from the PR.
270+
- run: git worktree add --detach "$RUNNER_TEMP/base-tooling" HEAD^1
266271
- name: Install codebase-memory-mcp
267272
run: <the pinned install from Prerequisites>
268273
- id: select
269274
env:
270275
GH_TOKEN: ${{ github.token }}
271-
run: scripts/ci/test-impact-select.sh --binary "$RUNNER_TEMP/cbm/codebase-memory-mcp" --out "$RUNNER_TEMP/test-impact" --gate "$RUNNER_TEMP/test-selection"
276+
run: |
277+
"$RUNNER_TEMP/base-tooling/scripts/ci/test-impact-select.sh" --repo "$GITHUB_WORKSPACE" \
278+
--binary "$RUNNER_TEMP/cbm/codebase-memory-mcp" \
279+
--out "$RUNNER_TEMP/test-impact" --gate "$RUNNER_TEMP/test-selection"
272280
- if: ${{ steps.select.outputs.selection == 'narrowed' }}
273281
uses: actions/upload-artifact@<pinned-sha>
274282
with:
@@ -297,7 +305,10 @@ answer writes:
297305
- `DIR/test-only.txt`: one token per line, either `suite` (run the whole
298306
suite) or `suite:test`;
299307
- `DIR/optional-suites.txt`: selected suites that are registered only under a
300-
preprocessor condition. A build may lack them.
308+
preprocessor condition. A build may lack them;
309+
- `DIR/portable/` (with `portable=narrowed`): the same two files for the legs
310+
the coverage is not recorded on, where every platform-specific suite runs
311+
whole (see below).
301312

302313
Everything else means `all`: a `run_all` answer, an unknown schema or filter
303314
encoding, a filter that disagrees with the lane's suite list, a crash, and a
@@ -338,25 +349,49 @@ spawner. A reached handler that no test names, a table entry the engine cannot
338349
read, and a change reaching `main` any other way still run everything; the
339350
engine note names the call chain into `main`.
340351

341-
**Only the legs the coverage speaks for should narrow.** Here the coverage
342-
comes from one Linux x86_64 clang-21 producer. Only the Linux x86_64 test leg
343-
downloads the selection (`runner.os == 'Linux' && runner.arch == 'X64'` in
344-
`_test.yml`). The arm64, macOS and Windows legs always run everything,
345-
because code under architecture or platform `#ifdef`s has static evidence
346-
only there. That leg builds with gcc, not clang: code under a compiler `#if`
347-
has static evidence only too, and `--platform` names the producer's platform,
348-
so the engine's platform check admits the map by construction.
349-
350-
### (d) Flip the variable
351-
352-
Shadow is the default: the variable is unset, `select-tests` is skipped, and
353-
every test runs. When the evidence is in, set the repository variable:
352+
A role handler may start another role's child: `__cbm_rss_tree_root` starts
353+
`__cbm_rss_tree_hold`. A handler that names another role's literal starts that
354+
role, so a reached started role is lifted with its starter: the tests that
355+
name the starter run, and the started role needs no test of its own.
356+
357+
**Platform code narrows only where the coverage speaks for it.** Here the
358+
coverage comes from one Linux x86_64 clang-21 producer. The Linux x86_64 test
359+
leg runs the selection as the engine answered it (`runner.os == 'Linux' &&
360+
runner.arch == 'X64'` in `_test.yml`). The arm64, macOS and Windows legs run
361+
its portable view (`DIR/portable/`): every platform-specific suite runs whole
362+
there, selected or not, because code under architecture or platform `#ifdef`s
363+
has static evidence only on those legs. A suite is platform-specific when its
364+
test file holds a `SKIP_PLATFORM(` call or an `#if`/`#ifdef` on an OS or
365+
architecture macro (the two markers `scripts/check-no-test-skips.sh` names for
366+
a genuinely platform-specific test), at the base or at HEAD, or when the
367+
runner registers it under such a conditional (`selection.py platform-suites`).
368+
Platform-neutral suites narrow on every leg. The x86_64 leg builds with gcc,
369+
not clang: code under a compiler `#if` has static evidence only too, and
370+
`--platform` names the producer's platform, so the engine's platform check
371+
admits the map by construction.
372+
373+
### (d) The mode switch
374+
375+
The expression `(vars.TEST_IMPACT_MODE || 'gate') == 'gate'` makes gate the
376+
default: a pull request from a fork gets no repository variables, so a test of
377+
`vars.TEST_IMPACT_MODE == 'gate'` alone never narrowed a fork PR. Never move
378+
the job to `pull_request_target` to get the variables: that runs fork code
379+
with a write token. Off switches:
354380

355381
```bash
356-
gh variable set TEST_IMPACT_MODE --body gate # gate
357-
gh variable delete TEST_IMPACT_MODE # back to shadow
382+
gh variable set TEST_IMPACT_MODE --body shadow # same-repository PRs: shadow
383+
gh variable delete TEST_IMPACT_MODE # back to the default (gate)
358384
```
359385

386+
For fork PRs the off switch is the default in the expression: change `'gate'`
387+
to `'shadow'` on main, and each PR picks it up with its next merge ref. A
388+
repository that wants shadow until its evidence is in writes `'shadow'` there
389+
from the start.
390+
391+
The selection runs the merge base's engine, `test-impact-select.sh` and
392+
`selection.py` (`--repo` points them at the PR checkout), so a PR cannot
393+
narrow its own tests by changing them.
394+
360395
Do not add `select-tests` to your required checks or to the aggregate gate.
361396
When it breaks, PRs cost more minutes. They do not lose coverage.
362397

@@ -439,7 +474,8 @@ map incompatible until the producer publishes a new one.
439474

440475
## Rollout and evidence
441476

442-
1. **Shadow.** Run (a) and (b). Leave `TEST_IMPACT_MODE` unset.
477+
1. **Shadow.** Run (a) and (b) with `'shadow'` as the default in the
478+
`select-tests` expression (d).
443479
2. **Measure.** For each PR run, compare `selection-engine.json` with what
444480
failed in that run's full test legs.
445481
- A **miss** is a test that failed in the full run but was not selected.
@@ -448,8 +484,8 @@ map incompatible until the producer publishes a new one.
448484

449485
Nothing in this repository computes misses for Layer 2 yet. Layer 1's
450486
history replay (`replay-selector.sh`) is the model to follow.
451-
3. **Gate.** Set `TEST_IMPACT_MODE=gate`. Keep the shadow job running, so that
452-
the evidence keeps coming.
487+
3. **Gate.** Make `'gate'` the default (this repository's state). Keep the
488+
shadow job running, so that the evidence keeps coming.
453489

454490
**Reading the step summary** (`### test-impact engine`):
455491

@@ -469,7 +505,7 @@ Common run-all reasons:
469505
| `RULE_RUN_ALL` | A `run_all` rule matched (see `run_all_rules`). |
470506
| `UNMAPPED_FILE` | A changed file matched no rule and is not C-family source. |
471507
| `RULE_TARGET_UNKNOWN` | A `referencing_tests` path that no test names (without `unreferenced: "ignore"`). |
472-
| `RUNNER_REACHED` | The change reaches the test runner's `main` (or a child role none of whose literals a test names). |
508+
| `RUNNER_REACHED` | The change reaches the test runner's `main` (or a child role none of whose literals a test names and that no role with known spawners starts). The engine note names the call chain. |
473509
| `GIT_UNAVAILABLE` | No git, a shallow clone, or a diff that cannot be read in full. |
474510
| `CONFIG_INVALID` / `POLICY_UNAVAILABLE` | The merge base's config or rules do not parse. |
475511

0 commit comments

Comments
 (0)