Skip to content

Commit 11cd290

Browse files
luisleo526claude
andcommitted
CI: PR tier excludes the slow rows on Release and kernel-only, with registration floors, and defers the slow verifier self-test classes (D1, CR-195)
On pull_request the Release matrix leg and the kernel-only job now pass --exclude-label slow, like Debug and sanitizers already did. Push to main, workflow_dispatch and release keep every row and every class, so coverage is unchanged and only its timing moves. - ci_preflight.py: the two policy checks that forbade this (only PR Debug matrix jobs may exclude slow rows; kernel-only must keep its full population) become one exact rule with must-fail forms; docs/ci.md says the same. This REVERSES the previous policy. - ci_verify.py: EXCLUDED_REGISTERED_MIN gains release 895 and kernel 309, derived on one registered-count basis from the PR-world job data (to be confirmed by a lab ctest -N). - test_ci_verify.py classes are partitioned into FAST (13) and SLOW (9) in scripts/ci_verifier_classes.py; the preflight verifier-tests stage runs the FAST classes on every pull request and the SLOW ones only when scripts/ci_verifier_paths.py finds a path the verifier reads; any unknown path or class runs. Non-PR events run everything. - New tests assert that push, dispatch and release carry no exclusion and no class filter. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent 18d8f3c commit 11cd290

11 files changed

Lines changed: 732 additions & 53 deletions

‎.github/workflows/ci.yml‎

Lines changed: 23 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,8 @@ jobs:
4343
# Whether the change is documentation only; scripts/ci_docs_only.py holds the
4444
# rule, and every doubt answers no. On a docs-only pull request or push the
4545
# proof jobs below skip and the build aggregate accepts exactly that skip.
46-
# Preflight, where the documentation guards run, does not wait for it, and
47-
# docs.yml builds the site as before. A few seconds of git: the light pool
46+
# Preflight consumes its independent verifier-class answer, and docs.yml
47+
# builds the site as before. A few seconds of git: the light pool
4848
# class, with the public fragment's hosted fallback.
4949
changes:
5050
runs-on: >-
@@ -59,6 +59,7 @@ jobs:
5959
timeout-minutes: 360
6060
outputs:
6161
docs_only: ${{ steps.classify.outputs.docs_only }}
62+
verifier_slow: ${{ steps.verifier.outputs.verifier_slow }}
6263
steps:
6364
# The tested commit and its parents: a pull request's merge and the base
6465
# it merged into, a push's commit and its parent.
@@ -83,7 +84,23 @@ jobs:
8384
echo "docs_only=false" >> "$GITHUB_OUTPUT"
8485
fi
8586
87+
# This helper uses the base ci_docs_only.py staged above to read the
88+
# same tested diff. A base without the new rule runs every class.
89+
- name: Classify verifier self-tests
90+
id: verifier
91+
env:
92+
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
93+
run: |
94+
if git show HEAD^1:scripts/ci_verifier_paths.py > "$RUNNER_TEMP/ci_verifier_paths.py"; then
95+
python3 "$RUNNER_TEMP/ci_verifier_paths.py" --root . --github-output "$GITHUB_OUTPUT"
96+
else
97+
echo "verifier SLOW: the base holds no scripts/ci_verifier_paths.py"
98+
echo "verifier_slow=true" >> "$GITHUB_OUTPUT"
99+
fi
100+
86101
preflight:
102+
needs: changes
103+
if: ${{ !cancelled() }}
87104
runs-on: >-
88105
${{ github.event_name != 'pull_request_target' &&
89106
github.actor != 'dependabot[bot]' && github.triggering_actor != 'dependabot[bot]' &&
@@ -93,6 +110,7 @@ jobs:
93110
fromJSON('["self-hosted","pf-ci","pf-ci-x64"]') || 'ubuntu-24.04' }}
94111
env:
95112
PF_CI_LABELS: '["self-hosted","pf-ci","pf-ci-x64"]'
113+
PINEFORGE_VERIFIER_SLOW: ${{ needs.changes.result == 'success' && needs.changes.outputs.verifier_slow || 'true' }}
96114
# 360 minutes is GitHub's job maximum and deliberately the only limit:
97115
# its verifier-tests stage runs test_ci_verify.py across parallel unittest
98116
# shards, and no stage of this preflight carries a timeout of its own any
@@ -237,7 +255,7 @@ jobs:
237255
# ratio -- for a runner whose CPU accounting is unusable; record the
238256
# measured reason here and in docs/ci.md before setting it.
239257
PINEFORGE_RUNTIME_BUDGET_CANDIDATE_ONLY: '0'
240-
run: python3 scripts/ci_verify.py ${{ matrix.build_type == 'Release' && 'release' || 'debug' }} --build-dir build --jobs "$(getconf _NPROCESSORS_ONLN)" --ccache ${{ github.event_name == 'pull_request' && matrix.build_type == 'Debug' && '--exclude-label slow' || '' }}
258+
run: python3 scripts/ci_verify.py ${{ matrix.build_type == 'Release' && 'release' || 'debug' }} --build-dir build --jobs "$(getconf _NPROCESSORS_ONLN)" --ccache ${{ github.event_name == 'pull_request' && '--exclude-label slow' || '' }}
241259

242260
- name: Stage and summarize diagnostics
243261
if: always()
@@ -340,7 +358,7 @@ jobs:
340358
fromJSON('["self-hosted","pf-ci","pf-ci-x64-16"]') || 'ubuntu-24.04' }}
341359
env:
342360
PF_CI_LABELS: '["self-hosted","pf-ci","pf-ci-x64-16"]'
343-
# 22.3 min on the standard runner at 0d76a099; every event runs the whole
361+
# PRs exclude the measured slow rows; push and dispatch run the whole
344362
# kernel set. 360 minutes is the platform maximum, the only hang catcher
345363
# (owner rule, 2026-10-12).
346364
timeout-minutes: 360
@@ -380,7 +398,7 @@ jobs:
380398
uses: ./.github/actions/setup-live-curl
381399

382400
- name: Verify (kernel)
383-
run: python3 scripts/ci_verify.py kernel --build-dir build-kernel --jobs "$(getconf _NPROCESSORS_ONLN)" --ccache --curl-dir "${{ steps.curl-deps.outputs.curl-dir }}"
401+
run: python3 scripts/ci_verify.py kernel --build-dir build-kernel --jobs "$(getconf _NPROCESSORS_ONLN)" --ccache --curl-dir "${{ steps.curl-deps.outputs.curl-dir }}" ${{ github.event_name == 'pull_request' && '--exclude-label slow' || '' }}
384402

385403
- name: Stage and summarize diagnostics
386404
if: always()

‎docs/ci.md‎

Lines changed: 88 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -134,7 +134,11 @@ Only the native cell accepts the caller's existing PR-only slow-label exclusion.
134134
The kernel profile also gates the CTest row count: `tests/CMakeLists.txt`
135135
drops every test TU whose include closure reaches `pineforge/source/` or
136136
`compat/pine/`, so a lane whose native witnesses shared a TU with an adapter
137-
twin would leave the kernel-only gate without any failure. The campaign's
137+
twin would leave the kernel-only gate without any failure. On a pull request
138+
the kernel-only leg excludes the `slow` rows, so that leg is gated by the
139+
registration floor and the selection identity described below, and the run
140+
floor binds the push and dispatch runs, which run the whole kernel set
141+
([PR tier versus MAIN/RELEASE](#pr-tier-versus-mainrelease-what-is-excluded-where)). The campaign's
138142
per-profile row floors live in ONE table in `scripts/ci_verify.py`,
139143
`PRIVATE_FIXTURE_PROFILE_FLOORS`, which states for each of `release` and
140144
`kernel` the profile's full overlay-present population and how many pinned
@@ -183,14 +187,15 @@ and records
183187
that left the selection or could not start, a lost
184188
registration below the PR registration floor, unreadable enumeration, or
185189
missing label fails; with the overlay present every skip fails, as before.
186-
The PR registration floors after lane FIX-E1E2 are 773
187-
for Debug and sanitizers and 782 for native (653 and 662 at the INT24 base,
188-
plus wave G's six rows, wave H's twenty-four, INT26's own tape row, INT27's
189-
thirteen, XSYM-D's four, K-SESSION-WINDOWS' four, INT28's twenty-four,
190-
INT28-FIX's four, INT29's seven, W15-KERNEL-CAL's two, INT30's twenty-four,
191-
TAIL-I's one, XAU-CAL's four and FIX-E1E2's two). The full-run release and
192-
kernel floors are the table's own numbers above (603 and 306 rows absent, 910
193-
and 337 with the overlay present); full runs do not exclude a label.
190+
The PR registration floors are the keys of `EXCLUDED_REGISTERED_MIN` in
191+
`scripts/ci_verify.py`, one for each profile whose pull-request leg excludes
192+
the label; their values are in
193+
[PR tier versus MAIN/RELEASE](#pr-tier-versus-mainrelease-what-is-excluded-where).
194+
The full-run release and kernel floors are the table's own numbers above (603
195+
and 306 rows absent, 910 and 337 with the overlay present). A pull request's
196+
release and kernel legs exclude the label and are held to the registration
197+
floor instead; the full runs on push and dispatch exclude no label and are
198+
held to these run floors.
194199

195200
CR-196 splits the ownership manifest controls into four CTest shards,
196201
`test_selected_window_ownership_manifest_controls_shard_1of4` through
@@ -722,10 +727,11 @@ as diagnostics.
722727

723728
Superseded pull-request runs are canceled. Main/post-merge and manual CI runs
724729
use distinct concurrency groups and remain uncanceled. A PR that changes more
725-
than documentation runs preflight, both Release jobs, kernel-only and the
726-
parity subset with their full sets;
727-
both Debug jobs, sanitizers and native-live run the registered set excluding
728-
the 29 CTest rows labelled `slow` in `tests/CMakeLists.txt`. The first 27 were
730+
than documentation runs preflight and the parity subset with its full set;
731+
both Release jobs, both Debug jobs, kernel-only, sanitizers and the native cell
732+
of native-live run the registered set excluding
733+
the 29 CTest rows labelled `slow` in `tests/CMakeLists.txt`
734+
([PR tier versus MAIN/RELEASE](#pr-tier-versus-mainrelease-what-is-excluded-where)). The first 27 were
729735
chosen from INT23/INT24 job logs: over 60 seconds in either sanitizer run or
730736
over 30 seconds in either Debug run. Preflight starts at once, and the proof
731737
jobs start alongside it as soon as the few-second `changes` job has decided
@@ -772,6 +778,67 @@ The full corpus sweep remains a separate acceptance step. The nightly and
772778
manual corpus workflow, and the maintainers' full parity verification, keep
773779
the broader population running.
774780

781+
### PR tier versus MAIN/RELEASE: what is excluded where
782+
783+
Decision D1 moves the slow work out of the pull-request tier and keeps all of
784+
it on push to `main`. Only the `pull_request` event changes. Push,
785+
`workflow_dispatch` and the maintainers' full verification keep every CTest row
786+
and every verifier class, so coverage is unchanged and only the time at which a
787+
regression is first seen moves. `release.yml` is unchanged.
788+
789+
| Event | Release jobs | `kernel-only` | Debug jobs, `sanitizers`, native cell | Verifier classes in `preflight` |
790+
|---|---|---|---|---|
791+
| pull request | exclude `slow` | exclude `slow` | exclude `slow` | FAST always; SLOW only when a path the verifier reads changed or the diff cannot be read |
792+
| push to `main`, `workflow_dispatch` | every row | every row | every row | every class |
793+
794+
The label `slow` is `PINEFORGE_PR_SLOW_TESTS` in `tests/CMakeLists.txt`: 29
795+
rows, among them `test_ci_verify`, the serial run of
796+
`scripts/test_ci_verify.py`, which dominates a full Release or kernel job.
797+
Debug, `sanitizers` and the native cell of `native-live` already excluded it on
798+
a pull request; D1 adds both Release jobs and `kernel-only`. A row a profile
799+
does not register carries no label there, and the kernel profile registers
800+
fewer of the 29 than Release does. `live-sanitizers`, `live-tsan` and the
801+
parity subset are unchanged.
802+
803+
The verifier's own self-test is split by TestCase class. A class is FAST only
804+
when its measured total in the serial run log is under 5 s and its passing
805+
runtime is measured; every other class is SLOW, and a class in neither list
806+
runs. On a pull request the `verifier-tests` stage of `preflight` runs the FAST
807+
classes always. It runs the SLOW classes only when `scripts/ci_verifier_paths.py`,
808+
the one place the path list lives, reports that the change touches a path the
809+
verifier reads (`include/pineforge`, `src/c_abi.cpp`, `tests/`, `runner/`, the
810+
workflows and the CI scripts among them); an unknown or unreadable diff
811+
runs the SLOW classes. Push to `main` and `workflow_dispatch` run every class.
812+
The CTest row `test_ci_verify` stays in the `slow` population: the pull-request
813+
legs exclude it by label and every full run executes it.
814+
815+
A pull-request leg that excludes the label is held to a registration floor in
816+
place of the run floor: the registered rows must be at least the larger of
817+
that profile's `EXCLUDED_REGISTERED_MIN` key and its overlay-aware run floor,
818+
the selection must remove some rows and leave some, and, with the overlay
819+
absent, the rows that ran plus the pinned skips must equal the selection with
820+
no other skip (the `--exclude-label` branch of `enforce_test_floor` in
821+
`scripts/ci_verify.py`). `EXCLUDED_REGISTERED_MIN` holds 876 for `debug`, 867
822+
for `sanitizers` and 963 for `native`, and with D1 895 for `release` and 309
823+
for `kernel`. The two new keys are derived, not measured: each is the
824+
registered count in the sampled public pull-request runs (957 and 371) less the
825+
62 rows by which the Debug key sits below the Debug registry in the same runs
826+
(938 less 876), and the lab job's `ctest -N` confirms them or moves them. The
827+
run floors in the table above (910 and 337 with the overlay present, 603 and
828+
306 absent) are not read on an excluded run; push, dispatch and every other
829+
full run still enforce them.
830+
831+
A regression in an excluded row, or in a SLOW class that a pull request did not
832+
trigger, is first seen by the push-to-main run for the merge commit, and by the
833+
maintainers' full verification where it runs the row; the pull request does not
834+
see it. That delay is not zero. In the one hosted push run sampled
835+
(`39126947405`) the Release job took 1876 s, `kernel-only` 3022 s, Debug 3416
836+
s, `sanitizers` 2707 s and the native cell 3132 s; the older
837+
`pf-linux-x64-16` runners' medians over five runs were 1815, 1812, 1770, 2050
838+
and 1838 s, and a push run on the current pool has not been measured. The
839+
pull-request wall time under D1 is to be measured on real PR runs, typical and
840+
tail; it is not claimed as met against the owner's 15-minute target.
841+
775842
### Documentation-only changes
776843

777844
A pull request or a push to `main` whose change is documentation only runs
@@ -895,6 +962,12 @@ target needs a measured sanitizer build improvement; the current job logs do
895962
not separate compile from link time or report a ccache hit rate, so CI-LITE
896963
does not assume a build optimization without evidence.
897964

965+
That model predates decision D1. It keeps Release and kernel at their full
966+
durations on a pull request, which they no longer have there: both now exclude
967+
the `slow` rows ([PR tier versus MAIN/RELEASE](#pr-tier-versus-mainrelease-what-is-excluded-where)).
968+
The pull-request wall time under that policy is to be measured on real PR
969+
runs, and this page does not claim either target as met.
970+
898971
## Runners and time limits
899972

900973
The ci-spot pool is live as of 2026-10-18 (bulletin #613, L6, 22:34).
@@ -1056,9 +1129,9 @@ measurements on the new runners.
10561129
| Job | Limit (min) | Measured basis |
10571130
| --- | --- | --- |
10581131
| `preflight` | 360 (was 45) | 7.3-9.8 min on the standard runner through 1a0e7ea1; cancelled at 10.3 min at 0d76a099, 593 s into `verifier-tests`, which takes about 15 min there at that tree (above), so preflight about 16 min. 4.8-8.2 min on the verification hosts. Preflight supplies no stage deadline -- the old 2400 s stage bound is gone (owner rule, 2026-10-12); the native-live aggregate job names a stop at the platform's 360-minute maximum `HANG (re-run)` in its log; the check still fails. |
1059-
| `build` | 360 (was 75) | A Release leg runs its full population on every event, a fork's pull request included: 30.3 min Ubuntu and 32.0 min macOS on the standard runners at 0d76a099, where the full Debug legs took 43.5 and 37.0 min. Pull-request Debug sets 17.2-17.7 min. 11.1-18.4 min on the verification hosts. |
1132+
| `build` | 360 (was 75) | A Release leg runs its full population on push and dispatch events; a pull request's Release legs, a fork's included, exclude the `slow` rows like its Debug legs, and that pull-request time is to be measured on real PR runs. Measured while Release ran its full population on every event, a fork's pull request included: 30.3 min Ubuntu and 32.0 min macOS on the standard runners at 0d76a099, where the full Debug legs took 43.5 and 37.0 min. Pull-request Debug sets 17.2-17.7 min. 11.1-18.4 min on the verification hosts. |
10601133
| `sanitizers` | 360 (was 120; the verifier's CTest stage bound was 60 min full, 30 min PR) | A fork's pull-request set took 35.2 min on the standard runner, whose compile alone took 1277-1335 s (pull-request run 36139176292, and main at 0d76a099). Full runs on the standard runner took 34.0-69.4 min (at 0d76a099 30 min of build and ABI providers, then 39 min of CTest, 1768 s of it the serial `test_ci_verify` row), and 18.2-19.0 min on the verification hosts. |
1061-
| `kernel-only` | 360 (was 60) | Every event runs the whole kernel set: 7.9-22.3 min on the standard runner, 6.5-9.8 min on the verification hosts. The hosted kernel-only run of 2026-10-12 lost its job to the old 1800 s stage bound (exit 124). |
1134+
| `kernel-only` | 360 (was 60) | Push and dispatch events run the whole kernel set; a pull request's kernel-only leg excludes the `slow` rows, and that pull-request time is to be measured on real PR runs. Measured while every event ran the whole kernel set: 7.9-22.3 min on the standard runner, 6.5-9.8 min on the verification hosts. The hosted kernel-only run of 2026-10-12 lost its job to the old 1800 s stage bound (exit 124). |
10621135
| `native-live (native)` | 360 (was 75; verification step 65) | Main runs 37398893123, 37461653250 and 37478400505 spent 35.40–40.65 min in native, including 25.32–29.53 min in full CTest. Their matching PRs took 2.39–5.67 min in native CTest with the existing slow exclusion. The longest measured `test_ci_verify` row was 1695 s there and 2178 s on a c5.4xlarge Spot host, cut off by a trial deadline. The verifier supplies no default row or stage deadline and the workflow supplies no step deadline: its old 3600 s row bound is what collided with the 2820 s stage deadline on main's push run 38217844683. These are observed GitHub and Spot timings; final-head cold proof remains required. |
10631136
| `native-live (live-sanitizers)` | 360 (was 45; verification step 35) | The same three main runs spent 15.10–16.33 min in this profile after native. Its old per-row 900 s and stage 1200 s deadlines are gone; the verifier supplies no replacement, and the platform's job maximum catches a hang. |
10641137
| `native-live (live-tsan)` | 360 (was 45; verification step 35) | The old shared 60-min job left only 9.20, 4.67 and 2.67 min for TSan before cancellation. Matching passing PR profiles took 10.87, 10.93 and 7.70 min. Its old 900/1200-s row/stage bounds are gone like live-sanitizers'; the verifier supplies no replacement. |

0 commit comments

Comments
 (0)