Skip to content

Commit 8154993

Browse files
luisleo526ci-cxclaude
authored
ci: move post-merge verification to GCP and forbid paid runners (#555)
Route the amd64 post shards (Release, Debug, kernel, sanitizers), the native shards and both live sanitizer profiles to ["self-hosted","pf-gcp-post"]. Shard counts, matrices, verification commands, aggregates and check names are unchanged. The workflow checker now refuses every numeric pf-linux-x64 and pf-linux-arm64 label and every large/larger/xlarge suffix at every job, with no per-job exception, and the paid label declarations leave the actionlint config. docs/ci.md is rewritten line for line and stays at 2014 lines. These sentences, introduced by 3ea5a0d (the routing and capacity claims), are replaced: - "Post Release/Debug/kernel and native shards use `pf-linux-x64-16`; post sanitizers use `pf-linux-x64-32`; native-live sanitizers use `pf-linux-x64-64`." - "One full post run needs four x64-16 and two x64-64 native runners concurrently." - "The amd64 post build shards request hosted `pf-linux-x64-16`; the two arm64 post cells run on hosted `ubuntu-24.04-arm`." - "Hosted exceptions include the named light jobs, amd64 post shards, native profiles, arm64 and the existing release/promotion pins." - "`native-live.yml` runs four `native` shards on hosted `pf-linux-x64-16`; `live-sanitizers` and `live-tsan` each build once on hosted `pf-linux-x64-64`." - "The four amd64 post legs and native verification use hosted runners; their aggregates use the light image." - The `ci-post.yml` and `native-live.yml` rows of the workflow table that name the pf-linux labels. - "Hosted shards also run on a branch dispatch; the remaining post fleet still requires `main`." - "Release uses four `build` shards, Debug two `build-debug` shards, and kernel two shards on 16 cores; sanitizers uses four on 32 cores." - "`HOSTED_JOB_RUNNERS` pins hosted jobs; its larger/arm64 exceptions exclude PR callers." - "Larger labels are allowed only at their exact named jobs' `runs-on` keys; `larger_runner` keys stay refused." The historical timing sentence introduced by 0212cab keeps every measured value and only names the former paid pool. The rollback sentence introduced by b0f4341 no longer tells a reader to restore a paid label. Co-authored-by: ci-cx <ci-cx@invalid> Co-authored-by: Claude Sonnet 5.5 <[email protected]>
1 parent f506849 commit 8154993

7 files changed

Lines changed: 237 additions & 105 deletions

File tree

‎.github/actionlint.yaml‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,3 @@ self-hosted-runner:
99
- pf-gcp-main
1010
- pf-ci-x64
1111
- pf-ci-x64-16
12-
- pf-linux-x64-16
13-
- pf-linux-x64-32
14-
- pf-linux-x64-64

‎.github/workflows/ci-post.yml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ env:
3333

3434
# The canon authz gate admits our events before any checkout.
3535
# Light jobs and per-leg aggregates use hosted ubuntu-24.04; amd64 CTest shards use
36-
# pf-linux-x64-16 or pf-linux-x64-32, and arm64 keeps ubuntu-24.04-arm.
36+
# the post-merge GCP fleet, and arm64 keeps ubuntu-24.04-arm.
3737
# Preflight remains on the post-merge fleet. scripts/ci_preflight.py pins routing.
3838
# Each verification sizes its parallelism to the runner that took it.
3939
# scripts/ci_preflight.py pins every runner and shard count. GitHub's platform
@@ -257,8 +257,8 @@ jobs:
257257
build_type: [Release]
258258
shard: [1, 2, 3, 4]
259259

260-
# Four 16-core shards divide roughly 15,300 measured CTest row-seconds.
261-
runs-on: pf-linux-x64-16
260+
# Four shards divide roughly 15,300 measured CTest row-seconds.
261+
runs-on: ["self-hosted","pf-gcp-post"]
262262

263263
steps:
264264
- name: Refuse an unauthorised run before anything else
@@ -359,7 +359,7 @@ jobs:
359359
- os: ubuntu-24.04
360360
build_type: [Debug]
361361
shard: [1, 2]
362-
runs-on: pf-linux-x64-16
362+
runs-on: ["self-hosted","pf-gcp-post"]
363363

364364
steps:
365365
- name: Refuse an unauthorised run before anything else
@@ -589,7 +589,7 @@ jobs:
589589
fail-fast: false
590590
matrix:
591591
shard: [1, 2, 3, 4]
592-
runs-on: pf-linux-x64-32
592+
runs-on: ["self-hosted","pf-gcp-post"]
593593
env:
594594
ASAN_OPTIONS: detect_leaks=1:halt_on_error=1:abort_on_error=1
595595
UBSAN_OPTIONS: print_stacktrace=1:halt_on_error=1
@@ -691,7 +691,7 @@ jobs:
691691
fail-fast: false
692692
matrix:
693693
shard: [1, 2]
694-
runs-on: pf-linux-x64-16
694+
runs-on: ["self-hosted","pf-gcp-post"]
695695
steps:
696696
- name: Refuse an unauthorised run before anything else
697697
if: ${{ needs.authz.result != 'success' }}

‎.github/workflows/native-live.yml‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ permissions:
2323

2424
# A reusable workflow's github.workflow is its caller, so use a distinct prefix.
2525
# Only post-merge callers and main/manual runs use this workflow.
26+
# Native shards and both sanitizer profiles use the post-merge GCP fleet.
2627
concurrency:
2728
group: native-live-${{ github.workflow }}-${{ github.run_id }}
2829
cancel-in-progress: false
@@ -101,8 +102,8 @@ jobs:
101102
native-live:
102103
name: native-live (${{ matrix.profile }})
103104
needs: [authz]
104-
# Each instrumented tree builds once on a dedicated 64-core hosted runner.
105-
runs-on: pf-linux-x64-64
105+
# Each instrumented tree builds once on the post-merge GCP fleet.
106+
runs-on: ["self-hosted","pf-gcp-post"]
106107
strategy:
107108
fail-fast: false
108109
matrix:
@@ -199,7 +200,7 @@ jobs:
199200
native-live-shards:
200201
name: native-live-shard (${{ matrix.profile }}, ${{ matrix.shard }}/4)
201202
needs: [authz]
202-
runs-on: pf-linux-x64-16
203+
runs-on: ["self-hosted","pf-gcp-post"]
203204
strategy:
204205
fail-fast: false
205206
matrix:

‎docs/ci.md‎

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -69,11 +69,11 @@ successful runs on `main` save that cache namespace.
6969

7070
Verification jobs keep literal runner pins at the workflow boundary: PR build/preflight and parity subset use `["self-hosted","pf-gcp-pr"]`;
7171
post preflight/parity and credentialed docs use `["self-hosted","pf-gcp-post"]`.
72-
Post Release/Debug/kernel and native shards use `pf-linux-x64-16`; post sanitizers
73-
use `pf-linux-x64-32`; native-live sanitizers use `pf-linux-x64-64`. No job routes by event.
72+
Post Release/Debug/kernel and native shards, post sanitizers and both native-live sanitizer
73+
profiles use `["self-hosted","pf-gcp-post"]`. No job routes by event.
7474
Hosted `ubuntu-24.04` runs all eight `authz` jobs, both CI `changes`/`build-gate` jobs, all post/native aggregates and `docs.yml`'s `build`.
7575
These light, secret-free jobs need no VM-only tools, mounts or caches.
76-
Local reusable calls still cannot mix PR and post-merge verification fleets. One full post run needs four x64-16 and two x64-64 native runners concurrently.
76+
Local reusable calls still cannot mix PR and post-merge verification fleets. A full post run schedules 22 post-fleet jobs; the parity calls share one concurrency group.
7777

7878
The manual `release-ruleset-probe.yml` workflow runs `authz` on
7979
hosted `ubuntu-24.04` and `probe` on `["self-hosted","pf-gcp-post"]`.
@@ -183,9 +183,9 @@ source-only header trees, and compile the native public roots and examples;
183183
the preflight source guard rejects those includes before a build.
184184

185185
CI uses Ubuntu 24.04 amd64 and arm64 cells and selects Python 3.12 explicitly.
186-
The amd64 post build shards request hosted `pf-linux-x64-16`; the two arm64 post cells run on
186+
The amd64 post build shards use the post-merge GCP fleet; the two arm64 post cells run on
187187
hosted `ubuntu-24.04-arm`. All three build jobs refuse an OS mismatch.
188-
Hosted exceptions include the named light jobs, amd64 post shards, native profiles, arm64 and the existing release/promotion pins.
188+
Hosted exceptions include the named light jobs, arm64 and the existing release/promotion pins.
189189
CMake uses the same interpreter as the
190190
verification driver, so its Python checks do not switch to a different system
191191
Python. CTest must discover tests; an empty suite is a failure. Which runner
@@ -617,8 +617,8 @@ requires the WebSocket test.
617617
The runner's curl version floor still applies at configure time.
618618
`live-tsan` runs the same inventory in a separate ThreadSanitizer build,
619619
checks every runner compile unit for instrumentation, and stops on a race.
620-
`native-live.yml` runs four `native` shards on hosted `pf-linux-x64-16`;
621-
`live-sanitizers` and `live-tsan` each build once on hosted `pf-linux-x64-64`.
620+
`native-live.yml` runs four `native` shards on the post-merge GCP fleet;
621+
`live-sanitizers` and `live-tsan` each build once on that same fleet.
622622
Both matrices keep `fail-fast: false`, pinned WebSocket curl and profile caches. A failed cell does not cancel its peers.
623623
Native compiler keys and diagnostic artifacts also carry the shard index.
624624
The `native-live (native)` aggregate requires all four successful summary proofs;
@@ -1482,7 +1482,7 @@ maintainers' full verification where it runs the row; the pull request does not
14821482
see it. That delay is not zero. In the one hosted push run sampled
14831483
(`39126947405`) the Release job took 1876 s, `kernel-only` 3022 s, Debug 3416
14841484
s, `sanitizers` 2707 s and the native cell 3132 s; the older
1485-
`pf-linux-x64-16` runners' medians over five runs were 1815, 1812, 1770, 2050
1485+
paid 16-core runners' medians over five runs were 1815, 1812, 1770, 2050
14861486
and 1838 s, and a push run on the current pool has not been measured. The
14871487
pull-request wall time under D1 is to be measured on real PR runs, typical and
14881488
tail; it is not claimed as met against the owner's 15-minute target.
@@ -1648,17 +1648,17 @@ runs, and this page does not claim either target as met.
16481648
## Runners and time limits
16491649

16501650
Credentialed work and the remaining full profiles retain their VM pins.
1651-
The four amd64 post legs and native verification use hosted runners; their aggregates use the light image.
1651+
The four amd64 post legs and native verification use the post-merge GCP fleet; their aggregates use the light image.
16521652
The native matrix is exactly `1..4` with `--ctest-shard I/4`; live sanitizers are unsharded. Each runner label remains a literal, checked per job:
16531653

16541654
| Workflow | Events/callers | Inline jobs' `runs-on` |
16551655
| --- | --- | --- |
16561656
| `ci.yml` | Unfiltered `pull_request` | `authz`, `changes`, `build-gate`: hosted `ubuntu-24.04`; preflight/build: `["self-hosted","pf-gcp-pr"]` |
16571657
| `corpus-parity-subset.yml` | Reusable call from PR CI | `authz`: hosted `ubuntu-24.04`; subset: `["self-hosted","pf-gcp-pr"]` |
16581658
| `docs.yml` | `pull_request`, except `.github/**`-only diffs | `authz` and `build`: hosted `ubuntu-24.04` |
1659-
| `ci-post.yml` | Push to `main`, 01:30 UTC schedule, dispatch | Light jobs and four leg aggregates: `ubuntu-24.04`; Release/Debug/kernel shards: `pf-linux-x64-16`; sanitizers shards: `pf-linux-x64-32`; `build-arm`: `ubuntu-24.04-arm`; preflight tiers: `["self-hosted","pf-gcp-post"]` |
1659+
| `ci-post.yml` | Push to `main`, 01:30 UTC schedule, dispatch | Light jobs and four leg aggregates: `ubuntu-24.04`; Release/Debug/kernel and sanitizer shards plus preflight tiers: `["self-hosted","pf-gcp-post"]`; `build-arm`: `ubuntu-24.04-arm` |
16601660
| `corpus-parity.yml` | Schedule, dispatch, post CI callers | `authz`: hosted `ubuntu-24.04`; both parity jobs: `["self-hosted","pf-gcp-post"]` |
1661-
| `native-live.yml` | Dispatch and post CI callers, including branch dispatch | `authz` and both aggregates: hosted `ubuntu-24.04`; four native shards: `pf-linux-x64-16`; `live-sanitizers` and `live-tsan`: `pf-linux-x64-64` |
1661+
| `native-live.yml` | Dispatch and post CI callers on `main` | `authz` and both aggregates: hosted `ubuntu-24.04`; four native shards, `live-sanitizers` and `live-tsan`: `["self-hosted","pf-gcp-post"]` |
16621662
| `docs-post.yml` | Push to `main`, dispatch | `authz`: hosted `ubuntu-24.04`; credentialed `build`: `["self-hosted","pf-gcp-post"]` |
16631663

16641664
The post fleet admits only push, schedule and dispatch on engine `main` and
@@ -1667,18 +1667,18 @@ refuses all PR events. Reusable workflows inherit their caller's event.
16671667
follow local calls to a fixed point, refuse mixed event classes and require
16681668
literal VM arrays outside the exact hosted pins; job conditions cannot excuse routing.
16691669
Fork PRs on hosted runners run without our secrets, so the VM-only fork-refusal hook is not needed.
1670-
Hosted shards also run on a branch dispatch; the remaining post fleet still requires `main`.
1671-
Rollback: relabel the affected job to its prior `pf-gcp-pr` or `pf-gcp-post` array and restore its checker/test pin.
1670+
Post-fleet shards require `main` for dispatch; hosted arm64 keeps its fixed pin.
1671+
Rollback may restore an earlier GCP fleet pin; paid larger-runner labels remain refused.
16721672

16731673
The four full build checks are `build (ubuntu-24.04, Release)`,
16741674
`build (ubuntu-24.04, Debug)`, `build (ubuntu-24.04-arm, Release)` and
16751675
`build (ubuntu-24.04-arm, Debug)`. Release uses four `build` shards, Debug two
1676-
`build-debug` shards, and kernel two shards on 16 cores; sanitizers uses four on 32 cores.
1676+
`build-debug` shards, and kernel two shards; sanitizers uses four, all on the post-merge GCP fleet.
16771677
Every shard builds the full tree and runs `--ctest-shard I/N` at the runner's core count. Each build refuses an OS mismatch before checkout.
16781678
Each leg aggregate retains its old check name and requires its own matrix's success. The two arm64 cells share `build-arm` on hosted `ubuntu-24.04-arm`. All three build jobs use the same verification steps and profiles.
16791679
Shard jobs and leg aggregates skip together only on classified docs-only changes.
16801680
One cache key per architecture/profile/SHA is restored by all shards; only shard 1 saves it.
1681-
`HOSTED_JOB_RUNNERS` pins hosted jobs; its larger/arm64 exceptions exclude PR callers. The PR build remains
1681+
`HOSTED_JOB_RUNNERS` pins hosted jobs and post-fleet verification arrays; arm64 exceptions exclude PR callers. The PR build remains
16821682
`build (ubuntu-24.04, Release)` and its aggregate remains `build`.
16831683

16841684
Release and promotion use the isolated release fleet:
@@ -1729,7 +1729,7 @@ merge time, and supply readable identity fields; failures there refuse red.
17291729
Legacy `PF_CI_LABELS` metadata does not select runners. Verification and
17301730
parity commands use `$(getconf _NPROCESSORS_ONLN)` for parallelism.
17311731
No job or step declares `timeout-minutes`, and no proof continues on error.
1732-
Larger labels are allowed only at their exact named jobs' `runs-on` keys;
1732+
Every paid larger-runner label is refused in every workflow, including pinned jobs;
17331733
`larger_runner` keys stay refused. GitHub's platform maximum remains the only time bound.
17341734

17351735
The read-only `promote-guard` stays hosted on `ubuntu-latest`; the credential

‎scripts/ci_preflight.py‎

Lines changed: 20 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -263,18 +263,18 @@ def post_build_name(count: int) -> str:
263263
'docs.yml': {'authz': None, 'build': None},
264264
'docs-post.yml': {'authz': None, 'build': None},
265265
}
266-
# Exact hosted pins; larger labels are allowed only at these jobs' runs-on keys.
266+
# Exact hosted pins and post-fleet verification arrays, checked per job.
267267
HOSTED_JOB_RUNNERS = {
268268
'ci.yml': dict.fromkeys(('authz', 'changes', 'build-gate'), 'ubuntu-24.04'),
269269
'ci-post.yml': {**dict.fromkeys(('authz', 'changes', 'build-gate'), 'ubuntu-24.04'),
270270
**dict.fromkeys(POST_LEG_GATES, 'ubuntu-24.04'),
271-
'build': 'pf-linux-x64-16', 'build-debug': 'pf-linux-x64-16',
272-
'sanitizers': 'pf-linux-x64-32', 'kernel-only': 'pf-linux-x64-16',
271+
'build': POST_RUNNER, 'build-debug': POST_RUNNER,
272+
'sanitizers': POST_RUNNER, 'kernel-only': POST_RUNNER,
273273
'build-arm': 'ubuntu-24.04-arm'},
274274
**{name: {'authz': 'ubuntu-24.04'} for name in (
275275
'corpus-parity.yml', 'corpus-parity-subset.yml', 'docs-post.yml')},
276-
'native-live.yml': {'authz': 'ubuntu-24.04', 'native-live': 'pf-linux-x64-64',
277-
'native-live-shards': 'pf-linux-x64-16',
276+
'native-live.yml': {'authz': 'ubuntu-24.04', 'native-live': POST_RUNNER,
277+
'native-live-shards': POST_RUNNER,
278278
'native-live-native': 'ubuntu-24.04', 'native-live-gate': 'ubuntu-24.04'},
279279
'docs.yml': {'authz': 'ubuntu-24.04', 'build': 'ubuntu-24.04'},
280280
'promote-baseline.yml': {'promote-guard': 'ubuntu-latest'},
@@ -289,10 +289,9 @@ def post_build_name(count: int) -> str:
289289
# Credential-bearing jobs outside the release pins and gated docs build retain
290290
# standard hosted labels. Existing hosted exceptions above are pinned separately.
291291
STANDARD_RUNNERS = ('ubuntu-24.04',)
292-
# Larger labels are allowed only as the exact runs-on of a pinned hosted job.
293-
# Refuse every other occurrence, including metadata and the release matrix.
292+
# Refuse paid labels everywhere, including pinned jobs, metadata and release matrices.
294293
_LARGER_RUNNER_LABEL = re.compile(
295-
r'(?<![\w.-])(?:pf-linux-x64-(?:16|32|64)|[\w.-]+-(?:xlarge|large))(?![\w.-])')
294+
r'(?<![\w.-])(?:pf-linux-(?:x64|arm64)-\d+|[\w.-]+-(?:larger|xlarge|large))(?![\w.-])')
296295
_LARGER_RUNNER_KEY = re.compile(r'(?<![\w.-])["\']?larger_runner["\']?[ \t]*:')
297296
# Retired release labels remain refused by the runner pins as well. Neither
298297
# spelling may grant an arbitrary workflow access to the isolated release pool.
@@ -1366,7 +1365,7 @@ def workflow_fleet_findings(workflows: dict[str, str]) -> list[str]:
13661365
if not reachable:
13671366
findings.append(f'{name} must have a reachable event for fleet routing')
13681367
for job, body in _jobs(workflow).items():
1369-
# runner_findings pins this exact job's hosted label and metadata.
1368+
# runner_findings pins this exact job's runner and metadata.
13701369
# A PR caller disables the arm64/promotion exceptions through calls.
13711370
if job in fixed_hosted:
13721371
continue
@@ -1395,23 +1394,6 @@ def _authz_header(workflow: str) -> str:
13951394
return AUTHZ_JOB_HEADER
13961395

13971396

1398-
def _unapproved_runner_code(workflow: str, fixed_hosted: dict[str, str]) -> str:
1399-
"""Exempt only the exact job-level runner scalar, never its whole job."""
1400-
checked = []
1401-
in_jobs, job = False, None
1402-
for line in _code(workflow):
1403-
if _indent(line) == 0:
1404-
in_jobs, job = _key_value(line, 'jobs') == '', None
1405-
elif in_jobs and _indent(line) == 2:
1406-
header = _JOB_HEADER.fullmatch(line)
1407-
job = header[1] if header else None
1408-
if (in_jobs and job in fixed_hosted and _indent(line) == 4
1409-
and _job_values(line, 'runs-on') == [fixed_hosted[job]]):
1410-
continue
1411-
checked.append(re.sub(r'\s+#.*$', '', line))
1412-
return '\n'.join(checked)
1413-
1414-
14151397
def runner_findings(workflows: dict[str, str]) -> list[str]:
14161398
"""Enforce event-derived fleets, isolated releases and command/core pins."""
14171399
findings = workflow_fleet_findings(workflows)
@@ -1423,7 +1405,7 @@ def runner_findings(workflows: dict[str, str]) -> list[str]:
14231405
timed = JOB_COMMANDS.get(name, {})
14241406
fixed_hosted = {job: label for job, label in HOSTED_JOB_RUNNERS.get(name, {}).items()
14251407
if label == 'ubuntu-24.04' or (routes[name] & POST_EVENTS and 'pull_request' not in routes[name])}
1426-
if _LARGER_RUNNER_LABEL.search(_unapproved_runner_code(workflow, fixed_hosted)):
1408+
if _LARGER_RUNNER_LABEL.search(code):
14271409
findings.append(f'{name} must not use larger-runner labels; use the GCP fleet or hosted exceptions')
14281410
fixed_release = RELEASE_JOB_RUNNERS.get(name, {})
14291411
event_runner = POST_RUNNER if routes[name] & POST_EVENTS else PR_RUNNER
@@ -1492,6 +1474,17 @@ def runner_findings(workflows: dict[str, str]) -> list[str]:
14921474
if any(re.search(r'\bPF_CI_LABELS["\']?\s*:', value) for value in environments):
14931475
findings.append(f'{name} job {job} must not set PF_CI_LABELS '
14941476
'on an isolated release job')
1477+
elif fixed_hosted.get(job) == POST_RUNNER:
1478+
actual = _job_values(body, 'runs-on', unquote=False)
1479+
if [_runner_value(value) for value in actual] != [_runner_value(POST_RUNNER)]:
1480+
findings.append(f'{name} job {job} must run on {POST_RUNNER}')
1481+
if credentials:
1482+
findings.append(f'{name} job {job} must keep production credentials '
1483+
'on a hosted standard runner')
1484+
environments = _job_values(workflow, 'env') + _job_values(body, 'env')
1485+
if any(re.search(r'\bPF_CI_LABELS["\']?\s*:', value) for value in environments):
1486+
findings.append(f'{name} job {job} must not set PF_CI_LABELS '
1487+
'on a post fleet job')
14951488
elif (job in fixed_hosted
14961489
or (credentials and (name, job) != ('docs-post.yml', 'build'))
14971490
or (job not in pinned and hosted_platform)):

0 commit comments

Comments
 (0)