Skip to content

Commit 5ce34fc

Browse files
luisleo526claude
andcommitted
ci(release): the probe's hidden-bypass_actors refusal states what was observed, not a permission; dispatch from main (CR-245 F1, round 3)
REV (Opus + sol, static) on the probe: NO-GO, small. P1: the refusal text printed exactly when a ruleset's bypass_actors is hidden, which is the probe's decisive output, ended "(App requires administration: read)". That is the permission claim round 2 removed from the docs, the release.yml comment and the binding test; it was introduced by a8b1276 ("the bump exemption also requires that only the release App can push to main") as a fifth place and it is unverified (GitHub's documentation says bypass_actors is returned only to a caller with write access to the ruleset). The text is now "(token cannot see this ruleset's bypass_actors)", the two assertions that pinned the old text pin the new exact text, and the absence guard now scans the WHOLE source of ci_release_gate.py (strings and code, not only comment lines) for the forbidden phrases in every spelling, with restoration controls that reinsert the old refusal text and seven spelling variants. F4: docs/ci.md says the one-time dispatch must run from main with main's copy of the workflow, lists the probe on the isolated pf-gcp-main fleet, and scopes the FLEET-POST=A routing sentence (authz on pf-gcp-pr, the probe job on pf-gcp-main). F2: comments in ci_preflight.py, its tests and release-ruleset-probe.yml claimed a pin binds the authz header byte for byte; for this workflow the pin binds the normalised executable lines (comments and blank lines removed), and the comments now say so. F3: the display branch for a hidden ruleset is unreachable (the reader refuses first); kept, and docs/ci.md says two hidden rulesets need two dispatches because the first refusal names only the first. Rewrites that replace earlier sentences (named for the doc-reverts check): docs/ci.md "FLEET-POST=A splits routing at the file boundary: PR jobs use" (introduced by 3a0d150); docs/ci.md "Release and promotion use the isolated release fleet: `release`, `notify-hub`, Linux x64" (opening from 833f7ac, second line from 3a0d150); docs/ci.md "Dispatch **Release ruleset probe** (`release-ruleset-probe.yml`) once to check" (6adae66); ci_preflight.py "The canon header is text, not a decoded array" (a8089b2) and "Canon authz keeps its space after the comma" (1d5dfd6); the "GCP fleet (owner #1050): non-release jobs on pf-gcp-pr; release-class jobs only on pf-gcp-main" copies in the probe workflow, ci_preflight.py and test_ci_preflight.py (1d5dfd6). Not changed here: the tag-pinned actions (repo-wide card) and the dispatch exposure of the App secrets (separate L6 card). Lane: ci-fu-cr486r (Codex gpt-6.1-sol). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent d3463f0 commit 5ce34fc

6 files changed

Lines changed: 145 additions & 22 deletions

File tree

‎.github/workflows/release-ruleset-probe.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,8 @@ permissions:
1111
jobs:
1212
authz:
1313
name: authz
14-
# GCP fleet (owner #1050): non-release jobs on pf-gcp-pr; release-class jobs only on pf-gcp-main.
14+
# Preflight pins executable workflow lines after removing comment-only lines.
15+
# authz uses pf-gcp-pr; the App-token probe uses pf-gcp-main.
1516
runs-on: ["self-hosted", "pf-gcp-pr"]
1617
permissions: {}
1718
# No timeout-minutes: the owner's no-time-limits rule (check-no-time-limits refuses new ones).

‎docs/ci.md‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -60,14 +60,17 @@ or `slow` rows. The target rebuilds those executables with their library.
6060
PRs restore the newest compatible default-branch compiler cache, and only
6161
successful runs on `main` save that cache namespace.
6262

63-
FLEET-POST=A splits routing at the file boundary: PR jobs use
64-
`["self-hosted","pf-gcp-pr"]`, and push/schedule/manual jobs use
65-
`["self-hosted","pf-gcp-post"]`. No job chooses its label by event.
63+
FLEET-POST=A splits verification and documentation routing at the file boundary:
64+
PR jobs use `["self-hosted","pf-gcp-pr"]`, and post-merge push/schedule/manual
65+
jobs use `["self-hosted","pf-gcp-post"]`. No job chooses its label by event.
6666
`corpus-parity-subset.yml` is reusable only from the PR pipeline;
6767
`corpus-parity.yml` keeps full parity and the full-event subset on the post
6868
fleet. The checker follows local reusable calls, so a PR caller cannot borrow
6969
a post workflow even behind a job condition.
7070

71+
The manual `release-ruleset-probe.yml` workflow runs `authz` on
72+
`["self-hosted","pf-gcp-pr"]` and `probe` on `["self-hosted","pf-gcp-main"]`.
73+
7174
`docs.yml` retains the PR build name and `.github/**` exclusion.
7275
`docs-post.yml` builds and deploys on main pushes and manual dispatch.
7376
Tag pushes no longer trigger docs: the post fleet admits only engine `main`.
@@ -1147,8 +1150,9 @@ pins that one job name and image, and the exception never applies to a
11471150
workflow reachable from `pull_request`. The PR build remains
11481151
`build (ubuntu-24.04, Release)` and its aggregate remains `build`.
11491152

1150-
Release and promotion use the isolated release fleet: `release`, `notify-hub`, Linux x64
1151-
`prebuilt` and baseline `promote` use `["self-hosted","pf-gcp-main"]`.
1153+
Release, promotion and the release ruleset probe use the isolated release fleet:
1154+
`release`, `notify-hub`, Linux x64 `prebuilt`, baseline `promote` and
1155+
`release-ruleset-probe.yml`'s `probe` job use `["self-hosted","pf-gcp-main"]`.
11521156
Release's ARM/macOS artifact cells keep their existing hosted pins, and
11531157
promotion runs on a push to `main` plus manual dispatch from `main`.
11541158

@@ -1369,13 +1373,18 @@ The workflow-level `contents: write` setting scopes GITHUB_TOKEN.
13691373
Preflight pins the unconditional gate invocation and its App token, and its
13701374
binding release-gate tests pin this visibility requirement in code and docs.
13711375

1372-
Dispatch **Release ruleset probe** (`release-ruleset-probe.yml`) once to check
1373-
actual ruleset visibility with the same App token mint as `release.yml`.
1376+
Dispatch **Release ruleset probe** (`release-ruleset-probe.yml`) once from
1377+
`main` to check actual ruleset visibility with the same App token mint as
1378+
`release.yml`. The workflow file must be on the default branch, and the run
1379+
must use `main`'s copy.
13741380
It runs `python3 scripts/ci_release_gate.py rulesets-probe --repo OWNER/REPO`,
13751381
reading only repository metadata, the parent-inclusive ruleset inventory and
13761382
branch details. Success prints the barrier id/name and one compact inventory
13771383
line with enforcement, source and bypass visibility. Refusal exits 1 with empty
13781384
stdout and the reader's `::error::` diagnostic; API/response failures name the endpoint.
1385+
Two hidden rulesets need two dispatches: the first refusal names only the
1386+
first hidden ruleset; make its `bypass_actors` visible before dispatching again
1387+
to identify the second.
13791388
The probe changes no release evidence or gate verdicts. A release `dry_run`
13801389
on a merged PR does not exercise the bump-only ruleset reader.
13811390

‎scripts/ci_preflight.py‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -271,8 +271,8 @@
271271
# PR-only ALLOWED_BOTS is dependabot[bot] and pineforge-app[bot]. No local canon
272272
# file is needed on CI runners. Changing the script requires a new canon pin.
273273
AUTHZ_STEP_SHA256 = '94bf3afefe4588bc33dad5fd4d8fa3d2a104cc45743ad59286c756fa3a17baa6'
274-
# The canon header is text, not a decoded array: the fleet comment and the
275-
# space after the comma bind every authz job byte for byte.
274+
# Scoped CI workflows bind this canon header byte for byte, including its
275+
# fleet comment and comma spacing. The release probe pins executable lines.
276276
AUTHZ_JOB_HEADER = ''' name: authz
277277
# Fleet routing is fixed for this workflow; releases use pf-gcp-main.
278278
runs-on: ["self-hosted", "pf-gcp-pr"]
@@ -1003,8 +1003,8 @@ def runner_findings(workflows: dict[str, str]) -> list[str]:
10031003
actual = _job_values(body, 'runs-on', unquote=False)
10041004
expected = [fixed_release[job]]
10051005
if fixed_release[job] == PR_RUNNER:
1006-
# Canon authz keeps its space after the comma; the complete
1007-
# probe workflow pin below binds that exact header.
1006+
# Normalize the runner array here; the probe workflow pin below
1007+
# compares nonblank lines after removing comment-only lines.
10081008
actual = [_runner_value(value) for value in actual]
10091009
expected = [_runner_value(value) for value in expected]
10101010
if actual != expected:
@@ -1185,7 +1185,8 @@ def _step_block(step: str, key: str) -> list[str]:
11851185
jobs:
11861186
authz:
11871187
name: authz
1188-
# GCP fleet (owner #1050): non-release jobs on pf-gcp-pr; release-class jobs only on pf-gcp-main.
1188+
# Preflight pins executable workflow lines after removing comment-only lines.
1189+
# authz uses pf-gcp-pr; the App-token probe uses pf-gcp-main.
11891190
runs-on: ["self-hosted", "pf-gcp-pr"]
11901191
permissions: {}
11911192
# No timeout-minutes: the owner's no-time-limits rule (check-no-time-limits refuses new ones).

‎scripts/ci_release_gate.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -328,7 +328,7 @@ def _ruleset_bypasses(payload: dict, endpoint: str) -> set[tuple[str, int | None
328328
actors = payload.get("bypass_actors")
329329
if not isinstance(actors, list):
330330
raise ReleaseGateError(f"API {endpoint} returned missing or invalid bypass_actors "
331-
"(App requires administration: read)")
331+
"(token cannot see this ruleset's bypass_actors)")
332332
bypasses = set()
333333
for actor in actors:
334334
if not isinstance(actor, dict):

‎scripts/test_ci_preflight.py‎

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,8 @@
7777
jobs:
7878
authz:
7979
name: authz
80-
# GCP fleet (owner #1050): non-release jobs on pf-gcp-pr; release-class jobs only on pf-gcp-main.
80+
# Preflight pins executable workflow lines after removing comment-only lines.
81+
# authz uses pf-gcp-pr; the App-token probe uses pf-gcp-main.
8182
runs-on: ["self-hosted", "pf-gcp-pr"]
8283
permissions: {}
8384
# No timeout-minutes: the owner's no-time-limits rule (check-no-time-limits refuses new ones).
@@ -3607,6 +3608,37 @@ def test_release_ruleset_probe_workflow_is_installed_and_pinned(self):
36073608
self.assertEqual(probe_app, release_app, 'the mint steps must stay byte-identical')
36083609
self.assertEqual(ci_workflow_findings(*self.ci_sources(), others=self.other_sources()), [])
36093610

3611+
def test_release_ruleset_probe_comments_describe_the_actual_pin_scope(self):
3612+
source = (ROOT / 'scripts/ci_preflight.py').read_text()
3613+
workflow = (ROOT / '.github/workflows/release-ruleset-probe.yml').read_text()
3614+
header = (
3615+
' # Preflight pins executable workflow lines after removing comment-only lines.\n'
3616+
' # authz uses pf-gcp-pr; the App-token probe uses pf-gcp-main.\n')
3617+
requirements = (
3618+
(source,
3619+
'# Scoped CI workflows bind this canon header byte for byte, including its\n'
3620+
'# fleet comment and comma spacing. The release probe pins executable lines.\n'),
3621+
(source,
3622+
' # Normalize the runner array here; the probe workflow pin below\n'
3623+
' # compares nonblank lines after removing comment-only lines.\n'),
3624+
(source, header),
3625+
(RULESETS_PROBE_WORKFLOW, header),
3626+
(workflow, header),
3627+
)
3628+
message = 'comments must describe the probe pin without claiming its header is bound'
3629+
for text, required in requirements:
3630+
self.assertIn(required, text, message)
3631+
changed = text.replace(required, '', 1)
3632+
self.assertNotEqual(changed, text)
3633+
with self.assertRaisesRegex(AssertionError, message):
3634+
self.assertIn(required, changed, message)
3635+
3636+
changed = workflow.replace(header, ' # A different header comment.\n', 1)
3637+
self.assertNotEqual(changed, workflow)
3638+
self.assertEqual(ci_workflow_findings(
3639+
*self.ci_sources(), others={**self.other_sources(),
3640+
'release-ruleset-probe.yml': changed}), [])
3641+
36103642
def test_release_ruleset_probe_refuses_the_original_unauthorized_workflow(self):
36113643
# The concrete main input has neither authz nor needs/step guards.
36123644
# Main's whole-workflow pin accepts it, so this is red for that omission.

‎scripts/test_ci_release_gate.py‎

Lines changed: 87 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1635,11 +1635,73 @@ def test_hidden_org_bypass_actors_refuses_with_the_reader_diagnostic(self):
16351635
api = self.api_for(rows)
16361636
self.assertEqual(self.run_probe(api), (
16371637
1, "", f"::error::API repos/{REPO}/rulesets/13094758 returned missing or invalid "
1638-
"bypass_actors (App requires administration: read)\n"))
1638+
"bypass_actors (token cannot see this ruleset's bypass_actors)\n"))
16391639
self.assertEqual(api.calls, [
16401640
(REPO_API, False, None), (RULESETS_API, True, None),
16411641
(f"repos/{REPO}/rulesets/13094758", False, None)])
16421642

1643+
def test_two_hidden_rulesets_require_two_dispatches_as_documented(self):
1644+
docs = (ROOT / "docs/ci.md").read_text()
1645+
limit = (
1646+
"Two hidden rulesets need two dispatches: the first refusal names only the\n"
1647+
"first hidden ruleset; make its `bypass_actors` visible before dispatching again\n"
1648+
"to identify the second.")
1649+
message = "docs must explain the first-hidden-ruleset refusal limit"
1650+
self.assertIn(limit, docs, message)
1651+
changed = docs.replace(limit, "", 1)
1652+
self.assertNotEqual(changed, docs)
1653+
with self.assertRaisesRegex(AssertionError, message):
1654+
self.assertIn(limit, changed, message)
1655+
1656+
rows = release_rulesets()
1657+
rows[0].update(id=13094758, name="default", source_type="Organization",
1658+
source="example")
1659+
rows[1].update(source_type="Repository", source=REPO)
1660+
visible_actors = rows[0].pop("bypass_actors")
1661+
del rows[1]["bypass_actors"]
1662+
first = self.api_for(rows)
1663+
self.assertEqual(self.run_probe(first), (
1664+
1, "", f"::error::API repos/{REPO}/rulesets/13094758 returned missing or invalid "
1665+
"bypass_actors (token cannot see this ruleset's bypass_actors)\n"))
1666+
self.assertEqual(first.calls, [
1667+
(REPO_API, False, None), (RULESETS_API, True, None),
1668+
(f"repos/{REPO}/rulesets/13094758", False, None)])
1669+
1670+
rows[0]["bypass_actors"] = visible_actors
1671+
second = self.api_for(rows)
1672+
self.assertEqual(self.run_probe(second), (
1673+
1, "", f"::error::API {STRICT_RULESET_API} returned missing or invalid "
1674+
"bypass_actors (token cannot see this ruleset's bypass_actors)\n"))
1675+
self.assertEqual(second.calls, [
1676+
(REPO_API, False, None), (RULESETS_API, True, None),
1677+
(f"repos/{REPO}/rulesets/13094758", False, None),
1678+
(STRICT_RULESET_API, False, None)])
1679+
1680+
def test_probe_dispatch_branch_and_fleets_are_documented(self):
1681+
docs = (ROOT / "docs/ci.md").read_text()
1682+
requirements = (
1683+
"FLEET-POST=A splits verification and documentation routing at the file boundary:\n"
1684+
'PR jobs use `["self-hosted","pf-gcp-pr"]`, and post-merge push/schedule/manual\n'
1685+
'jobs use `["self-hosted","pf-gcp-post"]`.',
1686+
"The manual `release-ruleset-probe.yml` workflow runs `authz` on\n"
1687+
'`["self-hosted","pf-gcp-pr"]` and `probe` on `["self-hosted","pf-gcp-main"]`.',
1688+
"Release, promotion and the release ruleset probe use the isolated release fleet:\n"
1689+
"`release`, `notify-hub`, Linux x64 `prebuilt`, baseline `promote` and\n"
1690+
'`release-ruleset-probe.yml`\'s `probe` job use `["self-hosted","pf-gcp-main"]`.',
1691+
"Dispatch **Release ruleset probe** (`release-ruleset-probe.yml`) once from\n"
1692+
"`main` to check actual ruleset visibility with the same App token mint as\n"
1693+
"`release.yml`. The workflow file must be on the default branch, and the run\n"
1694+
"must use `main`'s copy.",
1695+
)
1696+
message = "docs must state the probe's main dispatch and actual fleet routes"
1697+
for required in requirements:
1698+
with self.subTest(required=required):
1699+
self.assertIn(required, docs, message)
1700+
changed = docs.replace(required, "", 1)
1701+
self.assertNotEqual(changed, docs)
1702+
with self.assertRaisesRegex(AssertionError, message):
1703+
self.assertIn(required, changed, message)
1704+
16431705
def test_empty_inventory_refusal_and_documented_endpoint_scope(self):
16441706
# The reader report's complete [] inventory fails on main without an
16451707
# endpoint. The runtime assertions already hold there; the prose does not.
@@ -1757,7 +1819,7 @@ def test_full_source_mutant_skipping_reader_admits_the_hidden_org_fixture(self):
17571819
source="example")
17581820
del rows[0]["bypass_actors"]
17591821
refusal = (1, "", f"::error::API repos/{REPO}/rulesets/13094758 returned missing or "
1760-
"invalid bypass_actors (App requires administration: read)\n")
1822+
"invalid bypass_actors (token cannot see this ruleset's bypass_actors)\n")
17611823
self.assertEqual(self.run_probe(self.api_for(rows)), refusal)
17621824
source = Path(ci_release_gate.__file__).read_text()
17631825
before = " barrier, branches = _require_release_rulesets(api, repo)\n"
@@ -1913,11 +1975,10 @@ def test_ci_release_gate_is_unconditional_and_precedes_every_release_write(self)
19131975
self.assert_gate_pin(self.workflow())
19141976

19151977
def assert_ruleset_visibility_documentation(self, source, docs, workflow):
1916-
source_comments = "\n".join(line for line in source.splitlines()
1917-
if line.lstrip().startswith("#"))
1918-
for text in (source_comments, docs, workflow):
1919-
self.assertNotIn("administration: read", text,
1920-
"unverified App permission claim must stay absent")
1978+
for text in (source, docs, workflow):
1979+
self.assertNotRegex(text.replace("`", "").casefold(),
1980+
r"\badministration\s*:\s*read\b|\bapp\s+requires\b",
1981+
"unverified App permission claim must stay absent")
19211982
self.assertIn(
19221983
"# The gate needs bypass_actors visible on every branch ruleset in the inventory.\n"
19231984
"# If the release App token cannot see them, the bump is refused;\n"
@@ -1967,6 +2028,25 @@ def test_ruleset_visibility_prose_rejects_restored_permission_claims(self):
19672028
with self.assertRaisesRegex(AssertionError, message):
19682029
self.assert_ruleset_visibility_documentation(*changed)
19692030

2031+
def test_ruleset_visibility_guard_rejects_restored_refusal_and_spelling_variants(self):
2032+
original = [Path(ci_release_gate.__file__).read_text(),
2033+
(ROOT / "docs/ci.md").read_text(), self.workflow()]
2034+
self.assert_ruleset_visibility_documentation(*original)
2035+
neutral = "(token cannot see this ruleset's bypass_actors)"
2036+
self.assertEqual(original[0].count(neutral), 1,
2037+
"restoration must target the runtime refusal string")
2038+
for claim in ("App requires administration: read", "administration: read",
2039+
"`administration: read`", "administration:read",
2040+
"`administration:read`", "`administration`: `read`",
2041+
"ADMINISTRATION : READ", "App requires"):
2042+
with self.subTest(claim=claim):
2043+
changed = original.copy()
2044+
changed[0] = changed[0].replace(neutral, f"({claim})", 1)
2045+
self.assertNotEqual(changed[0], original[0])
2046+
with self.assertRaisesRegex(
2047+
AssertionError, "unverified App permission claim must stay absent"):
2048+
self.assert_ruleset_visibility_documentation(*changed)
2049+
19702050
def test_prebuilt_builds_the_emitted_sha_and_asserts_tag_identity(self):
19712051
jobs = _jobs(self.workflow())
19722052
self.assertRegex(jobs["release"],

0 commit comments

Comments
 (0)