Skip to content

release gate: require a green ci-post proof run for the released commit - #526

Merged
luisleo526 merged 16 commits into
mainfrom
p2-relgate
Oct 26, 2026
Merged

luisleo526 merged 16 commits into
mainfrom
p2-relgate

Conversation

@luisleo526

@luisleo526 luisleo526 commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

What this changes

The release gate (scripts/ci_release_gate.py) now also requires a successful, non-skipped ci-post run for the commit being released. Until now it read only the pineforge/verify and pineforge/parity statuses: the engine dry run 40260918204 printed release gate green: PR #518 head 9325b640...; verify=success; parity=success for a release of main 2561cfb whose own ci-post legs were never read. This closes that gap.

Rule

Proof commit: the release sha, or, when the release sha is docs-only, the nearest first-parent ancestor that is not docs-only (docs-only by the merge gate's own classifier ci_docs_only.is_documentation; at most 20 docs-only commits, then walk-too-long; unreadable history -> walk-blocked).

Every skipped docs-only commit (the release tip included) must have a newest ci-post run that exists and is completed with conclusion success (run-level conclusion only, no jobs read): else docs-commit-no-run / docs-commit-red.

For the proof commit, runs of that exact sha on main (any event) are examined newest first (created_at, then id), at most 10, one jobs read per examined completed run. Newer-run table:

Newer completed run on the proof sha Result
a build job failed / cancelled / timed_out (any row) ci-post: newer-run-failed (first, over any run conclusion)
run-level skipped, all four build legs skipped passes over (counted as a newer skipped-only run)
run-level skipped, any other leg shape not-success (contradictory data), naming the legs
any other non-success run conclusion (neutral, action_required, stale, startup_failure, null, ...) not-success, whatever the legs show
run-level success, all four legs present, each success or skipped passes over
run-level success, a leg neutral / null / unknown / missing not-success naming the leg (while an older run exists to protect)
not completed run-in-progress

The proof run itself needs run-level success AND all four build legs success (build (ubuntu-24.04|macos-26, Debug|Release), pinned to the ci-post.yml matrices by a workflow-pin test); a skipped leg there is no-full-run. Judgement call: a run-level success run with a bad/missing leg that is the LAST examined run (nothing older to protect, or the tenth examined run) ends in no-full-run instead of not-success: both are refusals, only the typed reason differs (pinned from both sides by test_single_run_with_a_missing_leg_gives_no_full_run and test_missing_leg_run_with_an_older_proof_refuses_not_success).

Read count

1 paginated runs read per skipped docs-only commit (at most 20, typically 0 or 1, no jobs reads), then 1 paginated runs read for the proof sha plus 1 jobs read per examined completed proof run (at most 10). A plain pass with a non-docs release sha: 1 + 1.

Scope notes

  • Runs on other branches are excluded (head_branch == main): they are not main evidence. Runs are matched by head_sha of the examined commit (filter at _ci_post_runs); a red or running run of another commit never counts, whatever its run time (rows: test_later_parent_failure_only_blocks_after_release_sha_flip and the three siblings flip only head_sha).
  • Phase 2 (receipt) is not live; the requirement applies to it from the day it goes live (same code path).
  • The release.yml gate step has no dry_run guard, so a dry_run=true dispatch executes it. The Actions API read scope of the release App token is unproven until that step runs: the engine dry run at the freeze commit is its first live proof.

Existing test lines changed (against main)

The diff against main removes lines in scripts/test_ci_release_gate.py in exactly six existing methods; everything else in the file is added, no method is deleted:

  • GateTests.assert_no_pr_fallback (takes the new ci_post keyword).
  • GateTests.run_gate (opt-in green ci-post fixture and the history= seam).
  • GateTests.test_green_reads_both_newest_statuses_and_names_the_reviewed_tree, GateTests.test_main_proof_with_full_contexts_green_passes_without_pr_reads, ReleaseBumpTests.assert_bump_green: the expected stdout gains ; ci-post run <id> for <sha>.
  • ReleaseRulesetTests.test_removing_each_new_condition_admits_its_unsafe_input: the full-module mutant gets history= and the suffix.

In scripts/ci_release_gate.py the removed lines are 13: the old module docstring sentence, the gate() signature (new history parameter), its return statements (the evidence now ends with the ci-post evidence), the phase-one / release-bump wrapper lines folded into the new flow, and the main(...) signature and call line (gains history). The public CLI (gate --repo --sha --ref) is unchanged; gate() and main() gain an optional injected history seam that defaults to git_history.

Evidence

Every test run below was on the project's cloud runner; none ran on a developer machine.

  • The release-gate test file and the rest of the preflight pass on the final tree.
  • The new tests fail against the first fail-closed reading of the table, on the rows that reading got wrong (over-blocking skipped runs, under-blocking a run-level success with a bad leg).
  • A mutant that treats a missing leg as skipped fails only the missing-leg rows.
  • One-line mutants of the decision block, each caught by the tests: in the success branch, never refusing for a bad leg fails 28 rows (all four cases of test_missing_leg_run_with_an_older_proof_refuses_not_success) and always refusing (dropping the older-run condition) fails 25 rows (all four cases of test_single_run_with_a_missing_leg_gives_no_full_run); never refusing in the run-level skipped branch fails 25 rows; never refusing in the branch for failed / cancelled / timed-out / other conclusions fails 47 rows. The success branch (both directions), the skipped branch and the other-conclusions branch are each mutation-checked.

🤖 Generated with Claude Code

luisleo526 and others added 14 commits October 26, 2026 09:03
scripts/test_ci_release_gate.py only. Adds an opt-in green ci-post fixture to FakeAPI (the runs read and
the jobs read of the proof commit; unknown endpoints still raise), a strict FakeHistory (first parent plus
changed paths, no classifier of its own) and 40 rows in CiPostGateTests: plain, docs-only-ancestor and
phase-two / release-bump passes with the evidence named in the green line; refusals for no run, running,
failed, cancelled, a newer red run next to an older green, a run for another sha or branch, each of the
four build jobs skipped / failed / missing, a jobs list shorter than its total, rate limit, 5xx, transport
and parse errors, a truncated or 1000-capped runs page, a docs walk blocked by a non-documentation path or
the old side of a rename, an empty diff, 21 docs-only commits, a root commit and unreadable history.

The gate does not read ci-post yet, so the new rows are expected RED on this commit (they pass the
`history` keyword gate() does not accept, and assert `ci-post: <code>` fragments); every existing row is
unchanged.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ere skipped (R-A rows) and ignores an older unfinished run

A ci-post run where preflight fails and the four build jobs are skipped is the
most common real red. An implementation that treats "all builds skipped" as
skipped-only before looking at the run conclusion would walk past that red run
to an older green one. Rows pin the check order: failed build job
(newer-run-failed), then run-level failure/cancelled/timed_out (not-success),
then full success, else continue; and that an older queued/in_progress run
below the proving run is never read.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…un (walk only across docs-only commits)

The release gate accepted a sha whose ci-post run was skipped or red, because
it only read the PR status contexts. It now also requires a ci-post run for
the proof sha (head_branch main, any event) whose four build jobs and run
conclusion are success. The proof sha is the release sha, or the nearest
first-parent ancestor reached across docs-only commits only (at most 20).

Runs are examined newest first (created_at, then id), at most 10, one jobs
read per completed run. A failed build job refuses with newer-run-failed; a
failed/cancelled/timed_out run conclusion refuses with not-success whatever
its build legs show (skipped legs after a failed preflight must not let the
walk slide to an older green run); a full success is the proof; any other run
is skipped-only and the walk continues to the next older run. Refusals carry
the codes no-run, run-in-progress, newer-run-failed, not-success, no-full-run,
unreadable, walk-blocked and walk-too-long. The required job names are one
tuple pinned against the ci-post.yml matrices by a workflow-pin test.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…e gate

git_history is the only production reader of first-parent history for the
ci-post walk, and every ci-post row injects a fake. These rows build a
throwaway repository per test and call the real function: root commit, normal
commit, rename (both sides), newline and non-ASCII paths, merge (first parent
only), unknown sha, empty commit, and no repository.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ts its walk skips (F-1 rows, red first)

A docs-only release sha whose own ci-post run is red (for example preflight
failed on doc-anchors) must not be released on the strength of a built
ancestor's green run. Rows pin: every skipped docs-only commit (tip first) has
a newest run that exists and is completed with conclusion success, judged by
the run-level conclusion only with no jobs read; docs-commit-red and
docs-commit-no-run refusals; newest run by (created_at, id) in both response
orders; unreadable / paginated / 1000-limit runs reads; the 20-commit cap
decided before any docs read; phase 2 and release-bump paths; and that a
success docs run with skipped builds is fine while a red built ancestor is
still refused. The docs-walk helpers register a successful docs run per
skipped sha and the exact read lists now include those reads.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…docs-only commit its walk skips

The walk across docs-only commits ignored the runs of the commits it skipped,
so a docs-only release sha whose own ci-post run was red (for example
preflight failed on doc-anchors) could be released on a built ancestor's green
run. _ci_post_proof now returns the skipped docs-only shas (tip first). Before
the proof commit's runs are read, each skipped sha's newest ci-post run (same
runs helper: main, exact sha, any event, newest by created_at then id) must
exist and be completed with conclusion success, judged by the run-level
conclusion only with no jobs read: ci-post: docs-commit-no-run sha <sha> or
ci-post: docs-commit-red <run id> sha <sha>. The 20-commit cap is still
decided during the walk, before any docs-run read. A docs run that succeeded
with skipped build jobs is fine. Also narrows the walk's except to
(OSError, ValueError), what the production history seam raises.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…the release gate (compare by commit, not run time)

The gate keeps only runs whose head_sha is the commit being examined (the
proof sha, or one skipped docs-only sha). Rows pin that a failed or
in-progress run of the parent commit, created and completed later than the
cut commit's green run and even listed in the same response, is neither
blocking nor read, while flipping only its head_sha to the examined commit
flips the verdict (newer-run-failed, not-success, run-in-progress for the
proof; docs-commit-red for a skipped docs-only commit). The docs-chain rows
also pin the exact runs and jobs reads.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Main moved by #517 and #518 (run_json, its tests, one docs page); none of the
release-gate files changed.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… of the proof sha (red first)

CORE-TOP ruled (10:51): on the proof sha a newer completed run blocks unless
its run-level conclusion is success; neutral, action_required, stale,
startup_failure, skipped and null each refuse with ci-post: not-success naming
the conclusion, for every examined run, while a failed build job keeps the
more specific newer-run-failed. Only a success run whose build legs are not a
full success (for example all four skipped) still falls through.

Two existing rows follow the ruling and keep their meaning: the loop of
test_skipped_only_newer_run_allows_older_full_green narrows to run-level
success (the run-level skipped case now belongs to the refusal rows), and the
middle run of test_only_skipped_runs_and_a_missing_matrix_name_give_no_full_run
is a success run with skipped legs instead of a run-level skipped run.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…f the proof sha

A newer completed ci-post run of the proof sha whose run-level conclusion is
neutral, action_required, stale, startup_failure, skipped or null was walked
past and an older green run released. The run-level conclusion must now be
success for every examined completed run, or the gate refuses with
ci-post: not-success naming the conclusion; a failed build job still takes
priority (newer-run-failed), and only a success run whose build legs are not
a full success falls through to an older run.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…s final rule), replacing the first fail-closed reading

A newer completed ci-post run of the proof sha: a failed/cancelled/timed_out
build job refuses newer-run-failed (first, over any run conclusion); a
run-level skipped run falls through only when all four build legs are skipped
(otherwise it refuses not-success as contradictory data); every other
non-success run conclusion (neutral, action_required, stale, startup_failure,
null) refuses not-success; a run-level success run falls through only when all
four legs are present and each success or skipped, a leg neutral, null,
unknown or missing refuses not-success naming the leg. The proof run itself
needs run-level success and all four legs success.

Existing rows follow the table and keep their meaning: the run-level-skipped
all-skipped shapes fall through again, test_other_completed_run_conclusions_refuse
expects not-success, and the uncertain-or-missing-cell row becomes a refusal
row. New rows name each table cell, including 132 failed-build-priority cases.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… CORE-TOP's table

Replaces the first fail-closed reading (any non-success run conclusion). A
newer completed ci-post run that is not a full green is passed over for an
older proof only when it is a run-level success whose four build legs are all
present and each success or skipped, or a run-level skipped run whose four legs
are all skipped; it is then counted as a newer skipped-only run. Every other
conclusion (neutral, action_required, stale, startup_failure, null, or skipped
with a non-skipped leg) refuses ci-post: not-success naming the conclusion and
the contradicting leg, and a run-level success with a neutral, null, unknown
or missing leg refuses not-success naming the leg while an older run exists
to be protected. A failed, cancelled or timed-out build job keeps priority
(newer-run-failed). The proof run is unchanged: run-level success and all four
legs success, otherwise no-full-run.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…it for a missing build leg

A newer run-level success run with a missing required build leg refuses
not-success while an older run exists to be protected, and ends in no-full-run
when it is the only examined run. Two rows pin both sides with the same
defective run for each of the four required build names.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Main moved by #519 (fd-flag tests only); none of the release-gate files changed.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@luisleo526 luisleo526 changed the title release gate: require a green ci-post proof run for the released commit (CR-353) release gate: require a green ci-post proof run for the released commit Oct 26, 2026
@luisleo526
luisleo526 merged commit d463222 into main Oct 26, 2026
13 checks passed
luisleo526 added a commit that referenced this pull request Oct 26, 2026
Adds one bullet to the CI and release process list from the merged change
#526: the release gate also requires a green, non-skipped post-merge run of the
released commit. Also corrects the scope of the mkstemp sentence in the #521
bullet to /tmp and /var/tmp. Documentation only.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526 added a commit that referenced this pull request Oct 26, 2026
Main moved by scripts and tests only (#525, #521, #526); CHANGELOG.md is
unchanged on main since the base of this branch, so the merge result is main's
tree with this branch's CHANGELOG.md.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526 added a commit that referenced this pull request Oct 26, 2026
…ase (#526)

A newer run with a failed, cancelled or timed-out build job refuses the
release gate; the previous wording said the run as such. Documentation only.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant