Skip to content

Show each fork CI job's status on gated pull requests - #75616

Merged
potiuk merged 1 commit into
apache:mainfrom
potiuk:fork-ci-job-statuses
Oct 25, 2026
Merged

potiuk merged 1 commit into
apache:mainfrom
potiuk:fork-ci-job-statuses

Conversation

@potiuk

@potiuk potiuk commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

A pull request gated to fork CI (AIP-120) shows one aggregate Fork CI / Tests (AMD) status, so to see which jobs are running, which failed and which passed, a maintainer or the contributor has to open the run in the fork. This reports every job of the fork run as its own commit status on the pull request, each linking to the job in the fork, the way project CI shows its jobs.

What it does

  • The fork CI monitor lists the jobs of the fork run it already tracks (GET /repos/<fork>/actions/runs/<id>/jobs?filter=latest) and posts one status per job as Fork CI job / <job name>: yellow while queued or running, green when passed, red when failed, timed out, cancelled or unable to start. Skipped jobs and the jobs every run has (Fork CI gate, Check if the run is needed, Platform: AMD, CI passed) are left out.
  • Only differences from the head commit's current statuses are posted, failures first. Queued and running share one status, so a job costs at most two writes.
  • Once every job of a completed run is reported, the run (<run id>:<attempt>:<conclusion>) is recorded in the state comment's marker, so later monitor runs don't read its jobs again. A run in progress is read on every monitor run, and a re-run in the fork is read again because its attempt changes.
  • A job context left over from an earlier run of the same commit, which the latest completed run doesn't have, is overwritten as success with "Not in the latest fork run - see Fork CI / Tests (AMD)", since commit statuses can't be deleted and have no neutral state.
  • The job statuses are informational. Merging still depends on CI passed and the state machine on Fork CI / Tests (AMD); neither changes.

Safety

  • Job names come from the contributor's workflow, so contexts get their own Fork CI job / prefix. No job can post the aggregate status the reconciler reads its decisions back from, or CI passed, which the merge ruleset requires. Non-printable characters are replaced, names are shortened to 100 characters, and contexts are grouped case-insensitively as GitHub compares them, keeping the worst result.
  • Reporting job statuses is caught on its own: a failure there is listed for Slack, and the pull request is still classified, so a CI passed that no longer holds is still revoked.
  • read_head_status now reads every page of the combined status, because with job statuses a commit can carry more than 100 contexts.

Write budget, and turning it off

GraphQL has no mutation for commit statuses (it can batch check runs, but a check run created by a workflow links to that workflow's run, not to the fork job), so each status is one REST write, counted against GitHub's limit of 500 content-creating requests an hour that every workflow of apache/airflow shares.

Measured on 2026-10-25 from apache/airflow's own Tests (AMD) PR runs (the same workflow runs in forks):

Pull request Run Started jobs Reported job statuses
Breeze test fix 40138784120 11 7
Documentation fix 40169750323 16 12
Google provider change 40158907296 43 39
CI change (full tests) 40207881565 131 127

apache/airflow had 1898 Tests (AMD) PR runs from 2026-10-18 to 2026-10-24, about 11 an hour. During the contributor pilot (a few gated pushes a day) job statuses use under 5% of the content-creating limit. With the gate enabled, and assuming about half of the runs come from gated pull requests, they would need about 400 writes an hour, so each monitor run posts at most 60 (at most 240 an hour, 48% of the limit). What doesn't fit is posted by later runs; once the budget is spent, no further jobs or statuses are read in that run, and the monitor posts a Slack note (fixed wording, so once while deferrals persist). The share of gated runs and the jobs per push are assumptions; the measurements and the estimate are in the "Write budget" section of dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md.

job_statuses = false in .github/fork-ci-gate.toml turns job statuses off; the aggregate status and CI passed are posted either way. dev/AGENTS.md says to do that when the budget is exceeded and lists the signs.

Verified

  • The dev/breeze test suite passes (1854 passed, 7 skipped), with new tests for the job statuses, the budget, the marker key, stale contexts, case-insensitive contexts, the switch, the Slack note and the paginated head status.
  • prek pre-commit hooks pass (the Docker-based lychee link check was not run), and so does mypy-dev.
  • Dry run against live apache/airflow on 2026-10-25 (breeze ci fork-ci-reconcile --dry-run from this branch, 4 GraphQL and 23 REST calls): for the one gated pull request, [test - do not merge]Update README.md #75586, it planned 7 job statuses, all success. That matches its fork run 40220648518 exactly: 11 started jobs, minus the 4 that are not reported. It also planned the usual Fork CI / Tests (AMD) and CI passed successes and a state-comment update, with no draft changes and nothing for Slack. Nothing was written: the head commit still had no statuses afterwards.

Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

🤖 Generated with Claude Code

A gated pull request showed one aggregate fork CI status, so a maintainer
or contributor had to open the fork run to see which jobs were running,
which failed and which passed. Reporting every job as its own check, each
linking to the job in the fork, puts that on the pull request itself,
where project CI shows its jobs.

Generated-by: Claude Opus 5
@potiuk
potiuk merged commit 2f7a7ea into apache:main Oct 25, 2026
28 of 108 checks passed
@potiuk
potiuk deleted the fork-ci-job-statuses branch October 25, 2026 18:37
@github-actions

Copy link
Copy Markdown
Contributor

Backport failed to create: v3-4-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
❌ v3-4-test Commit Link

You can attempt to backport this manually by running:

cherry_picker 2f7a7ea v3-4-test

This should apply the commit to the v3-4-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

vatsrahul1001 pushed a commit that referenced this pull request Oct 26, 2026
…75654, #75655) (#75680)

* Show each fork CI job's status on gated pull requests (#75616)

A gated pull request showed one aggregate fork CI status, so a maintainer
or contributor had to open the fork run to see which jobs were running,
which failed and which passed. Reporting every job as its own check, each
linking to the job in the fork, puts that on the pull request itself,
where project CI shows its jobs.

Generated-by: Claude Opus 5
(cherry picked from commit 2f7a7ea)

* Cancel the workflow runs of pull requests that have already merged (#75619)

Merging a pull request does not stop its pull_request runs, so they keep
using the project's GitHub-hosted minutes and self-hosted runners for a
result nobody reads once the change is on its target branch. Until now
maintainers cancelled such runs by hand, when they noticed them.

Generated-by: Claude Opus 5
(cherry picked from commit e5f2c87)

* Auto-approve cheap CI runs of external contributors' PRs (#75455)

* Auto-approve cheap CI runs of external contributors' PRs

GitHub holds the workflow runs of fork PRs until a maintainer approves
them. With self-hosted runners (AIP-118) the approval gate becomes part
of the trust model, so it should cover all external contributors, but
approving every push by hand does not scale and makes contributors wait.

Runs whose change cannot touch the CI definition and whose CI is cheap
are safe to start without a maintainer. Expensive or large changes, and
anything that changes how CI or runners are chosen, still get a human
look first, which also keeps the load on the shared public runners down.
The CI-minutes estimate lets maintainers see what a run will cost before
approving it.

Generated-by: Claude Opus 5

* Add inline script metadata to the CI-minutes calibration script

The script uses only the standard library, so with a PEP 723 block it runs
with plain `uv run dev/calibrate_ci_minutes_estimates.py` instead of syncing
the whole Breeze environment. Document that invocation in the script and in
.github/AGENTS.md.

* Keep one PR's API or git error from aborting the approval tick

Each PR is now processed in its own try block. A requests.RequestException
or subprocess.CalledProcessError for one PR no longer stops the remaining
PRs or skips the job summary. The failed PR is recorded with failed=True and
shown as "failed" in the summary, keeping its CI-time estimate when one was
computed and giving a short, single-line form of the error as the reason.
The workflow also tries to add the 'need workflow approval' label to it, so
it is not lost once its runs leave the query window. Nothing is approved on
failure.

approve_workflow_runs exits 1 after writing the summary when any PR failed,
so a persistent failure turns the scheduled run red instead of passing
silently.

* Approve the runs of fork CI pull requests unless they change CI

A pull request whose CI runs in the contributor's fork (AIP-120) still
gets pull_request runs in apache/airflow, and GitHub holds them until a
maintainer approves them. Those runs only execute the fork CI gate, so
judging them by the cost and size of project CI left cheap gate-only
runs waiting for a maintainer for no benefit, and a gated run that never
ran could not be cleaned up by the reconciler.

What still needs a maintainer is a change to the files that define CI,
since it could remove the gate or add a workflow that does run. Using
the reconciler's own list keeps the two decisions consistent, and the
author is told why their runs wait instead of only seeing a label.

Generated-by: Claude Opus 5

* Keep CI-minutes estimates true to the runs they approve

A run selects its tests from the labels the PR had when it started, so
approving it on the current labels could start a full-tests run whose
label was removed while it waited. Calibrating from self-hosted and
all-versions runs, and leaving out the jobs that need no CI image, made
the estimate describe runs other than those it is used to approve.

Generated-by: Claude Opus 5

---------

Co-authored-by: Shahar Epstein <[email protected]>
(cherry picked from commit 2a3ee7f)

* Approve fork CI runs that started before "use fork ci" was added (#75654)

When a committer adds "use fork ci" right after a PR opens, the PR's first
runs were created without it, and the workflow-run approver left them for a
maintainer. It had to: the fork CI gate fell back to the labels of the run's
event when it could not read the current ones, and those labels could still
send the run to full project CI on code nobody had reviewed. Anyone able to
exhaust the shared token quota could force that fallback.

The gate now fails when it cannot read the labels, so a run only ever acts on
the PR's current labels and a re-run decides again. With that, runs started
before the label changed are as safe to approve as later ones.

Generated-by: Claude Opus 5
(cherry picked from commit 437daf6)

* Check for workflow runs to auto-approve every 15 minutes (#75655)

External contributors' runs wait for the next approver pass before they
start, and GitHub regularly drops scheduled runs under load, which turns the
30-minute wait into an hour or more. A pass takes about a minute, so running
it twice as often costs little and halves the wait.

Generated-by: Claude Opus 5
(cherry picked from commit 54f645d)

* Pin the real selective-checks auto-approve test to main

The test decides approval for a PR targeting main, but built SelectiveChecks
with the checkout's default branch. On a release branch that skips provider
tests, so the large provider change was estimated cheap and approved.

Generated-by: Claude Opus 5

---------

Co-authored-by: Shahar Epstein <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants