Skip to content

Commit 2f7a7ea

Browse files
authored
Show each fork CI job's status on gated pull requests (#75616)
A gated pull request showed one aggregate fork CI status, so a maintainer or contributor had to open the fork run to see which jobs were running, which failed and which passed. Reporting every job as its own check, each linking to the job in the fork, puts that on the pull request itself, where project CI shows its jobs. Generated-by: Claude Opus 5
1 parent 4386cb1 commit 2f7a7ea

9 files changed

Lines changed: 690 additions & 9 deletions

File tree

‎.github/fork-ci-gate.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,11 @@ enabled_since = 2026-01-01T00:00:00Z
2929
# Days without any fork CI activity after which a gated pull request is closed.
3030
close_after_days = 7
3131

32+
# Report each job of a gated pull request's fork run as its own commit status. Set to false when
33+
# the fork CI monitor exceeds GitHub's rate limits (see dev/AGENTS.md); the `Fork CI / Tests (AMD)`
34+
# status is posted either way.
35+
job_statuses = true
36+
3237
# Organisations exempted under AIP-120 §2.2. Keys are display names used in comments,
3338
# values are GitHub logins. Changes go through a reviewed pull request.
3439
[allowlist]

‎dev/AGENTS.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77

88
- [dev/ directory guidelines](#dev-directory-guidelines)
99
- [Scripts](#scripts)
10+
- [Fork CI job statuses and GitHub's rate limits](#fork-ci-job-statuses-and-githubs-rate-limits)
1011

1112
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
1213

@@ -42,3 +43,19 @@ uv run dev/my_script.py [args...]
4243
```
4344

4445
Document `uv run` (not `python`) as the invocation method in READMEs and instructions.
46+
47+
## Fork CI job statuses and GitHub's rate limits
48+
49+
The fork CI monitor (`breeze ci fork-ci-reconcile`, run by `.github/workflows/fork-ci-monitor.yml`)
50+
reports every job of a gated pull request's fork run as its own commit status. Those writes are the
51+
largest share of what the monitor does with the repository's `GITHUB_TOKEN`, whose rate limits every
52+
workflow of `apache/airflow` shares (see "Write budget" in
53+
`dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md`).
54+
55+
If the budget is exceeded, turn the job statuses off: set `job_statuses = false` in
56+
`.github/fork-ci-gate.toml`. Signs that it is exceeded are the monitor's Slack note that job
57+
statuses "did not fit in the budget" persisting across runs, rate-limit retries ("GitHub rate limit,
58+
retrying") or "still rate limited" errors in the monitor's run log or its Slack messages, or other
59+
workflows of the repository failing on secondary rate limits. The aggregate `Fork CI / Tests (AMD)`
60+
status and `CI passed` are posted either way, so turning job statuses off loses detail, not gating.
61+
Lowering `MAX_JOB_STATUSES_PER_RUN` in `fork_ci_reconcile.py` is the softer alternative.

‎dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md‎

Lines changed: 86 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -661,6 +661,87 @@ Commit statuses were chosen over check runs because a check run created with the
661661
token is attached to the creating workflow's check suite and its details link then opens the
662662
monitor run rather than `details_url`. A status honours `target_url`.
663663

664+
### Job statuses
665+
666+
Next to the aggregate status, each job of the fork run is reported as its own commit status, so
667+
the pull request shows which jobs run, failed or passed, each linking to the job in the fork.
668+
The run is the one `Fork CI / Tests (AMD)` reports (its `databaseId` is fetched with the check
669+
suites); its jobs come from `GET /repos/<fork>/actions/runs/<id>/jobs?filter=latest`.
670+
671+
- The context is `Fork CI job / <job name>`, with non-printable characters replaced and the name
672+
shortened to 100 characters. Job names come from the workflow of the contributor's commit, so
673+
the prefix is what keeps a job from posting `Fork CI / Tests (AMD)`, from which the reconciler
674+
reads its decisions back, or `CI passed`, which the merge ruleset requires. Jobs that share a
675+
context, which GitHub compares case-insensitively, are reported as the worst of them.
676+
- A queued or running job is `pending` ("Running in the fork"), a successful one `success`, and a
677+
failed, timed-out, cancelled or unstartable one `failure`. Skipped and neutral jobs, and the jobs
678+
every run has (`Fork CI gate`, `Check if the run is needed`, `Platform: AMD` and `CI passed`), are
679+
not reported. Queued and running share one status so that starting a job costs no write.
680+
- The desired statuses are compared with the head commit's current ones
681+
(`GET /repos/apache/airflow/commits/<sha>/status`), and only differences are posted, failures
682+
first. GraphQL has no mutation for commit statuses (`createCheckRun` exists, but see above for
683+
why check runs do not fit), so each one is a REST call, and they count towards GitHub's limits on
684+
content-creating requests: a reconciler run posts at most `MAX_JOB_STATUSES_PER_RUN` (60) of
685+
them, each write charged as it is made. What does not fit is posted by the next run, and once
686+
the budget is spent no further pull request's jobs or statuses are read in that run. A run that
687+
defers job statuses adds a note to the Slack message, with fixed wording so it is posted once
688+
while deferrals persist; the run log lists the pull requests.
689+
- Commit statuses cannot be deleted and have no neutral state. When a run has completed, a job
690+
context posted for an earlier run of the same commit that the completed run does not have, or
691+
skipped, is overwritten with `success` and the description "Not in the latest fork run - see
692+
Fork CI / Tests (AMD)", so it no longer shows as running or failed.
693+
- Once every job of a completed run is reported, the run is recorded in the marker as
694+
`jobs: "<run id>:<attempt>:<conclusion>"` (`runAttempt` is fetched with the check suites), so
695+
later runs do not read its jobs again. A run in progress is read on every run, and a re-run in
696+
the fork is read again because its attempt changes, even when it ends with the same conclusion.
697+
- Reporting job statuses is caught on its own: when it fails, the failure is listed with the
698+
unknown items for Slack, and the pull request is still classified, so for example a `CI passed`
699+
that no longer holds is still revoked.
700+
701+
The job statuses are informational. Merging still depends on `CI passed`, and the state machine
702+
on `Fork CI / Tests (AMD)`; a job status never changes either.
703+
704+
`job_statuses = false` in `.github/fork-ci-gate.toml` turns them off; the reconciler then posts
705+
only `Fork CI / Tests (AMD)` and `CI passed`, as before. Statuses already posted stay on their
706+
commits. Turn them off when the fork CI monitor's runs hit GitHub's rate limits (rate-limit
707+
retries in the run log, or "still rate limited" errors in Slack); `dev/AGENTS.md` records this for
708+
agents.
709+
710+
#### Write budget
711+
712+
Measured on 2026-10-25 from `apache/airflow`'s own `Tests (AMD)` pull request runs (the same
713+
workflow runs in the forks; `GET /repos/apache/airflow/actions/runs/<id>/jobs?filter=latest`,
714+
jobs whose conclusion is not `skipped`, minus the four jobs that are not reported):
715+
716+
| Pull request | Run | Started jobs | Reported job statuses |
717+
|---|---|---|---|
718+
| Breeze test fix | [40138784120](https://github.com/apache/airflow/actions/runs/40138784120) | 11 | 7 |
719+
| Documentation fix | [40169750323](https://github.com/apache/airflow/actions/runs/40169750323) | 16 | 12 |
720+
| Google provider change | [40158907296](https://github.com/apache/airflow/actions/runs/40158907296) | 43 | 39 |
721+
| CI change (full tests) | [40207881565](https://github.com/apache/airflow/actions/runs/40207881565) | 131 | 127 |
722+
723+
`apache/airflow` had 1898 `Tests (AMD)` pull request runs from 2026-10-18 to 2026-10-24, about 270 a
724+
day or 11 an hour. Each reported job costs at most two writes (running, then its result) and one
725+
when it starts and finishes between two reconciler runs; a fork run of 30-60 minutes spans two to
726+
four of them, so most jobs cost two.
727+
728+
The limits that matter are GitHub's documented ones for the `GITHUB_TOKEN` of `apache/airflow`,
729+
shared by every workflow of the repository: 1000 REST requests an hour, and 80 content-creating
730+
requests a minute and 500 an hour. Assuming about 35 reported jobs for an average push:
731+
732+
- **Contributor pilot**, a few gated pushes a day: about 70 writes a push, under 20 an hour, under
733+
5% of the content-creating limit.
734+
- **Gate enabled**, assuming about half of the pull request runs come from gated pull requests
735+
(about 5-6 pushes an hour): about 400 writes an hour without a cap, 80% of the content-creating
736+
limit and more at peaks. The cap of 60 writes a run, four runs an hour, holds job statuses to at
737+
most 240 an hour: 48% of the content-creating limit and 24% of the REST limit, plus two reads per
738+
pull request with a run in progress. Job statuses then lag behind at busy times, the aggregate
739+
status does not.
740+
741+
These are estimates: the share of gated runs, the jobs per push and the pushes per pull request
742+
are assumptions, not measurements. Watch the monitor's call counts in its run log after enabling
743+
the gate, and turn `job_statuses` off if they approach the limits.
744+
664745
### Relationship to the triage skill
665746

666747
The `pr-management-triage` skill described in `contributing-docs/25_maintainer_pr_triage.md`
@@ -760,7 +841,9 @@ Per run, with ~700 gated pull requests: ~28 GraphQL calls for pass 2, ~14 for fo
760841
calls for the recent `pull_request` runs, one REST call per base branch for its workflow
761842
files, a jobs call and a compare call only per long `ci-amd.yml` run of a gated pull request,
762843
one changed-files call per head commit that turns green, plus one mutation per actual state
763-
change. Well inside
844+
change. Job statuses add a jobs call and a status call per pull request with a fork run in
845+
progress or a completed run not yet reported, and up to two status mutations per job, at most
846+
`MAX_JOB_STATUSES_PER_RUN` per run; see "Write budget". Well inside
764847
the `GITHUB_TOKEN` limits at a 15-minute cadence. The reconciler logs its call counts so the
765848
budget stays observable.
766849

@@ -906,6 +989,8 @@ as a failed check rather than a crash.
906989
workflow files on the base branch.
907990
- A green fork run is trusted only for pull requests that do not change what fork CI runs;
908991
see "Trusting a green fork run".
992+
- Job statuses carry names from the contributor's workflow under their own `Fork CI job / <name>`
993+
prefix, so they cannot post the reconciler's own contexts or `CI passed`; see "Job statuses".
909994
- The reconciler runs on `main` with `pull-requests: write` and never checks out or executes
910995
pull request content. Its inputs are API data; its comment templates are static.
911996
- Fork runs get the fork's `GITHUB_TOKEN` only. Nothing in project CI trusts a fork run's

‎dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,7 @@ class GateConfig:
128128
close_after_days: int = DEFAULT_CLOSE_AFTER_DAYS
129129
allowlist: Mapping[str, tuple[str, ...]] = field(default_factory=dict)
130130
pilot: Mapping[str, datetime] = field(default_factory=dict)
131+
job_statuses: bool = True
131132

132133
def is_allowlisted(self, login: str) -> bool:
133134
wanted = login.casefold()
@@ -173,12 +174,16 @@ def parse_config(text: str) -> GateConfig:
173174
login.casefold(): _require_aware_datetime(joined_at, f"pilot entry {login!r}")
174175
for login, joined_at in data.get("pilot", {}).items()
175176
}
177+
job_statuses = data.get("job_statuses", True)
178+
if not isinstance(job_statuses, bool):
179+
raise ValueError(f"job_statuses must be true or false, got {job_statuses!r}")
176180
return GateConfig(
177181
enabled=enabled,
178182
enabled_since=enabled_since,
179183
close_after_days=close_after_days,
180184
allowlist=allowlist,
181185
pilot=pilot,
186+
job_statuses=job_statuses,
182187
)
183188

184189

‎dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,10 @@
3030
from datetime import datetime
3131

3232
STATUS_CONTEXT = "Fork CI / Tests (AMD)"
33+
# Each job of the fork run is reported under this prefix. Job names come from the contributor's
34+
# workflow, so the prefix keeps them from posting STATUS_CONTEXT or CI_PASSED_CONTEXT.
35+
JOB_STATUS_CONTEXT_PREFIX = "Fork CI job / "
36+
JOB_NAME_MAX_LENGTH = 100
3337
# The check the "Require CI on main" ruleset requires before a pull request can (auto-)merge. Project
3438
# CI's gate job reports it; for a gated pull request, which never runs project CI, the reconciler
3539
# reports it from the fork run.
@@ -74,12 +78,16 @@ class Marker:
7478
since: datetime
7579
# Commits the head commit was behind its base branch when its fork CI turned green.
7680
behind: int | None = None
81+
# The completed fork run (``<run id>:<attempt>:<conclusion>``) whose jobs are all reported.
82+
jobs: str | None = None
7783

7884

7985
def render_marker(marker: Marker) -> str:
8086
data: dict[str, object] = {"state": marker.state, "sha": marker.sha, "since": marker.since.isoformat()}
8187
if marker.behind is not None:
8288
data["behind"] = marker.behind
89+
if marker.jobs is not None:
90+
data["jobs"] = marker.jobs
8391
return f"{MARKER_PREFIX}{json.dumps(data, sort_keys=True)} -->"
8492

8593

@@ -95,6 +103,7 @@ def parse_marker(body: str) -> Marker | None:
95103
sha=data["sha"],
96104
since=datetime.fromisoformat(data["since"]),
97105
behind=data.get("behind"),
106+
jobs=data.get("jobs"),
98107
)
99108
except (json.JSONDecodeError, KeyError, TypeError, ValueError) as e:
100109
raise ValueError(f"Malformed fork CI marker: {match.group(0)!r}") from e
@@ -217,6 +226,33 @@ def tamper_close_comment(*, run_url: str) -> str:
217226
"ungated": "Project CI in use (base branch without fork CI) - see the checks",
218227
}
219228

229+
# (commit status state, description) of a fork job, by its status and conclusion. A queued job is
230+
# reported like a running one: each change is one more write, and commit statuses can only be posted
231+
# one at a time (GraphQL has no mutation for them).
232+
JOB_STATUSES = {
233+
"running": ("pending", "Running in the fork - see Details"),
234+
"success": ("success", "Passed in the fork - see Details"),
235+
"failure": ("failure", "Failed in the fork - see Details"),
236+
"timed_out": ("failure", "Timed out in the fork - see Details"),
237+
"startup_failure": ("failure", "Failed to start in the fork - see Details"),
238+
"cancelled": ("failure", "Cancelled in the fork - see Details"),
239+
}
240+
241+
242+
# Commit statuses cannot be deleted and have no neutral state. A job reported for an earlier run that
243+
# the latest completed run does not have is overwritten with this, so it no longer shows as running
244+
# or failed; `Fork CI / Tests (AMD)` is what tells the result.
245+
STALE_JOB_STATUS = ("success", "Not in the latest fork run - see Fork CI / Tests (AMD)")
246+
247+
248+
def job_status_context(job_name: str) -> str:
249+
# A line break or control character would make an unreadable check name.
250+
name = "".join(char if char.isprintable() else " " for char in job_name).strip()
251+
if len(name) > JOB_NAME_MAX_LENGTH:
252+
name = f"{name[: JOB_NAME_MAX_LENGTH - 1]}…"
253+
return f"{JOB_STATUS_CONTEXT_PREFIX}{name}"
254+
255+
220256
STATUS_STATES = {
221257
"setup_required": "error",
222258
"awaiting_push": "pending",

0 commit comments

Comments
 (0)