Skip to content

Commit 437daf6

Browse files
authored
Approve fork CI runs that started before "use fork ci" was added (#75654)
When a committer adds "use fork ci" right after a PR opens, the PR's first runs were created without it, and the workflow-run approver left them for a maintainer. It had to: the fork CI gate fell back to the labels of the run's event when it could not read the current ones, and those labels could still send the run to full project CI on code nobody had reviewed. Anyone able to exhaust the shared token quota could force that fallback. The gate now fails when it cannot read the labels, so a run only ever acts on the PR's current labels and a re-run decides again. With that, runs started before the label changed are as safe to approve as later ones. Generated-by: Claude Opus 5
1 parent 8dde17b commit 437daf6

8 files changed

Lines changed: 56 additions & 58 deletions

File tree

‎.github/workflows/ci-amd.yml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -233,7 +233,6 @@ jobs:
233233
PR_AUTHOR: ${{ github.event.pull_request.user.login }}
234234
PR_AUTHOR_ASSOCIATION: ${{ github.event.pull_request.author_association }}
235235
PR_CREATED_AT: ${{ github.event.pull_request.created_at }}
236-
PR_EVENT_LABELS: ${{ toJSON(github.event.pull_request.labels.*.name) }}
237236
# A base branch with neither gate file predates the gate (the pull request adding it,
238237
# or a release branch cut before it) and keeps running project CI. With only one of
239238
# them present, the missing file fails the job instead of opening the gate.

‎dev/breeze/doc/ci/05_workflows.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -138,9 +138,9 @@ run started before a label that changes which tests run was added or removed
138138
(`full tests needed`, `all versions`, `default versions only`, `latest versions only`, `canary`,
139139
`upgrade to newer dependencies`, `area:e2e-tests`, `area:kubernetes-tests`) is left for a
140140
maintainer, since it still runs the selection it started with; the next push is decided afresh.
141-
For a PR whose CI runs in the fork the selection does not matter, but the gate falls back to the
142-
labels a run started with when it cannot read the current ones, so a run started before
143-
`use fork ci` or `use project ci` was added or removed is left for a maintainer instead. Other PRs get the
141+
For a PR whose CI runs in the fork neither the selection nor the labels a run started with matter:
142+
the gate reads the current labels when the run starts and fails when it cannot, so a run started
143+
before `use fork ci` or `use project ci` was added or removed is approved too. Other PRs get the
144144
`need workflow approval` label and wait for a maintainer. When processing a PR fails (for example on
145145
a GitHub API error), the workflow tries to add the label to it and the workflow run fails, so the
146146
failure stays visible even if the label could not be added. The label is removed when a later commit

‎dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -310,9 +310,11 @@ directly with the runner's `python3` without installing breeze. The reconciler i
310310
full project CI as `base_without_gate`.
311311
2. For pull requests, one read-only API call for the current labels
312312
(`gh api repos/apache/airflow/pulls/<n> --jq '.labels[].name'`), so override labels added
313-
after the event are seen on a re-run. If that call fails, the gate uses the labels of the
314-
event payload instead. It never falls back to running project CI: anyone able to exhaust
315-
the shared token quota would otherwise switch the gate off for every pull request.
313+
after the event are seen on a re-run. If that call fails, the gate job fails and nothing
314+
after it runs; re-running the job decides again. It falls back neither to running project
315+
CI, since anyone able to exhaust the shared token quota would otherwise switch the gate off
316+
for every pull request, nor to the labels of the event payload, since runs started before
317+
`use fork ci` was added are approved automatically (see "Approving gated runs").
316318
3. `python3 dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py` with the event payload
317319
fields and the labels passed as environment variables. It writes `project-ci=true|false`
318320
and `reason=...` to `GITHUB_OUTPUT`. A base branch with neither file skips the script and
@@ -367,9 +369,10 @@ applies instead:
367369
`github-actions[bot]` with the marker count. A change to `.github/` can remove the gate or add a
368370
workflow that does run, so such a run must not start without a review.
369371

370-
A run started before `use fork ci` or `use project ci` was last added or removed is not approved
371-
automatically: the gate falls back to the labels of the run's event when it cannot read the current
372-
ones, and those can still send the run to project CI. The next push is decided afresh.
372+
A run started before `use fork ci` or `use project ci` was last added or removed is approved too,
373+
so a committer can add `use fork ci` after the PR's first runs were created and before they are
374+
approved. The gate decides from the labels the PR has when the run starts, never from those of the
375+
run's event, and fails when it cannot read them.
373376

374377
Both rules use the same list as "Trusting a green fork run", so a gated pull request whose runs are
375378
approved automatically is also one whose green fork run is trusted, and one that changes CI is both

‎dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -214,6 +214,10 @@ class Decision:
214214
reason: Reason
215215

216216

217+
class LabelsUnavailableError(RuntimeError):
218+
"""The current labels of a pull request could not be read."""
219+
220+
217221
def _is_canonical_branch(ref: str) -> bool:
218222
branch = ref.removeprefix("refs/heads/")
219223
return any(pattern.match(branch) for pattern in CANONICAL_PUSH_BRANCH_PATTERNS)
@@ -300,10 +304,12 @@ def evaluate_from_env(env: Mapping[str, str]) -> Decision:
300304
if event_name == "pull_request":
301305
fetched = fetch_pull_request_labels(repository, env.get("PR_NUMBER", ""))
302306
if fetched is None:
303-
# Never fail open: anyone able to exhaust the shared token quota would otherwise switch
304-
# the gate off for every pull request. The event's labels miss only labels added since.
305-
print("Falling back to the labels of the triggering event", file=sys.stderr)
306-
fetched = tuple(json.loads(env.get("PR_EVENT_LABELS") or "[]"))
307+
# Neither project CI nor the event's labels: the workflow-run approver approves runs
308+
# started before `use fork ci` was added, which the event's labels would send to project
309+
# CI, and anyone able to exhaust the shared token quota could force that path.
310+
raise LabelsUnavailableError(
311+
f"Could not read the labels of pull request #{env.get('PR_NUMBER', '')}; re-run the job"
312+
)
307313
labels = fetched
308314
event = GateEvent(
309315
event_name=event_name,
@@ -321,7 +327,11 @@ def evaluate_from_env(env: Mapping[str, str]) -> Decision:
321327

322328
def main(env: Mapping[str, str] | None = None) -> int:
323329
env = os.environ if env is None else env
324-
decision = evaluate_from_env(env)
330+
try:
331+
decision = evaluate_from_env(env)
332+
except LabelsUnavailableError as e:
333+
print(e, file=sys.stderr)
334+
return 1
325335
project_ci = "true" if decision.run else "false"
326336
print(f"project-ci={project_ci} reason={decision.reason.value}")
327337
_write_line("GITHUB_OUTPUT", f"project-ci={project_ci}", env)

‎dev/breeze/src/airflow_breeze/utils/workflow_run_approval.py‎

Lines changed: 4 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -49,8 +49,6 @@
4949
from airflow_breeze.utils.console import console_print
5050
from airflow_breeze.utils.fork_ci_gate import (
5151
GATED_REASONS,
52-
LABEL_USE_FORK_CI,
53-
LABEL_USE_PROJECT_CI,
5452
GateConfig,
5553
GateEvent,
5654
decide,
@@ -132,11 +130,6 @@
132130
}
133131
)
134132

135-
# Labels that move a PR between project CI and fork CI. The gate reads them when it runs, but falls
136-
# back to the labels of the run's event when it cannot, so a run started before one changed could
137-
# still run project CI.
138-
GATE_OVERRIDE_LABELS = frozenset({LABEL_USE_FORK_CI, LABEL_USE_PROJECT_CI})
139-
140133
# Read from the trusted main checkout, which is also the base branch whose gate job decides the
141134
# PRs that are approved automatically.
142135
FORK_CI_GATE_CONFIG_PATH = AIRFLOW_ROOT_PATH / ".github" / "fork-ci-gate.toml"
@@ -225,7 +218,9 @@ def decide_gated_approval(*, changed_files: list[ChangedFile]) -> ApprovalDecisi
225218
return ApprovalDecision(
226219
approve=True,
227220
reasons=["its CI runs in the contributor's fork, so its project CI run only executes the gate"],
228-
decision_labels=GATE_OVERRIDE_LABELS,
221+
# The gate reads `use fork ci` and `use project ci` when the run starts and fails when it
222+
# cannot, so runs started before one of them changed are approved too.
223+
decision_labels=frozenset(),
229224
)
230225

231226

@@ -655,7 +650,7 @@ def find_runs_started_for_current_pr(
655650
:param decision_labels: the labels the decision depends on. It is made from the current labels,
656651
while a run can act on those of its event: with ``full tests needed`` removed, or
657652
``default versions only`` added to a run of all versions, the estimate is smaller than the
658-
run, and without ``use fork ci`` in its event a run can fall back to project CI.
653+
run.
659654
"""
660655
started_after = parse_github_time(pr["created_at"])
661656
changes = [

‎dev/breeze/tests/test_fork_ci_gate.py‎

Lines changed: 20 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -391,44 +391,39 @@ def _outputs(env: dict[str, str]) -> list[str]:
391391

392392
@mock.patch("airflow_breeze.utils.fork_ci_gate.subprocess.run", autospec=True)
393393
@pytest.mark.parametrize(
394-
("returncode", "stdout", "event_labels", "expected"),
394+
("stdout", "expected"),
395395
[
396-
pytest.param(0, "[]\n", "[]", ["project-ci=false", "reason=external"], id="external"),
397-
pytest.param(
398-
0, '["use project ci"]\n', "[]", ["project-ci=true", "reason=label_project_ci"], id="labelled"
399-
),
400-
pytest.param(1, "", "[]", ["project-ci=false", "reason=external"], id="gh-fails-stays-gated"),
401-
pytest.param(
402-
0, "not json", "[]", ["project-ci=false", "reason=external"], id="bad-output-stays-gated"
403-
),
404-
pytest.param(
405-
1,
406-
"",
407-
'["use project ci"]',
408-
["project-ci=true", "reason=label_project_ci"],
409-
id="gh-fails-uses-event-labels",
410-
),
396+
pytest.param("[]\n", ["project-ci=false", "reason=external"], id="external"),
397+
pytest.param('["use project ci"]\n', ["project-ci=true", "reason=label_project_ci"], id="labelled"),
411398
],
412399
)
413-
def test_main_pull_request(mock_run, gate_env, returncode, stdout, event_labels, expected):
414-
mock_run.return_value = subprocess.CompletedProcess([], returncode, stdout=stdout, stderr="boom")
415-
gate_env["PR_EVENT_LABELS"] = event_labels
400+
def test_main_pull_request(mock_run, gate_env, stdout, expected):
401+
mock_run.return_value = subprocess.CompletedProcess([], 0, stdout=stdout, stderr="")
416402

417403
assert main(gate_env) == 0
418404

419405
assert _outputs(gate_env) == expected
420406
assert mock_run.call_args_list[0].args[0][:3] == ["gh", "api", "repos/apache/airflow/pulls/1"]
421407

422408

423-
@mock.patch(
424-
"airflow_breeze.utils.fork_ci_gate.subprocess.run", autospec=True, side_effect=FileNotFoundError("gh")
409+
@mock.patch("airflow_breeze.utils.fork_ci_gate.subprocess.run", autospec=True)
410+
@pytest.mark.parametrize(
411+
("returncode", "stdout", "side_effect"),
412+
[
413+
pytest.param(1, "", None, id="gh-fails"),
414+
pytest.param(0, "not json", None, id="bad-output"),
415+
pytest.param(0, "", FileNotFoundError("gh"), id="no-gh"),
416+
],
425417
)
426-
def test_main_without_gh_uses_the_event_labels(mock_run, gate_env):
427-
gate_env["PR_EVENT_LABELS"] = '["use project ci"]'
418+
def test_main_pull_request_fails_when_labels_cannot_be_read(
419+
mock_run, gate_env, returncode, stdout, side_effect
420+
):
421+
mock_run.return_value = subprocess.CompletedProcess([], returncode, stdout=stdout, stderr="boom")
422+
mock_run.side_effect = side_effect
428423

429-
main(gate_env)
424+
assert main(gate_env) == 1
430425

431-
assert _outputs(gate_env) == ["project-ci=true", "reason=label_project_ci"]
426+
assert not Path(gate_env["GITHUB_OUTPUT"]).exists()
432427

433428

434429
@mock.patch("airflow_breeze.utils.fork_ci_gate.subprocess.run", autospec=True)

‎dev/breeze/tests/test_workflow_run_approval.py‎

Lines changed: 5 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,6 @@
3131
from airflow_breeze.utils.selective_checks import SelectiveChecks
3232
from airflow_breeze.utils.shared_options import set_dry_run
3333
from airflow_breeze.utils.workflow_run_approval import (
34-
GATE_OVERRIDE_LABELS,
3534
MAX_AUTO_APPROVED_JOB_MINUTES,
3635
MAX_AUTO_APPROVED_SOURCE_LINES,
3736
MAX_ERROR_TEXT_LENGTH,
@@ -995,13 +994,11 @@ def test_get_last_label_change_ignores_other_labels_and_reads_the_timeline_once(
995994
@mock.patch(f"{MODULE}.load_config", autospec=True, return_value=GATE_ENABLED)
996995
@mock.patch(f"{MODULE}.get_changed_files", autospec=True, return_value=CORE_CHANGE)
997996
@mock.patch(f"{MODULE}.fetch_merge_commit", autospec=True, return_value="merge")
998-
def test_gated_approval_depends_on_the_gate_override_labels(
999-
mock_fetch, mock_get_changed_files, mock_load_config
1000-
):
997+
def test_gated_approval_depends_on_no_label(mock_fetch, mock_get_changed_files, mock_load_config):
1001998
api = mock.MagicMock(spec=GitHubApi)
1002999
api.github_repository = "apache/airflow"
10031000

1004-
assert decide_for_pull_request(api=api, pr=_gated_pr()).decision_labels == GATE_OVERRIDE_LABELS
1001+
assert decide_for_pull_request(api=api, pr=_gated_pr()).decision_labels == frozenset()
10051002

10061003

10071004
@pytest.mark.parametrize("event", ["labeled", "unlabeled"])
@@ -1016,14 +1013,14 @@ def test_get_last_label_change_counts_every_test_selection_label(label, event):
10161013

10171014

10181015
@mock.patch(f"{MODULE}.decide_for_pull_request", autospec=True)
1019-
def test_process_waiting_runs_leaves_gated_runs_started_before_use_fork_ci_was_added(mock_decide):
1016+
def test_process_waiting_runs_approves_gated_runs_started_before_use_fork_ci_was_added(mock_decide):
10201017
api = GitHubApi(github_repository="apache/airflow", github_token="token", report_only=False)
10211018
timeline = [
10221019
{"event": "labeled", "created_at": "2026-10-22T10:30:00Z", "label": {"name": "use fork ci"}},
10231020
{"event": "unlabeled", "created_at": "2026-10-22T10:40:00Z", "label": {"name": "full tests needed"}},
10241021
]
10251022
mock_decide.return_value = ApprovalDecision(
1026-
approve=True, reasons=["fork CI"], decision_labels=GATE_OVERRIDE_LABELS
1023+
approve=True, reasons=["fork CI"], decision_labels=frozenset()
10271024
)
10281025
with (
10291026
mock.patch.object(
@@ -1041,4 +1038,4 @@ def test_process_waiting_runs_leaves_gated_runs_started_before_use_fork_ci_was_a
10411038
):
10421039
process_waiting_runs(api, created_since=CREATED_SINCE)
10431040

1044-
assert approve_run.call_args_list == [mock.call(3)]
1041+
assert approve_run.call_args_list == [mock.call(2), mock.call(3)]

‎scripts/ci/prek/check_ci_workflows_in_sync.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -229,7 +229,6 @@
229229
PR_AUTHOR: ${{ github.event.pull_request.user.login }}
230230
PR_AUTHOR_ASSOCIATION: ${{ github.event.pull_request.author_association }}
231231
PR_CREATED_AT: ${{ github.event.pull_request.created_at }}
232-
PR_EVENT_LABELS: ${{ toJSON(github.event.pull_request.labels.*.name) }}
233232
# A base branch with neither gate file predates the gate (the pull request adding it,
234233
# or a release branch cut before it) and keeps running project CI. With only one of
235234
# them present, the missing file fails the job instead of opening the gate.

0 commit comments

Comments
 (0)