Skip to content

Commit 2a3ee7f

Browse files
potiukshahar1
andauthored
Auto-approve cheap CI runs of external contributors' PRs (#75455)
* Auto-approve cheap CI runs of external contributors' PRs GitHub holds the workflow runs of fork PRs until a maintainer approves them. With self-hosted runners (AIP-118) the approval gate becomes part of the trust model, so it should cover all external contributors, but approving every push by hand does not scale and makes contributors wait. Runs whose change cannot touch the CI definition and whose CI is cheap are safe to start without a maintainer. Expensive or large changes, and anything that changes how CI or runners are chosen, still get a human look first, which also keeps the load on the shared public runners down. The CI-minutes estimate lets maintainers see what a run will cost before approving it. Generated-by: Claude Opus 5 * Add inline script metadata to the CI-minutes calibration script The script uses only the standard library, so with a PEP 723 block it runs with plain `uv run dev/calibrate_ci_minutes_estimates.py` instead of syncing the whole Breeze environment. Document that invocation in the script and in .github/AGENTS.md. * Keep one PR's API or git error from aborting the approval tick Each PR is now processed in its own try block. A requests.RequestException or subprocess.CalledProcessError for one PR no longer stops the remaining PRs or skips the job summary. The failed PR is recorded with failed=True and shown as "failed" in the summary, keeping its CI-time estimate when one was computed and giving a short, single-line form of the error as the reason. The workflow also tries to add the 'need workflow approval' label to it, so it is not lost once its runs leave the query window. Nothing is approved on failure. approve_workflow_runs exits 1 after writing the summary when any PR failed, so a persistent failure turns the scheduled run red instead of passing silently. * Approve the runs of fork CI pull requests unless they change CI A pull request whose CI runs in the contributor's fork (AIP-120) still gets pull_request runs in apache/airflow, and GitHub holds them until a maintainer approves them. Those runs only execute the fork CI gate, so judging them by the cost and size of project CI left cheap gate-only runs waiting for a maintainer for no benefit, and a gated run that never ran could not be cleaned up by the reconciler. What still needs a maintainer is a change to the files that define CI, since it could remove the gate or add a workflow that does run. Using the reconciler's own list keeps the two decisions consistent, and the author is told why their runs wait instead of only seeing a label. Generated-by: Claude Opus 5 * Keep CI-minutes estimates true to the runs they approve A run selects its tests from the labels the PR had when it started, so approving it on the current labels could start a full-tests run whose label was removed while it waited. Calibrating from self-hosted and all-versions runs, and leaving out the jobs that need no CI image, made the estimate describe runs other than those it is used to approve. Generated-by: Claude Opus 5 --------- Co-authored-by: Shahar Epstein <[email protected]>
1 parent 47296ef commit 2a3ee7f

25 files changed

Lines changed: 3332 additions & 114 deletions

‎.github/AGENTS.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
<!-- SPDX-License-Identifier: Apache-2.0
2+
https://www.apache.org/licenses/LICENSE-2.0 -->
3+
4+
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
5+
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
6+
**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
7+
8+
- [The `.github` folder — Agent Instructions](#the-github-folder--agent-instructions)
9+
- [Keep the CI-minutes estimates in sync with the workflows](#keep-the-ci-minutes-estimates-in-sync-with-the-workflows)
10+
11+
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
12+
13+
# The `.github` folder — Agent Instructions
14+
15+
## Keep the CI-minutes estimates in sync with the workflows
16+
17+
`breeze ci approve-workflow-runs` (run by `workflows/auto-approve-workflow-runs.yml`) decides whether
18+
to approve an external contributor's CI runs from an estimate of the job-minutes they take. The
19+
numbers live in `dev/breeze/src/airflow_breeze/utils/ci_minutes_estimate.py` and are medians
20+
measured on recent `Tests (AMD)` PR runs on public runners.
21+
22+
After you change the CI workflows in a way that adds, removes, splits or resizes jobs (new test
23+
groups, a changed matrix, new composite workflows, a move between runner types), refresh the
24+
estimates once the change has run on enough PRs, and in any case about once a month:
25+
26+
1. Run `uv run dev/calibrate_ci_minutes_estimates.py --since <date>`
27+
from the repository root with an authenticated `gh` CLI. Pick a date that covers at least a
28+
week of runs after the workflow change.
29+
2. Copy the medians and fits it prints into the constants of `ci_minutes_estimate.py`, and update
30+
the "Last calibrated" line next to them.
31+
3. When you add a new job group to the workflows, add a matching group to both
32+
`ci_minutes_estimate.py` (with the selective-check output that turns it on) and the
33+
`JOB_GROUPS` patterns of the calibration script. Unmatched jobs fall into the base group.
34+
4. Run `uv run --project dev/breeze pytest dev/breeze/tests/test_ci_minutes_estimate.py
35+
dev/breeze/tests/test_workflow_run_approval.py -xvs`.
36+
37+
The estimates cover public runners only. Self-hosted (RunsOn) runners need their own set once there
38+
is enough data from them.
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
#
18+
---
19+
# Approves the waiting workflow runs of external contributors' PRs to main when the change is safe to
20+
# run without a maintainer's look, and labels the others "need workflow approval". A PR whose CI runs
21+
# in the contributor's fork (AIP-120) is approved unless it changes CI definition files, and then gets
22+
# a comment saying why its runs wait. The rules are in
23+
# dev/breeze/src/airflow_breeze/utils/workflow_run_approval.py and documented in
24+
# contributing-docs/05_pull_requests.rst.
25+
#
26+
# This workflow runs only trusted code from main. It reads the PR's metadata, changed-file list and
27+
# git objects, and never checks out or runs the PR's code. Approving with GITHUB_TOKEN starts the
28+
# approved runs right away, and workflow_run workflows that follow them still trigger.
29+
name: Auto approve workflow runs
30+
on: # yamllint disable-line rule:truthy
31+
schedule:
32+
- cron: '*/30 * * * *'
33+
workflow_dispatch:
34+
inputs:
35+
report-only:
36+
description: "Only report the decisions and estimates, without approving or labelling"
37+
type: boolean
38+
default: false
39+
permissions:
40+
# Approving workflow runs.
41+
actions: write
42+
# Adding and removing the "need workflow approval" label, and commenting on a fork CI PR whose
43+
# CI change keeps its runs waiting.
44+
pull-requests: write
45+
contents: read
46+
concurrency:
47+
group: auto-approve-workflow-runs
48+
cancel-in-progress: false
49+
jobs:
50+
approve-workflow-runs:
51+
name: "Approve workflow runs of external contributors"
52+
runs-on: ubuntu-latest
53+
if: github.repository == 'apache/airflow'
54+
timeout-minutes: 20
55+
steps:
56+
- name: "Checkout main"
57+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
58+
with:
59+
persist-credentials: false
60+
ref: main
61+
- name: "Install Breeze"
62+
uses: ./.github/actions/breeze
63+
- name: "Approve workflow runs"
64+
env:
65+
GITHUB_TOKEN: ${{ github.token }}
66+
GITHUB_REPOSITORY: ${{ github.repository }}
67+
REPORT_ONLY: ${{ inputs.report-only && 'true' || 'false' }}
68+
run: breeze ci approve-workflow-runs

‎contributing-docs/05_pull_requests.rst‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,47 @@ is the PMC private list — see the
242242
``[email protected]``. That document also covers the further steps the project may
243243
take when the same behaviour persists across PRs.
244244

245+
CI workflow runs of external contributors
246+
-----------------------------------------
247+
248+
GitHub does not start the CI workflows of a PR from a fork until a maintainer approves them,
249+
for the contributors that the repository's approval setting covers. To keep you from waiting
250+
for a maintainer on every push, a scheduled workflow checks the waiting runs every 30 minutes
251+
and approves them automatically when all of these hold:
252+
253+
* The PR targets ``main``. PRs to release branches, such as ``v3-4-test``, always wait for a
254+
maintainer.
255+
* The PR does not change anything under ``.github/``, ``dev/breeze/`` or ``scripts/ci/``. Those
256+
files define the CI workflows and choose the runners they use.
257+
* The change does not need the full set of tests, does not change dependencies (``pyproject.toml``,
258+
lock files, ``package.json``, ``go.mod``, Gradle build files) and does not add a new provider.
259+
* The CI jobs your change selects are estimated to take at most 310 job-minutes, about a quarter of
260+
a full CI run. Changes to documentation, to the UI, to the Helm chart, to the language SDKs, to
261+
one area of Airflow core or to a small provider usually stay under it. Changes to large providers
262+
such as Amazon or Google, to several providers, or to the Task SDK (which runs the tests of all
263+
providers) usually do not.
264+
* The PR changes at most 100 lines of source code. Tests (including end-to-end tests),
265+
documentation, newsfragments, translations and lock files are not counted, so a fix with
266+
thorough tests still qualifies. Larger changes are usually new features, and a maintainer looks
267+
at them first.
268+
269+
If your PR's CI runs in your fork (it carries the ``fork ci required`` label), the CI run in
270+
``apache/airflow`` only checks that, so none of the rules above apply. Its runs are approved
271+
automatically unless the PR changes files that define CI: anything under ``.github/``,
272+
``dev/breeze/``, ``scripts/ci/``, ``scripts/docker/`` or ``scripts/in_container/``, a ``Dockerfile``
273+
or ``.pre-commit-config.yaml``. Then the runs wait for a maintainer and a comment on the PR explains
274+
why. CI in your fork runs either way.
275+
276+
Each new push is checked again, because approval applies to one commit. Only runs started in the
277+
last 24 hours are considered, so older waiting runs stay with maintainers. If your PR does not
278+
qualify, it gets the ``need workflow approval`` label and a maintainer approves the runs after
279+
looking at the change. That can take a while, so please do not ping maintainers about it; it is
280+
not a judgement of your PR. The label is removed when a later commit is approved automatically.
281+
282+
Maintainers can see each decision, the CI jobs selective checks expect to run and their estimated
283+
time in the summary of the ``Auto approve workflow runs`` workflow. The rules are described in
284+
more detail in `CI workflows <../dev/breeze/doc/ci/05_workflows.md>`__.
285+
245286
Gen-AI Assisted contributions
246287
-----------------------------
247288

‎contributing-docs/25_maintainer_pr_triage.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -238,6 +238,7 @@ The triage workflow uses the following labels and states:
238238
| Label / State | Meaning |
239239
|---|---|
240240
| `ready for maintainer review` | PR has passed Stage 1 and is queued for human review. Applied only after the skill has verified that no workflow runs are awaiting approval. |
241+
| `need workflow approval` | The PR's CI workflow runs wait for a maintainer, because the `Auto approve workflow runs` workflow did not approve them: a PR to a branch other than `main`, a change to CI definition files, a full-tests or dependency change, a change whose CI jobs are estimated at more than 310 job-minutes, or a change of more than 100 lines of source code (see [Pull Requests](05_pull_requests.rst)). For a PR whose CI runs in the contributor's fork, only a change to CI definition files keeps its runs waiting, and the workflow comments on the PR to say so. The workflow also tries to add the label when processing the PR failed; its run then fails too. Approve the runs after looking at the diff. The workflow removes the label when a later commit is approved automatically. |
241242
| `closed because of multiple quality violations` | PR was closed because the author has multiple open PRs with quality issues. |
242243
| `suspicious changes detected` | PR (and all open PRs by the same author) was closed because the diff contained suspicious patterns (secret exfiltration, malicious CI modifications, etc.). |
243244
| `closed because of open PR limit` | PR was closed in the one-time closure that introduced the [open PR limit](32_open_pull_request_limit.rst). Applied by `dev/close_prs_over_open_pr_limit.py`, never by the triage skill. If the author reopens the PR, triage it like any other. |

‎dev/breeze/doc/08_ci_tasks.rst‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,26 @@ These are all available flags of ``fork-ci-reconcile`` command:
200200
:width: 100%
201201
:alt: Breeze ci fork-ci-reconcile
202202

203+
Approving workflow runs of external contributors
204+
------------------------------------------------
205+
206+
The ``auto-approve-workflow-runs`` workflow runs ``approve-workflow-runs`` every 30 minutes. It approves
207+
the waiting workflow runs of external contributors' PRs to ``main`` whose CI jobs are estimated at no more
208+
than 310 job-minutes, that change at most 100 lines of source code and that do not touch CI definition files, and labels the other PRs ``need workflow approval``. PRs whose CI runs in
209+
the contributor's fork (AIP-120) only run the fork CI gate in ``apache/airflow``, so their runs are
210+
approved whatever their size unless they change CI definition files; such a PR also gets a comment
211+
explaining that its CI change needs a maintainer. For every PR it also
212+
estimates the CI jobs selective checks will run and how long they take, and writes them to the job
213+
summary. Use ``--report-only`` to see the decisions without approving runs or changing labels. The rules
214+
are described in `CI workflows <ci/05_workflows.md>`_.
215+
216+
These are all available flags of ``approve-workflow-runs`` command:
217+
218+
.. image:: ./images/output_ci_approve-workflow-runs.svg
219+
:target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci_approve-workflow-runs.svg
220+
:width: 100%
221+
:alt: Breeze ci approve-workflow-runs
222+
203223
-----
204224

205225
Next step: Follow the `Release management tasks <09_release_management_tasks.rst>`_ guide to learn how

‎dev/breeze/doc/ci/05_workflows.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,68 @@ whose CI runs in the contributor's fork (AIP-120), the gate skips `build-info` a
9898
everything after it, so the run finishes in seconds; see
9999
[Fork CI for external contributors](08_fork_ci_for_external_contributors.md).
100100

101+
### Approval of pull request runs
102+
103+
GitHub holds the runs of a fork PR in `action_required` until someone with write access approves
104+
them, for the contributors the repository's approval setting covers. The
105+
`auto-approve-workflow-runs.yml` workflow runs every 30 minutes on `main` and calls
106+
`breeze ci approve-workflow-runs`, which approves the waiting runs of a commit when all of these hold:
107+
108+
- the PR targets `main`; PRs to release branches (`v3-4-test`, `airflow-ctl/v0-1-test`, ...) run those
109+
branches' own workflows and always wait for a maintainer
110+
- the PR changes nothing under `.github/`, `dev/breeze/` or `scripts/ci/`, which define the workflows
111+
and choose the runners
112+
- the PR changes no dependency manifest or lock file (`pyproject.toml`, `uv.lock`, `package.json`,
113+
`pnpm-lock.yaml`, `go.mod`, Gradle build files, ...) and adds no new provider (selective checks
114+
run from `main`'s tree and do not know it)
115+
- selective checks for the PR's merge commit succeed, and do not need the full set of tests, all Python
116+
and database versions, a canary build or upgraded dependencies (the estimate does not model them)
117+
- the jobs selective checks select are estimated at no more than 310 job-minutes on public runners
118+
(about 23% of a full-tests run, just above the cluster of single core areas and the Helm chart)
119+
- the PR changes at most 100 lines of source code; tests (including e2e tests), docs, newsfragments,
120+
translations, lock files and images are not counted
121+
122+
A PR to `main` whose CI runs in the contributor's fork (AIP-120: the `Fork CI gate` job would skip
123+
project CI for it) is decided by one rule instead, because its `Tests (AMD)` run only executes the
124+
gate: its runs are approved whatever its size, cost or dependency changes, unless it changes a file
125+
the fork CI reconciler counts as CI-defining (`.github/`, `dev/breeze/`, `scripts/ci/`,
126+
`scripts/docker/`, `scripts/in_container/`, `Dockerfile*`, `.pre-commit-config.yaml`). Such a PR gets
127+
the `need workflow approval` label and one comment telling the author that the CI change needs a
128+
maintainer; see
129+
[Approving gated runs](08_fork_ci_for_external_contributors.md#approving-gated-runs).
130+
131+
Only runs created in the last 24 hours (`--max-run-age-hours`) are considered, so the workflow never
132+
approves a backlog of older waiting runs at once; those stay with maintainers. A commit that is the
133+
head of several open PRs is never approved automatically, because its runs cannot be told apart by
134+
PR, and only runs started after the PR was opened and after its base branch last changed are
135+
approved (retargeting a PR does not start new runs). A run selects its jobs from the labels the PR
136+
had when it started, while the decision uses the current labels, so for PRs that run project CI a
137+
run started before a label that changes which tests run was added or removed
138+
(`full tests needed`, `all versions`, `default versions only`, `latest versions only`, `canary`,
139+
`upgrade to newer dependencies`, `area:e2e-tests`, `area:kubernetes-tests`) is left for a
140+
maintainer, since it still runs the selection it started with; the next push is decided afresh.
141+
For a PR whose CI runs in the fork the selection does not matter, but the gate falls back to the
142+
labels a run started with when it cannot read the current ones, so a run started before
143+
`use fork ci` or `use project ci` was added or removed is left for a maintainer instead. Other PRs get the
144+
`need workflow approval` label and wait for a maintainer. When processing a PR fails (for example on
145+
a GitHub API error), the workflow tries to add the label to it and the workflow run fails, so the
146+
failure stays visible even if the label could not be added. The label is removed when a later commit
147+
of the PR is approved automatically.
148+
149+
The workflow uses only `main`'s code. It reads the PR's metadata and fetches its merge commit as git
150+
objects to list the changed files; it never checks out or runs the PR's code. The merge commit must
151+
have been built from the head commit whose runs get approved, so a push after the check cannot slip
152+
into an approved run. Approving with the workflow's `GITHUB_TOKEN` starts the runs right away, and
153+
`workflow_run` workflows that follow them still trigger.
154+
155+
The estimate comes from `dev/breeze/src/airflow_breeze/utils/ci_minutes_estimate.py`, which turns the
156+
selective-check outputs into job groups with job-minutes and a wall-clock estimate, weighting
157+
providers by the size of their test suites. The job summary lists every waiting PR with its decision,
158+
the expected job groups and the estimated time. The numbers are medians of recent `Tests (AMD)` PR
159+
runs on public runners; refresh them with `dev/calibrate_ci_minutes_estimates.py`, as
160+
`.github/AGENTS.md` describes. Run `breeze ci approve-workflow-runs --report-only` (or dispatch the
161+
workflow with `report-only`) to see the decisions without approving or labelling anything.
162+
101163
## Fork push run
102164

103165
These runs are triggered by a push to any branch of a contributor's fork in which the

0 commit comments

Comments
 (0)