Skip to content

Commit bad9f9c

Browse files
authored
Add the fork CI reconciler and its scheduled monitor (#74822)
* Add the fork CI reconciler and its scheduled monitor Gated pull requests need labels, draft state, a state comment, a commit status that links to the fork run, and closing after inactivity or gate tampering, and none of that may be written in reaction to an untrusted pull_request event. A scheduled, idempotent reconciler that derives every change from what GitHub recorded is the only writer, so missed or repeated runs are harmless and nothing runs pull request code with write permissions. Whatever it cannot classify is reported to maintainers instead of being guessed. The monitor needs the breeze setup action, which frees disk space with sudo and docker, so it runs on ubuntu-latest rather than ubuntu-slim. Generated-by: Claude Opus 5 * Report CI passed from fork CI and address reconciler review The merge ruleset requires a CI passed check, which gated pull requests never get from project CI. The reconciler now posts a CI passed commit status when the head commit's fork CI is green and trusted, and overwrites it with pending when the commit leaves green, for example after a failed re-run in the fork. Also from review: drop the collaborator-permission lookup the monitor's token cannot make; stop posting green_unverified to Slack, since the needs project ci label lists those pull requests; judge only completed runs and finished jobs for gate tampering, so a job waiting for a runner does not look like project CI work; read compare files from the first page only; let gh's partial GraphQL output through when it exits non-zero; and warn in the state comment when a green branch is far behind its base. Generated-by: Claude Opus 5 * Do not count jobs every gated run starts as gate tampering Since #75420 every ci-amd.yml run starts check-run-needed, and once the CI passed gate of the auto-merge ruleset is in, a gated run also starts it as "CI passed (project CI not run)". The tamper detector counted any started job other than the gate as project CI work, so it would have closed every gated pull request. Those two jobs are now expected, under the same 5-minute limit as the gate. Generated-by: Claude Opus 5 * Count only jobs a runner picked up as fork CI gate tampering A run cancelled by a newer push before its gate started marks every job completed, so the tampering check read the never-started jobs as project CI work and could close a gated pull request that edits workflows legitimately. A job now counts only when it has a runner or steps, and "CI passed", which such a run reports under its plain name, is bounded like the gate. Statuses are read from the head commit when the pull request's last listed commit is another one, and the docs no longer claim a backlog reset never closes a pull request that changes workflows. Generated-by: Claude Opus 5
1 parent 164a812 commit bad9f9c

21 files changed

Lines changed: 3943 additions & 56 deletions
Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
#
18+
---
19+
# Drives pull requests whose CI runs in contributor forks (AIP-120): labels, draft state, the
20+
# state comment, the `Fork CI / Tests (AMD)` commit status and closing. It is the only writer for
21+
# those pull requests and never checks out or runs pull request code. See
22+
# dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md.
23+
name: "Fork CI monitor"
24+
on: # yamllint disable-line rule:truthy
25+
schedule:
26+
- cron: '*/15 * * * *'
27+
workflow_dispatch:
28+
inputs:
29+
dry-run:
30+
description: "Compute the changes without applying them to pull requests"
31+
type: boolean
32+
default: false
33+
permissions:
34+
# Listing the runs and jobs of the gate tampering check; unlisted scopes are none.
35+
actions: read
36+
contents: read
37+
# Labels, draft state, comments and closing of gated pull requests.
38+
pull-requests: write
39+
# The `Fork CI / Tests (AMD)` commit status on pull request head commits.
40+
statuses: write
41+
concurrency:
42+
group: fork-ci-monitor
43+
cancel-in-progress: false
44+
45+
jobs:
46+
reconcile:
47+
name: "Reconcile pull requests gated to fork CI"
48+
if: github.repository == 'apache/airflow'
49+
# The breeze action frees disk space with sudo and docker, which ubuntu-slim does not provide.
50+
runs-on: ubuntu-latest
51+
timeout-minutes: 20
52+
steps:
53+
- name: "Checkout main"
54+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
55+
with:
56+
ref: main
57+
persist-credentials: false
58+
- name: "Install Breeze"
59+
uses: ./.github/actions/breeze
60+
- name: "Reconcile"
61+
id: reconcile
62+
env:
63+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
64+
DRY_RUN: ${{ inputs.dry-run && 'true' || 'false' }}
65+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
66+
run: breeze ci fork-ci-reconcile --output-file slack-message.json
67+
# A dry run reports what it would have done, which must not read as done on Slack.
68+
- name: "Post to Slack"
69+
if: steps.reconcile.outputs.has-alerts == 'true' && !inputs.dry-run
70+
uses: slackapi/slack-github-action@dcb1066f776dd043e64d0e8ba94ca15cc7e1875d # v4.0.0
71+
with:
72+
method: chat.postMessage
73+
token: ${{ secrets.SLACK_BOT_TOKEN }}
74+
payload-file-path: "slack-message.json"
75+
# A run that fails before writing its message, for example on a discovery query or a rate
76+
# limit it cannot wait out, would otherwise stay silent.
77+
- name: "Report a failed run to Slack"
78+
if: failure() && !inputs.dry-run
79+
uses: slackapi/slack-github-action@dcb1066f776dd043e64d0e8ba94ca15cc7e1875d # v4.0.0
80+
with:
81+
method: chat.postMessage
82+
token: ${{ secrets.SLACK_BOT_TOKEN }}
83+
payload: |
84+
channel: "internal-airflow-ci-cd"
85+
text: >-
86+
Fork CI monitor run failed:
87+
${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

‎dev/breeze/doc/08_ci_tasks.rst‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,26 @@ If it cannot determine which milestone should be added, it also adds a comment t
161161
:width: 100%
162162
:alt: Milestone Tag Assistant
163163

164+
Reconciling pull requests gated to fork CI
165+
------------------------------------------
166+
167+
Pull requests from external contributors can run their CI in the contributor's fork instead of in
168+
``apache/airflow`` (AIP-120). The ``fork-ci-reconcile`` command, run every 15 minutes by the
169+
``fork-ci-monitor.yml`` workflow, is the only place that changes those pull requests: it labels newly
170+
gated pull requests ``fork ci required``, keeps them in draft until the fork run is green, maintains
171+
one state comment and a ``Fork CI / Tests (AMD)`` commit status linking to the run in the fork, closes
172+
pull requests without fork CI activity for a week, closes gated pull requests that made project CI run
173+
by changing workflows, and writes everything it could not classify to a Slack message. With
174+
``--dry-run`` it prints the changes instead of applying them. See
175+
`Fork CI for external contributors <ci/08_fork_ci_for_external_contributors.md>`_ for the design.
176+
177+
These are all available flags of ``fork-ci-reconcile`` command:
178+
179+
.. image:: ./images/output_ci_fork-ci-reconcile.svg
180+
:target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci_fork-ci-reconcile.svg
181+
:width: 100%
182+
:alt: Breeze ci fork-ci-reconcile
183+
164184
-----
165185

166186
Next step: Follow the `Release management tasks <09_release_management_tasks.rst>`_ guide to learn how

‎dev/breeze/doc/adr/0020-run-ci-for-external-contributions-in-forks.md‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -97,11 +97,16 @@ record captures the decisions that shape it and the alternatives rejected.
9797
the pull request's tree or only its head commit, so a later push that restores the gate
9898
does not hide it: a run did project CI work (its jobs show more than the gate, or it is a
9999
workflow not on the base branch) *and* that run's own commit changes files under
100-
``.github/workflows/``. Both are required so that legitimate project CI runs (backlog
101-
reset, a removed ``use project ci``) and legitimate workflow edits that keep the gate
102-
intact are never closed. A green fork run is trusted only when the pull request does not
100+
``.github/workflows/``. Both are required so that pull requests with legitimate project CI
101+
runs (backlog reset, a removed ``use project ci``) that do not change
102+
``.github/workflows/``, and legitimate workflow edits that keep the gate intact, are never
103+
closed. A pull request that is gated again after legitimate project CI runs and also edits
104+
workflows can still be closed while those runs are in the tampering window. A green fork run is trusted only when the pull request does not
103105
change what fork CI runs (``.github/``, ``dev/breeze/``, ``scripts/ci/`` and the like);
104-
otherwise it stays a draft until a maintainer runs project CI. Commit statuses were chosen over check
106+
otherwise it stays a draft until a maintainer runs project CI. A trusted green fork run also
107+
gets a ``CI passed`` commit status, the check the merge ruleset requires, because a gated
108+
pull request never runs project CI to report it; it is only ever posted as success, and
109+
overwritten when the head commit is no longer green. Commit statuses were chosen over check
105110
runs because a check run created with the workflow token links to the creating workflow
106111
instead of the fork run. ``pull_request_target`` was rejected: it would react to
107112
untrusted events with write permissions for a gain of a few minutes of latency.
@@ -111,7 +116,8 @@ record captures the decisions that shape it and the alternatives rejected.
111116
``internal-airflow-ci-cd`` Slack channel through the same action and payload convention
112117
as the CI duration monitor. Two expected outcomes are posted as well, in their own
113118
section: a pull request closed for gate tampering and a pull request carrying both
114-
override labels.
119+
override labels. A green fork run that cannot be trusted is not posted: the
120+
``needs project ci`` label lists those pull requests for maintainers.
115121

116122
6. **``breeze ci audit`` is the contributor's pre-flight.** It checks ``gh``, the
117123
``upstream``/``origin`` remote convention, fork existence, who the user is to the project,

0 commit comments

Comments
 (0)