Skip to content

Commit f92b7c3

Browse files
authored
Run full CI on release branches on a schedule instead of every push (#75420)
* Run full CI on release branches on a schedule instead of every push While a release is prepared, backports land on vX-Y-test in bursts and every push started a full AMD and ARM run. In the week of 15-21 October v3-4-test triggered 177 such runs; 133 of them were cancelled by the next push after already using hundreds of runner-minutes each, which made release-branch pushes one of the largest single consumers of the ASF's shared runners. Each backport PR has already run CI against the branch, so the branch needs a full run of its current HEAD a few times a day rather than one per merged commit. Release managers still need quick feedback on a broken backport, so a push still runs the tests for what changed since the last successful run of the branch. Testing from the last success rather than from the pushed commit alone means changes whose runs failed or were cancelled by a newer push are not left untested. Scheduled canaries that have nothing new to test - no commit since the last successful full run - only repeat that run, so they now finish without building anything. Generated-by: Claude Opus 5 * Address review: fail open, keep docs publishing on main, check full push range - check-run-needed lookups fail open (push: all tests, canary: runs), and notify-slack posts unless the canary was a no-op. - Docs publishing steps of a canary run only on main: release branches build a subset of the docs, which `aws s3 sync --delete` would publish over the docs bucket. release-branch-ci.yml skips branches this is not backported to. - Basic static checks of a push cover the files changed since the last tested commit, as its test selection does. - List pushed changes with --no-renames, as diff-tree does. - A dispatched full run upgrades to newer dependencies, as a scheduled one. Generated-by: Claude Opus 5
1 parent 24a8095 commit f92b7c3

20 files changed

Lines changed: 871 additions & 115 deletions

‎.github/workflows/basic-tests.yml‎

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,11 @@ on: # yamllint disable-line rule:truthy
7676
description: 'Platform for the build - linux/amd64 or linux/arm64'
7777
required: true
7878
type: string
79+
push-base-commit:
80+
description: "For a push: the last successfully tested commit of the branch (empty otherwise)"
81+
required: false
82+
type: string
83+
default: ""
7984
permissions:
8085
contents: read
8186
jobs:
@@ -274,11 +279,35 @@ jobs:
274279
ref: ${{ github.sha }}
275280
fetch-depth: 2
276281
persist-credentials: false
282+
- name: "Fetch the last tested push commit"
283+
if: inputs.push-base-commit != ''
284+
shell: bash
285+
env:
286+
PUSH_BASE_COMMIT: ${{ inputs.push-base-commit }}
287+
run: |
288+
git fetch --no-tags --depth=1 origin "${PUSH_BASE_COMMIT}" \
289+
|| echo "::warning::Could not fetch ${PUSH_BASE_COMMIT}, all files will be checked"
277290
- name: "Static checks: basic checks only"
278-
run: >
279-
prek --show-diff-on-failure --color always
280-
--from-ref "${{ github.sha }}"^ --to-ref "${{ github.sha }}"
291+
# A push is checked from the last tested commit of the branch, like its test selection: the
292+
# commits of earlier failed or cancelled push runs are checked again. `--from-ref` needs the
293+
# merge base, which the shallow fetch does not have, so the changed files are listed here.
294+
shell: bash
295+
run: |
296+
set -euo pipefail
297+
prek_args=(--show-diff-on-failure --color always)
298+
if [[ -z "${PUSH_BASE_COMMIT}" ]]; then
299+
prek "${prek_args[@]}" --from-ref "${GITHUB_SHA}^" --to-ref "${GITHUB_SHA}"
300+
elif git cat-file -e "${PUSH_BASE_COMMIT}^{commit}"; then
301+
mapfile -t files < <(git diff --name-only --no-renames --diff-filter=d \
302+
"${PUSH_BASE_COMMIT}" "${GITHUB_SHA}")
303+
if (( ${#files[@]} > 0 )); then
304+
prek "${prek_args[@]}" --files "${files[@]}"
305+
fi
306+
else
307+
prek "${prek_args[@]}" --all-files
308+
fi
281309
env:
310+
PUSH_BASE_COMMIT: ${{ inputs.push-base-commit }}
282311
VERBOSE: "false"
283312
SKIP_BREEZE_PREK_HOOKS: "true"
284313
SKIP: ${{ inputs.skip-prek-hooks }}

‎.github/workflows/ci-amd.yml‎

Lines changed: 111 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@
1616
# under the License.
1717
#
1818
---
19-
# AMD CI workflow. Triggered on every PR and on push to release branches.
20-
# Companion to `ci-arm.yml`, which runs the same pipeline on ARM as the
19+
# AMD CI workflow. Triggered on every PR, on the canary cron, and on release branches by
20+
# `release-branch-ci.yml`. Companion to `ci-arm.yml`, which runs the same pipeline on ARM as the
2121
# scheduled canary. The two workflows are physical copies of each other
2222
# (GitHub Actions has no cross-file YAML include), differing only in their
2323
# trigger surface, the hardcoded `runner-type` / `platform` values in
@@ -43,11 +43,10 @@ on: # yamllint disable-line rule:truthy
4343
- airflow-ctl/v[0-9]+-[0-9]+-stable
4444
types: [opened, reopened, synchronize, ready_for_review]
4545
push:
46-
# Post-merge pushes to release-prep / providers branches run on both
47-
# AMD and ARM (the matching block lives in the other wrapper too —
48-
# `check-ci-workflows-in-sync` enforces they stay identical). `main` is
49-
# intentionally NOT in this list: main pushes stay AMD-only via
50-
# `ci-amd.yml`; ARM main coverage is the cron-driven canary above.
46+
# Pushes to release-prep / providers branches run only the tests needed by the changes since the
47+
# last successful run of the branch (selected as for a PR). Full runs of these branches, on AMD
48+
# and ARM, are dispatched by `release-branch-ci.yml`. `main` does not build on push; its full
49+
# runs are the canary above.
5150
branches:
5251
- v[0-9]+-[0-9]+-test
5352
- providers-[a-z]+-?[a-z]*/v[0-9]+-[0-9]+
@@ -66,12 +65,97 @@ env:
6665
VERBOSE: "true"
6766

6867
concurrency:
69-
group: ci-amd-${{ github.event.pull_request.number || github.ref }}
70-
cancel-in-progress: true
68+
# Push runs have a group of their own per branch: a newer push cancels the older push run (the
69+
# newer run then also tests the cancelled run's changes, see `check-run-needed`), and a push never
70+
# cancels a full run of the branch.
71+
group: >-
72+
ci-amd-${{ github.event.pull_request.number || github.ref }}${{
73+
github.event_name == 'push' && '-push' || '' }}
74+
# A scheduled canary queues behind a run in progress instead of cancelling it: it may well be a
75+
# manual run of the same commit, which the canary would otherwise cancel and then skip itself.
76+
cancel-in-progress: ${{ github.event_name != 'schedule' }}
7177

7278
jobs:
79+
check-run-needed:
80+
name: "Check if the run is needed"
81+
# A scheduled canary is not needed when the last completed full run of this workflow on the
82+
# branch tested the same commit and succeeded: it would only repeat that run. A push run tests
83+
# the changes since the last successful run of this workflow on the branch, found here.
84+
# This job runs for every event, with no steps for the others: every other job depends on it
85+
# through `build-info`, and GitHub skips the whole dependency chain below a skipped job unless
86+
# each job in it has a status function such as `always()` in its `if:`.
87+
runs-on: ["ubuntu-slim"]
88+
timeout-minutes: 5
89+
permissions:
90+
actions: read
91+
contents: read
92+
outputs:
93+
run-needed: ${{ steps.schedule.outputs.run-needed || 'true' }}
94+
push-base-commit: ${{ steps.push.outputs.push-base-commit }}
95+
steps:
96+
- name: "Find the last successful run of the branch"
97+
if: github.event_name == 'push'
98+
id: push
99+
shell: bash
100+
env:
101+
GH_TOKEN: ${{ github.token }}
102+
REPOSITORY: ${{ github.repository }}
103+
WORKFLOW_REF: ${{ github.workflow_ref }}
104+
BRANCH: ${{ github.ref_name }}
105+
run: |
106+
set -euo pipefail
107+
workflow_path="${WORKFLOW_REF%@*}"
108+
# A successful push or full run tested everything up to its commit. Changes after it -- from
109+
# pushes whose runs failed, or were cancelled by a newer push -- are all tested by this run.
110+
# A successful run of this same commit is passed over, so that a re-run tests again.
111+
if ! base="$(gh api "repos/${REPOSITORY}/actions/workflows/${workflow_path##*/}/runs" -X GET \
112+
-f branch="${BRANCH}" -f status=success -F per_page=30 \
113+
--jq "[.workflow_runs[] | select((.event == \"push\" or .event == \"workflow_dispatch\")
114+
and .head_sha != \"${GITHUB_SHA}\")][0].head_sha // \"\"")"; then
115+
# Fail open: a failed lookup must not skip the run.
116+
echo "::warning::Could not find the last successful run of ${BRANCH}, all tests will run"
117+
base=""
118+
fi
119+
# With no successful run to build on, selective checks run all tests.
120+
echo "push-base-commit=${base}" >> "${GITHUB_OUTPUT}"
121+
echo "Testing the changes since ${base:-the start: no successful run yet, so all tests run}." \
122+
>> "${GITHUB_STEP_SUMMARY}"
123+
- name: "Check the last full run of ${{ github.sha }}"
124+
if: github.event_name == 'schedule'
125+
id: schedule
126+
shell: bash
127+
env:
128+
GH_TOKEN: ${{ github.token }}
129+
REPOSITORY: ${{ github.repository }}
130+
WORKFLOW_REF: ${{ github.workflow_ref }}
131+
BRANCH: ${{ github.ref_name }}
132+
run: |
133+
set -euo pipefail
134+
workflow_path="${WORKFLOW_REF%@*}"
135+
workflow="${workflow_path##*/}"
136+
# Full runs are the scheduled canaries and manual `workflow_dispatch` runs; this run is
137+
# still in progress, so `status=completed` leaves it out.
138+
if ! last="$(gh api \
139+
"repos/${REPOSITORY}/actions/workflows/${workflow}/runs?branch=${BRANCH}&status=completed" \
140+
-F per_page=20 -X GET \
141+
--jq '[.workflow_runs[] | select(.event == "schedule" or .event == "workflow_dispatch")][0]
142+
| "\(.head_sha // "") \(.conclusion // "") \(.html_url // "")"')"; then
143+
# Fail open: a canary that cannot check the last run runs, and reports to Slack.
144+
echo "::warning::Could not find the last full run of ${BRANCH}, running the canary"
145+
last=""
146+
fi
147+
read -r last_sha last_conclusion last_url <<< "${last}"
148+
if [[ "${last_sha}" == "${GITHUB_SHA}" && "${last_conclusion}" == "success" ]]; then
149+
echo "run-needed=false" >> "${GITHUB_OUTPUT}"
150+
echo "No new commits since the last successful full run: ${last_url}" >> "${GITHUB_STEP_SUMMARY}"
151+
else
152+
echo "run-needed=true" >> "${GITHUB_OUTPUT}"
153+
fi
154+
73155
build-info:
74156
name: "Build info"
157+
needs: [check-run-needed]
158+
if: needs.check-run-needed.outputs.run-needed == 'true'
75159
# At build-info stage we do not yet have outputs so we need to hard-code the runs-on to public runners
76160
runs-on: ["ubuntu-22.04"]
77161
outputs:
@@ -190,6 +274,16 @@ jobs:
190274
ref: ${{ github.sha }}
191275
fetch-depth: 2
192276
persist-credentials: false
277+
- name: "Fetch the last tested push commit"
278+
# Selective checks of a push diff against this commit. A two-commit diff only needs the
279+
# commit itself, not the history in between; if it cannot be fetched, all tests run.
280+
if: needs.check-run-needed.outputs.push-base-commit != ''
281+
shell: bash
282+
env:
283+
PUSH_BASE_COMMIT: ${{ needs.check-run-needed.outputs.push-base-commit }}
284+
run: |
285+
git fetch --no-tags --depth=1 origin "${PUSH_BASE_COMMIT}" \
286+
|| echo "::warning::Could not fetch ${PUSH_BASE_COMMIT}, all tests will run"
193287
- name: "Install Breeze"
194288
uses: ./.github/actions/breeze
195289
id: breeze
@@ -231,6 +325,8 @@ jobs:
231325
VERBOSE: "false"
232326
GITHUB_CONTEXT_INPUT: "${{ runner.temp }}/github_context.json"
233327
PLATFORM: "linux/amd64"
328+
SELECTIVE_TESTS_ON_PUSH: "true"
329+
PUSH_BASE_COMMIT: ${{ needs.check-run-needed.outputs.push-base-commit }}
234330
run: breeze ci selective-check 2>> ${GITHUB_OUTPUT}
235331
- name: env
236332
run: printenv
@@ -247,7 +343,7 @@ jobs:
247343

248344
basic-tests:
249345
name: "Basic tests"
250-
needs: [build-info]
346+
needs: [build-info, check-run-needed]
251347
uses: ./.github/workflows/basic-tests.yml
252348
with:
253349
runners: ${{ needs.build-info.outputs.runner-type }}
@@ -264,6 +360,7 @@ jobs:
264360
use-uv: ${{needs.build-info.outputs.use-uv}}
265361
platform: ${{ needs.build-info.outputs.platform }}
266362
shared-distributions-as-json: ${{needs.build-info.outputs.shared-distributions-as-json}}
363+
push-base-commit: ${{ needs.check-run-needed.outputs.push-base-commit }}
267364

268365
build-ci-images:
269366
name: Build CI images
@@ -1202,7 +1299,7 @@ jobs:
12021299
contents: write
12031300
packages: write
12041301
# This will fire when all the jobs from "needs" are either successful or skipped
1205-
if: always() && !failure() && !cancelled()
1302+
if: always() && !failure() && !cancelled() && needs.build-info.result == 'success'
12061303
needs:
12071304
- additional-ci-image-checks
12081305
- additional-prod-image-tests
@@ -1255,12 +1352,15 @@ jobs:
12551352
name: "Notify Slack"
12561353
needs:
12571354
- build-info
1355+
- check-run-needed
12581356
- finalize-tests
12591357
- tests-agent-frameworks
1358+
# A no-op canary (`run-needed` is 'false') does not post; a failed check does.
12601359
if: >-
12611360
always() &&
12621361
!cancelled() &&
12631362
github.event_name == 'schedule' &&
1363+
needs.check-run-needed.outputs.run-needed != 'false' &&
12641364
github.run_attempt == 1
12651365
runs-on: ["ubuntu-slim"]
12661366
steps:

0 commit comments

Comments
 (0)