Skip to content

Commit 5edef09

Browse files
zozo123claude
andcommitted
Keep dependency reports going when one input is gone
- Skip entries whose saved constraints or CI image are missing or expired, with a warning and a summary line, instead of failing every report. - Name report inputs by canary attempt and explain each input once, so a re-run canary no longer queues a second report from the same inputs. - Post the same notice on every attempt when the canary saved no inputs. - Free up disk space and move Docker to /mnt before loading the image. - Save the checked constraints through breeze's --save-constraints-to, pinned to the resolved constraints commit; failing to save only warns, so the canary no longer depends on raw `gh api` calls. - Say that no cache step, not workflow_run, keeps the cache untouched, and skip the plan tests when node is unavailable. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 6648f4f commit 5edef09

12 files changed

Lines changed: 502 additions & 116 deletions

‎.github/workflows/dependency-upgrade-report.yml‎

Lines changed: 54 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,9 @@ name: Dependency upgrade report
2727
permissions:
2828
contents: read
2929
actions: read
30+
# Every canary attempt fires this workflow and gets its own report of the inputs it saved.
3031
concurrency:
31-
group: dependency-report-${{ github.event.workflow_run.id }}
32+
group: dependency-report-${{ github.event.workflow_run.id }}-${{ github.event.workflow_run.run_attempt }}
3233
cancel-in-progress: false
3334
jobs:
3435
plan:
@@ -49,14 +50,19 @@ jobs:
4950
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
5051
env:
5152
SOURCE_RUN_ID: ${{ github.event.workflow_run.id }}
53+
SOURCE_RUN_ATTEMPT: ${{ github.event.workflow_run.run_attempt }}
5254
with:
5355
script: |
5456
const rawId = process.env.SOURCE_RUN_ID;
5557
if (!/^[1-9][0-9]*$/.test(rawId)) throw new Error('Invalid source run ID');
5658
const run_id = Number(rawId);
5759
if (!Number.isSafeInteger(run_id)) throw new Error('Invalid source run ID');
60+
const rawAttempt = process.env.SOURCE_RUN_ATTEMPT;
61+
if (!/^[1-9][0-9]{0,3}$/.test(rawAttempt)) throw new Error('Invalid source run attempt');
62+
const attempt_number = Number(rawAttempt);
5863
const repo = context.repo;
59-
const { data: run } = await github.rest.actions.getWorkflowRun({ ...repo, run_id });
64+
const { data: run } = await github.rest.actions.getWorkflowRunAttempt(
65+
{ ...repo, run_id, attempt_number });
6066
if (run.head_repository.full_name !== `${repo.owner}/${repo.repo}` ||
6167
run.head_branch !== 'main' || run.path !== '.github/workflows/ci-amd.yml' ||
6268
!['schedule', 'workflow_dispatch'].includes(run.event) || run.status !== 'completed') {
@@ -65,15 +71,28 @@ jobs:
6571
const artifacts = await github.paginate(github.rest.actions.listWorkflowRunArtifacts,
6672
{ ...repo, run_id, per_page: 100 });
6773
const include = [];
74+
const skipped = [];
75+
const explained = [];
6876
const inputName = new RegExp('^dependency-report-inputs-(3\\.[0-9]+)-' +
69-
'(constraints(?:-source-providers|-no-providers)?)-(true|false)$');
70-
for (const input of artifacts) {
71-
const match = inputName.exec(input.name);
72-
if (!match) continue;
73-
const [, python, mode, useUv] = match;
77+
'(constraints(?:-source-providers|-no-providers)?)-(true|false)-([1-9][0-9]*)$');
78+
const saved = artifacts.map(a => [a, inputName.exec(a.name)]).filter(([, match]) => match);
79+
for (const [input, [, python, mode, useUv, attempt]] of saved) {
80+
if (attempt !== rawAttempt) continue;
81+
const entry = `${python}:${mode}`;
82+
// A re-run canary saves its inputs again; they are explained only for the first attempt.
83+
if (saved.some(([, m]) => m[1] === python && m[2] === mode && m[3] === useUv &&
84+
Number(m[4]) < attempt_number)) {
85+
explained.push(entry);
86+
continue;
87+
}
88+
// The stash action names the artifact after the stash key, with '/' as '_', and the branch.
7489
const image = artifacts.find(a => a.name === `ci-image-save-v3-linux_amd64-${python}-main`);
75-
if (input.expired || !image || image.expired) {
76-
throw new Error(`Original inputs or CI image unavailable for ${python}:${mode}`);
90+
const missing = input.expired ? 'inputs expired' :
91+
!image ? 'CI image missing' : image.expired ? 'CI image expired' : '';
92+
if (missing) {
93+
core.warning(`Skipping ${entry}: original inputs or CI image unavailable (${missing})`);
94+
skipped.push(`${entry} (${missing})`);
95+
continue;
7796
}
7897
include.push({ python, mode, 'use-uv': useUv,
7998
'inputs-id': input.id, 'image-id': image.id });
@@ -82,15 +101,19 @@ jobs:
82101
core.setOutput('has-reports', include.length > 0 ? 'true' : 'false');
83102
core.setOutput('source-sha', run.head_sha);
84103
core.setOutput('source-run-id', rawId);
85-
if (!include.length) {
86-
if (Number(process.env.GITHUB_RUN_ATTEMPT) > 1) {
87-
throw new Error('Saved report inputs are missing or expired');
88-
}
89-
core.notice('No saved report inputs: this run did not request independent diagnostics');
104+
if (!include.length && skipped.length && Number(process.env.GITHUB_RUN_ATTEMPT) > 1) {
105+
throw new Error('Saved report inputs are missing or expired');
106+
}
107+
if (!include.length && !skipped.length) {
108+
core.notice(explained.length ?
109+
'The saved report inputs were already explained for an earlier canary attempt' :
110+
'No saved report inputs: this run did not request independent diagnostics');
90111
}
91112
await core.summary.addHeading('Dependency upgrade diagnostics')
92-
.addLink('Source canary run', run.html_url)
113+
.addLink(`Source canary run (attempt ${rawAttempt})`, run.html_url)
93114
.addRaw(`\n\nSource revision: ${run.head_sha}\n\nReports: ${include.length}\n`)
115+
.addRaw(skipped.length ? `\nSkipped: ${skipped.join(', ')}\n` : '')
116+
.addRaw(explained.length ? `\nExplained for an earlier attempt: ${explained.join(', ')}\n` : '')
94117
.write();
95118
96119
explain:
@@ -115,12 +138,18 @@ jobs:
115138
with:
116139
ref: ${{ needs.plan.outputs.source-sha }}
117140
persist-credentials: false
118-
- name: "Install Breeze"
119-
uses: ./.github/actions/breeze
141+
# Docker moves to /mnt as in the image build jobs: the loaded image and uv resolutions outgrow /.
142+
- name: "Free up disk space"
143+
shell: bash
144+
run: ./scripts/tools/free_up_disk_space.sh
120145
- name: "Make /mnt writeable"
121146
run: ./scripts/ci/make_mnt_writeable.sh
147+
- name: "Move docker to /mnt"
148+
run: ./scripts/ci/move_docker_to_mnt.sh
149+
- name: "Install Breeze"
150+
uses: ./.github/actions/breeze
122151
- name: "Download the original CI image"
123-
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
152+
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
124153
with:
125154
artifact-ids: ${{ matrix.image-id }}
126155
run-id: ${{ needs.plan.outputs.source-run-id }}
@@ -132,7 +161,7 @@ jobs:
132161
PYTHON_VERSION: ${{ matrix.python }}
133162
run: breeze ci-image load --platform linux/amd64 --python "${PYTHON_VERSION}" --image-file-dir /mnt
134163
- name: "Download the constraints used by the quick summary"
135-
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
164+
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
136165
with:
137166
artifact-ids: ${{ matrix.inputs-id }}
138167
run-id: ${{ needs.plan.outputs.source-run-id }}
@@ -145,20 +174,24 @@ jobs:
145174
PYTHON_VERSION: ${{ matrix.python }}
146175
CONSTRAINTS_MODE: ${{ matrix.mode }}
147176
SOURCE_RUN_ID: ${{ needs.plan.outputs.source-run-id }}
177+
SOURCE_SHA: ${{ needs.plan.outputs.source-sha }}
148178
run: |
149179
{
150180
echo "## Dependency upgrade explanations"
151-
echo "Source canary run: ${SOURCE_RUN_ID}"
181+
echo "Source canary run: ${SOURCE_RUN_ID}; source revision: ${SOURCE_SHA}"
152182
echo "Constraints commit: $(cat files/dependency-report/constraints-commit.txt)"
153183
echo "Python: ${PYTHON_VERSION}; mode: ${CONSTRAINTS_MODE}"
184+
echo "The constraints are the constraints branch the canary's summary checked, not ones built"
185+
echo "from the source revision: dependency bounds changed between the two can show up below as"
186+
echo "conflicts or as stale constraints."
154187
} >> "${GITHUB_STEP_SUMMARY}"
155188
breeze release-management constraints-version-check \
156189
--python "${PYTHON_VERSION}" --airflow-constraints-mode "${CONSTRAINTS_MODE}" \
157190
--constraints-file files/dependency-report/constraints.txt --explain-why \
158191
2>&1 | tee files/dependency-report/explanations.txt
159192
- name: "Save diagnostic output, including partial output on failure"
160193
if: always()
161-
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
194+
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
162195
with:
163196
name: dependency-explanations-${{ matrix.python }}-${{ matrix.mode }}
164197
path: files/dependency-report/

‎.github/workflows/finalize-tests.yml‎

Lines changed: 21 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -149,55 +149,37 @@ jobs:
149149
python: ${{ matrix.python-version }}
150150
use-uv: ${{ inputs.use-uv }}
151151
make-mnt-writeable-and-cleanup: true
152-
- name: "Save constraints for the independent dependency report"
153-
shell: bash
154-
if: >-
155-
github.ref == 'refs/heads/main' &&
156-
(github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')
157-
env:
158-
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
159-
CONSTRAINTS_BRANCH: ${{ inputs.constraints-branch }}
160-
PYTHON_VERSION: ${{ matrix.python-version }}
161-
CONSTRAINTS_MODE: ${{ matrix.constraints-mode }}
162-
run: |
163-
mkdir -p files/dependency-report
164-
REF_API="repos/apache/airflow/git/ref/heads/${CONSTRAINTS_BRANCH}"
165-
CONSTRAINTS_SHA=$(gh api "${REF_API}" --jq '.object.sha')
166-
FILE_API="repos/apache/airflow/contents/${CONSTRAINTS_MODE}-${PYTHON_VERSION}.txt"
167-
gh api "${FILE_API}?ref=${CONSTRAINTS_SHA}" \
168-
--jq '.content' | base64 --decode > files/dependency-report/constraints.txt
169-
echo "${CONSTRAINTS_SHA}" > files/dependency-report/constraints-commit.txt
170-
- name: "Upload dependency report inputs"
171-
if: >-
172-
github.ref == 'refs/heads/main' &&
173-
(github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')
174-
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
175-
with:
176-
name: "dependency-report-inputs-${{ matrix.python-version }}-${{ matrix.constraints-mode }}\
177-
-${{ inputs.use-uv }}"
178-
path: files/dependency-report/
179-
if-no-files-found: error
180-
retention-days: 2
181-
overwrite: true
152+
# Scheduled and dispatched canaries on main leave the explanations to dependency-upgrade-report.yml
153+
# and save the constraints they checked for it. Saving them never fails this step.
182154
- name: "Deps: ${{ matrix.python-version }}:${{ matrix.constraints-mode }}"
183155
shell: bash
184156
run: |
185-
CONSTRAINTS_ARGS=()
186-
if [[ -f files/dependency-report/constraints.txt ]]; then
187-
CONSTRAINTS_ARGS=(--constraints-file files/dependency-report/constraints.txt)
157+
REPORT_ARGS=(--explain-why)
158+
if [[ "${INDEPENDENT_REPORT}" == "true" ]]; then
159+
REPORT_ARGS=(--no-explain-why --save-constraints-to files/dependency-report)
188160
fi
189161
breeze release-management constraints-version-check \
190162
--python "${MATRIX_PYTHON_VERSION}" \
191-
--airflow-constraints-mode "${MATRIX_CONSTRAINTS_MODE}" "${EXPLAIN_WHY_FLAG}" \
192-
"${CONSTRAINTS_ARGS[@]}"
163+
--airflow-constraints-mode "${MATRIX_CONSTRAINTS_MODE}" "${REPORT_ARGS[@]}"
193164
env:
194165
MATRIX_PYTHON_VERSION: "${{ matrix.python-version }}"
195166
MATRIX_CONSTRAINTS_MODE: "${{ matrix.constraints-mode }}"
196-
EXPLAIN_WHY_FLAG: >-
197-
${{ (github.ref == 'refs/heads/main' &&
198-
(github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'))
199-
&& '--no-explain-why' || '--explain-why' }}
167+
INDEPENDENT_REPORT: >-
168+
${{ github.ref == 'refs/heads/main' &&
169+
(github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') }}
200170
VERBOSE: "false"
171+
- name: "Upload dependency report inputs"
172+
if: >-
173+
!cancelled() && hashFiles('files/dependency-report/constraints-commit.txt') != ''
174+
continue-on-error: true
175+
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
176+
with:
177+
name: "dependency-report-inputs-${{ matrix.python-version }}-${{ matrix.constraints-mode }}\
178+
-${{ inputs.use-uv }}-${{ github.run_attempt }}"
179+
path: files/dependency-report/
180+
if-no-files-found: error
181+
retention-days: 2
182+
overwrite: true
201183

202184
push-buildx-cache-to-github-registry:
203185
name: Push Regular Image Cache ${{ inputs.platform }}

‎dev/breeze/doc/ci/05_workflows.md‎

Lines changed: 22 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -373,25 +373,39 @@ When adding a job, reach for `ubuntu-slim` if it only shuffles metadata around,
373373
Scheduled and manually dispatched AMD canaries on `main` keep the dependency freshness
374374
summary in `finalize-tests.yml`, but run it with `--no-explain-why`. Tests, constraint
375375
generation and validation, and image publication still run in the canary. Release-branch
376-
and other run types retain their existing explanation behavior.
376+
and other run types retain their existing explanation behavior. The summary saves the
377+
constraints it checked, and the constraints-branch commit they came from, with
378+
`--save-constraints-to`; a failure to save them only warns and never fails the canary.
377379

378380
After the canary completes, `dependency-upgrade-report.yml` runs the detailed explanations
379-
independently, including when tests failed but the report inputs were saved. It checks out
381+
independently, from the inputs the canary saved; a canary with failed jobs saves none. It checks out
380382
the original source SHA and downloads the original image and saved constraints by artifact
381-
ID from that run. It never substitutes the latest branch image or constraints. PyPI release
383+
ID from that run. It never substitutes the latest branch image or constraints: an entry whose
384+
saved constraints or image are missing or expired is skipped with a warning and listed in the
385+
summary, and the others are still explained. The image comes from the CI image stash, which
386+
keeps only the newest image per Python version and branch, so a later build on `main` can
387+
replace it within hours. Re-running the canary triggers the workflow again; inputs that an earlier attempt
388+
already saved are not explained again.
389+
390+
The saved constraints are the constraints branch as the summary saw it, not constraints
391+
generated from the source SHA, so dependency bounds that changed between the two can appear
392+
as conflicts or as stale constraints. The report summary shows both commits. PyPI release
382393
metadata is fetched when the report executes, so the available upgrade targets can change
383394
between the summary and the explanations.
384395

385-
The workflow has read-only repository permissions and runs only on `workflow_run`, which
386-
cannot write to the default branch's GitHub Actions cache. Reports use the saved image artifact.
396+
The workflow has read-only repository permissions. It runs on `workflow_run`, in the context
397+
of the default branch, so it could write that branch's GitHub Actions cache: what keeps the
398+
cache untouched is that no step saves one. Do not add caching (for example `actions/cache` or
399+
the cache option of a `setup-*` action) to this workflow. Reports use the saved image artifact.
387400

388401
The report runs at most three jobs concurrently. Each job has a 90-minute limit and saves
389402
its output, including partial output on failure. A failed or timed-out report is incomplete;
390403
it does not invalidate the canary's test result. Find diagnostics in the **Dependency
391404
upgrade report** workflow, whose summary links to the source canary. To repeat diagnostics,
392-
use **Re-run jobs** on that report before its two-day input/image retention expires (or run
393-
`gh run rerun REPORT_RUN_ID`). Reruns keep the original source canary ID. Missing or expired
394-
original inputs fail the rerun rather than using a newer image or constraints.
405+
use **Re-run jobs** on that report before its two-day input retention expires (or run
406+
`gh run rerun REPORT_RUN_ID`). Reruns keep the original source canary ID and attempt. A rerun
407+
fails when inputs were saved but none can be explained any more, rather than using a newer
408+
image or constraints; a report whose canary saved no inputs posts the same notice every time.
395409

396410
`breeze release-management constraints-version-check --constraints-file PATH` can also
397411
analyze a saved constraints file locally. Detailed explanations reuse one baseline resolution

0 commit comments

Comments
 (0)