Skip to content

ci(e2e): rebuild the image artifact when a retest cannot download it - #8094

Merged
yuluo-yx merged 1 commit into
apache:masterfrom
cloudbase-ai:ci/e2e-artifact-retest
Oct 24, 2026
Merged

yuluo-yx merged 1 commit into
apache:masterfrom
cloudbase-ai:ci/e2e-artifact-retest

Conversation

@cloudbase-ai

Copy link
Copy Markdown
Contributor

Problem

/retest retries a run with reRunWorkflowFailedJobs, which re-executes only the failed jobs. In the
e2e workflow the jobs that fail are e2e-case / e2e-storage, and both need the shenyu-images
artifact produced by build-docker-images, which is uploaded with retention-days: 1.

The producer job is success in the original run, so a failed-job rerun leaves it untouched and the
artifact is never rebuilt. Once it expires, every retry dies on

##[error]Unable to download artifact(s): Artifact not found for name: shenyu-images

Observed on run 39037187784 (PR #7766): run_attempt=4, created_at=2026-10-17, and
/actions/runs/39037187784/artifacts returns an empty list. The e2e gate then fails on
needs.e2e-case.result == failure, so the check can never turn green again — the PR has to be
re-pushed even though nothing in it is broken. This is behind a large share of the 26 open PRs whose
only red check is e2e.

What this changes

retest.yml — before retrying a run, probe its artifacts. If it is an e2e run and
shenyu-images is missing or expired, request a full workflow rerun instead of a failed-job rerun,
so build-docker-images runs again and regenerates the artifact. Runs that still hold the artifact
keep the cheaper failed-job rerun, and every other workflow is unaffected.

e2e-k8s.yml — after downloading, check that the two image tarballs are actually there and fail
with an actionable message. A manual "Re-run failed jobs" from the UI now says what is wrong and what
to do, instead of surfacing a docker load error.

Verification

  • Both workflows parse (YAML.load_file), and the embedded github-script passes node --check
    when wrapped in an async function.
  • The preflight was exercised against a directory holding only one of the two tarballs: it reports
    the missing one and exits non-zero; with both present it passes.
  • Not exercised end to end here: it needs a real expired artifact. The cheapest way to confirm after
    merge is to /retest an old e2e-failed PR and watch for a full rerun instead of a failed-job
    rerun.

Notes

I did not raise retention-days, although that would also help: two tarballs per run times the
number of active PRs makes artifact storage grow quickly, and the fallback above is correct without
it. Say the word if you would rather trade storage for fewer full reruns.

The e2e workflow downloads the shenyu-images artifact that
build-docker-images uploads with retention-days: 1. /retest calls
reRunWorkflowFailedJobs, which re-executes only the failed jobs: the
producer job stays successful and is never re-run. Once the artifact has
expired, every retry of e2e-case or e2e-storage fails on the download

  Unable to download artifact(s): Artifact not found for name: shenyu-images

and no amount of /retest can recover the run.

retest.yml now probes the artifacts of an e2e run before retrying it and
falls back to a full workflow rerun when shenyu-images is missing or
expired, so build-docker-images runs again and regenerates it. Runs that
still have the artifact keep the cheaper failed-job rerun.

e2e-k8s.yml also checks the two image tarballs after the download, so a
manual "Re-run failed jobs" fails with an actionable message instead of an
error from docker load.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants