Skip to content

fix(ci): retry Maven Central downloads and restore E2E cache - #7781

Merged
Aias00 merged 1 commit into
apache:masterfrom
lymerin:7767
Oct 19, 2026
Merged

Aias00 merged 1 commit into
apache:masterfrom
lymerin:7767

Conversation

@lymerin

@lymerin lymerin commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #7767

The final approach is retry + restore + conditional save, supported by controlled fork benchmarks and a read-only analysis of the official CI environment.

Problem

Maven Central transfer failures abort CI builds without a workflow-level retry. The E2E image-build job also saves the Maven repository cache but does not restore it before building.

Impact

Dependency-download failures stop build jobs and fail their aggregate checks, blocking PRs before the affected validation completes.

A seven-day review of 198 official failed-job logs confirmed at least 17 Maven Central reset/502 terminal failures, across 14 commit SHAs and three dates. This is a recurring CI reliability problem rather than an isolated failed run.

Changes

  • Add bounded retries for eligible Maven Central connection resets and HTTP 429/502 responses, including the observed Wagon Transfer failed for … 502 Bad Gateway format. Allow at most three executions, with 10-second and 20-second delays.
  • Retry only when all classified terminal Maven failures are eligible Central transfer errors. Compilation, test, HTTP 404, missing-artifact, other-repository, and mixed failures are not retried.
  • Apply the helper to PR builds, integration build/test steps, E2E image/example builds, the SQL list-contract check, and the MySQL, WebSocket, and Spring Cloud E2E Maven invocations.
  • Retry only the Maven command, without restarting the surrounding Compose lifecycle.
  • Restore the Maven repository before the E2E image build, retaining the existing cache key, path, and restore prefix.
  • Use conditional saving:
    • Exact hit → skip save.
    • Fallback or miss → save the current exact key after a successful build.
  • Preserve the existing Maven artifact and Docker image delivery paths.
  • Add focused regression coverage, including a fixture derived from the observed Wagon 502 failure.

Benchmark results

The fork benchmark completed 12 serial runs across four configurations, plus seven concurrent runs. Every valid run used the same source revision and root Maven/Docker workload. Each serial configuration had three samples.

Metric Retry without restore Retry with warm exact-hit restore
Mean Maven build time 159.2 s 120.2 s
Mean cache restore time — 6.3 s
Mean Central Downloaded log entries 2,943 0

The build mean decreased by 39.0 seconds (24.5%). Accounting for restore time gives approximately 32.7 seconds of net build-plus-restore savings.

Exact-hit save attempts spent 2–5 seconds performing archive/reservation work, with zero new cache-content upload. Skipping these attempts removes redundant work while retaining cache publication after fallback hits and misses.

Resource findings

Resource Measured result and design implication
Dependency downloads Official sampled E2E builders downloaded 2,912–2,943 Central entries without restore. Repeated cold downloads are an avoidable cost and expose builds to the transfer failures reported in this issue.
Cache transfer The fork cache was 740.79 MiB compressed, with a mean restore time of 6.3 seconds. Restore cost was substantially smaller than the measured build-time reduction.
Runner disk and Docker staging The minimum sampled free disk was 80.8 GiB, including cache restore, the Maven/Docker build, cache saving, and both Docker image exports. Disk exhaustion was not a bottleneck in the measured workload.
Repository cache storage The official inventory snapshot contained 11 entries totaling 9.78 GiB. Five PR refs held separate entries with the same key/version, consuming approximately 3.88 GB. This explains why an inherited exact hit should not create another redundant PR-local cache.
Cache quota versus runner disk Repository cache storage and runner-local disk are separate resources. No cache-related runner startup failure was observed. The configured official quota was unavailable through the read-only endpoint (HTTP 402), so the inventory is a usage measurement, not proof that the repository is full.

These measurements support restoring dependencies before building, skipping redundant exact-hit saves, and retaining miss/fallback saves for downstream reuse.

Validation

  • All 21 local retry behavior cases passed.
  • The existing storage Compose script test passed for all five runners.
  • Bash syntax checks and Git whitespace checks passed.
  • Offline replay against 198 official failed-job logs recognized 15 of the 17 target Central-transfer failures. The mixed spring-release 401 failure and the unsupported registry build-extension failure remained rejected. All 181 other scanned failures remained rejected.

The fork benchmark measured warm-cache performance; it did not naturally encounter reset/429/502 failures. Real transient-recovery success rates were therefore not measured. The subsequent conditional-save and bare-502 refinements were verified separately through local tests, source checks, and historical-log replay.

Detailed measurements and supporting evidence:

Non-blocking note

Artifact expiration during partial reruns and unrelated test failures are outside this PR’s scope.

Unsupported registry build-extension failures also remain outside this change.

Checklist

Make sure that:

  • You have read the contribution guidelines.
  • You submit test cases (unit or integration tests) that back your changes.
  • Your local test passed ./mvnw clean install -Dmaven.javadoc.skip=true.

@lymerin
lymerin marked this pull request as draft October 17, 2026 15:44
@lymerin
lymerin marked this pull request as ready for review October 18, 2026 09:06

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved.

This directly addresses the failure mode in #7767, and the approach is conservative in the right places. I traced the classifier and the wrapper end to end:

Classifier (is_transfer_failure) — verified conservative, not permissive:

  • Requires errors > 0 && references > 0 && !invalid && model_errors == classified_model_errors, so a log with any second, unrelated terminal error is rejected. Confirmed against the compile, tests, not-found, other-repository, mixed-failures, mixed-failures-reversed, mixed-dependencies, mixed-transfers and invalid-model fixtures — all correctly refuse to retry.
  • Only central (https://repo.maven.apache.org/maven2) transfers matching reset / 429 / 502 (plus the Wagon Transfer failed for … 502 Bad Gateway shape) qualify; Could not find artifact short-circuits to invalid, so the real apache.snapshots miss seen in #7184 is not retried.
  • Handles mvnd prefixes, ANSI colors and CRLF before classification, and the Help-reference allow-list (DependencyResolution|PluginResolution|ProjectBuilding|UnresolvableModel) keeps MojoFailureException out.
  • Each attempt re-classifies only the current attempt log, so a retried transfer error cannot be "reused" by a later attempt (covered by current-attempt-only).

Wrapper — control flow is correct:

  • At most 3 executions; 10s then 20s backoff; statuses[1] -ne 0 (tee failure) and statuses[0] -ne 1 both abort instead of retrying.
  • Statuses are captured from PIPESTATUS immediately after the pipeline under set +e, then set -e is restored — no masking of the original exit code.
  • INT -> 130 / TERM -> 143 with the EXIT trap still removing the temp log; matches the cancellation case.
  • exec is only ever the last statement of its branch, so no following step is silently skipped.
  • Argument fidelity is proven by the NUL-separated printf | cmp check with two words and literal $(command); * — no re-splitting and no eval.

Cache change in e2e-k8s.yml — restore and save use the identical ${{ runner.os }}-maven-${{ hashFiles(**/pom.xml) }} key, cache-hit != true skips only exact hits, and fallback/miss still publish. Restore-before-build is what removes the ~2.9k cold Central downloads measured in the benchmark.

Callers — the three compose scripts call the wrapper as a plain command and keep their $? / if ! checks; none of them use set -e, so error handling is preserved. native-sql-matrix.yml folded scalar renders correctly on one line.

CI is green on this branch (27 pass / 7 path-filtered skips), including the new Test Maven transfer retries step.

Non-blocking notes:

  1. Rollout sequencing matters. pull_request workflows are read from the PR head, so merging this does not retroactively repair the ~49 currently-red PRs — each needs to merge/rebase master to pick up the new script. Operationally: merge this first, then batch-resync the red queue and re-run; otherwise it will look like nothing improved.
  2. Scope reminder for release planning. This retries Maven Central transfer failures only. The e2e failures I sampled also involve Docker/Compose and other repositories, which stay unretried. Worth stating explicitly so the queue is not expected to fully clear.
  3. Cache save race (minor). Several jobs/workflows on the same ref compute the same key; if a reservation conflict ever surfaces as a step failure, continue-on-error: true on the Save step would absorb it.
  4. Docs (optional). A short table in the script header listing which Maven error shapes are eligible would help future maintainers extend the allow-list when Maven changes its output wording.

Good to merge.

@Aias00
Aias00 merged commit 0c495e6 into apache:master Oct 19, 2026
34 checks passed
@Aias00 Aias00 added this to the 2.7.2 milestone Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI] Maven Central dependency-resolution failures (Connection reset / 502) break required build jobs and block unrelated PRs

2 participants