Skip to content

Add retry logic to WildFly download in smoke test Dockerfiles#16814

Closed
zeitlinger wants to merge 9 commits into
open-telemetry:mainfrom
zeitlinger:fix-wildfly-download-retry
Closed

Add retry logic to WildFly download in smoke test Dockerfiles#16814
zeitlinger wants to merge 9 commits into
open-telemetry:mainfrom
zeitlinger:fix-wildfly-download-retry

Conversation

@zeitlinger

Copy link
Copy Markdown
Member

download.jboss.org intermittently returns HTTP 503, causing CI flakes in the WildFly smoke test image build (e.g. #16810 CI failure).

  • Linux: add --tries=5 --waitretry=3 --retry-connrefused to wget
  • Windows: replace ADD (no retry support) with curl.exe --retry 5 --retry-delay 3 --retry-all-errors

@zeitlinger
zeitlinger marked this pull request as ready for review March 20, 2026 12:38
@zeitlinger
zeitlinger requested a review from a team as a code owner March 20, 2026 12:38
@jaydeluca
jaydeluca self-requested a review March 20, 2026 16:47
@zeitlinger
zeitlinger marked this pull request as draft March 21, 2026 08:31
@zeitlinger
zeitlinger marked this pull request as ready for review March 21, 2026 09:07
@trask

trask commented Mar 25, 2026

Copy link
Copy Markdown
Member

@zeitlinger is this related

* What went wrong:
Execution failed for task ':smoke-tests:images:servlet:libertyImage-23.0.0.12-jdk17-windows'.
> Could not build image: ADD failed: Get "https://public.dhe.ibm.com/ibmdl/export/pub/software/openliberty/runtime/release/23.0.0.12/openliberty-23.0.0.12.zip": dial tcp 170.225.126.18:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

@zeitlinger

Copy link
Copy Markdown
Member Author

@trask please check again

@trask

trask commented Apr 1, 2026

Copy link
Copy Markdown
Member

I haven't noticed this issue recently (was definitely happening a lot a couple of weeks ago), wonder if they've fixed something and whether this is still worth it

download.jboss.org intermittently returns 503, causing CI flakes
in the Windows WildFly image build job.
Signed-off-by: Gregor Zeitlinger <[email protected]>
- Linux: use --retry-on-http-error=503 instead of --retry-connrefused
  (the latter only handles TCP-level failures, not HTTP 503)
- Windows: remove quotes around URL so Docker ARG substitution works
  with the default cmd shell
Docker expands ${baseDownloadUrl} before passing to cmd /S /C, but
without quotes special characters in the URL can be misinterpreted
by cmd.exe, causing curl exit code 3 (URL malformed).
Transient Docker daemon (connection refused) and HCS (CreateComputeSystem)
failures cause frequent Windows CI flakes. Wrap both Gradle steps in a
3-attempt retry loop with 10s backoff, only on windows-latest.
Transient Docker daemon (connection refused) and HCS (CreateComputeSystem)
failures cause frequent Windows CI flakes. Wrap both Gradle steps with a
retry script (3 attempts on Windows, 1 on Linux) to recover automatically.
Remove duplicated inline retry loops from Build and Test steps in favor
of the shared .github/scripts/retry.sh wrapper.
Wrap the Windows docker image builds in pr-smoke-test-servlet-images and
publish-smoke-test-servlet-images with retry.sh to handle transient
Docker/HCS failures.
Replace ADD (which has no retry support) with PowerShell
Invoke-WebRequest wrapped in a retry loop (5 attempts, 3s wait)
to handle transient HTTP failures like 503s.
@zeitlinger
zeitlinger force-pushed the fix-wildfly-download-retry branch from 335f1aa to e5988eb Compare April 8, 2026 12:15
@zeitlinger

Copy link
Copy Markdown
Member Author

I haven't noticed this issue recently (was definitely happening a lot a couple of weeks ago), wonder if they've fixed something and whether this is still worth it

maybe no windows builds? But I'll put it back into draft - and we'll see.

@zeitlinger
zeitlinger marked this pull request as draft April 8, 2026 16:32
@trask

trask commented Apr 8, 2026

Copy link
Copy Markdown
Member

yeah, was seeing it again today, found another option: #17721

@zeitlinger zeitlinger closed this Apr 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants