Repository navigation
Allow opting status codes out of the retry classification #2193
Description
Use case
Lychee currently retries every 5xx code (per RetryExt for StatusCode). For some 5xx codes, notably Cloudflare's CDN-specific ones like 521 Web Server Down, 523 Origin Unreachable, 525 SSL Handshake Failed, and 530 Origin DNS Error / bot block, the underlying condition is typically on a minutes-to-hours scale. A within-seconds exponential-backoff retry has near-zero chance of changing the outcome but consumes the retry budget anyway.
accept is a partial workaround but conflates "transient, pass silently" with "non-retriable, fail fast". Many users want the latter without the former: the URL should still appear in the error report so the operator can investigate, but lychee shouldn't waste the retry budget on it.
Proposal
A subtractive config option that removes codes from the built-in retry classification:
# Status codes to skip retrying (subtracted from built-in retry defaults).
# Useful for codes whose recovery time exceeds typical retry windows
# (e.g., Cloudflare 521/523/525/530 origin-side failures).
retry_skip_status_codes = [521, 523, 525, 530]A matching CLI flag --retry-skip-status-codes 521,523,525,530 would mirror it. Default-empty preserves current behavior.
Why subtractive (not just accept)
accept reclassifies a code as success, so the URL never appears in the error list. That is the wrong tool when the operator wants to know the URL failed (to investigate later) but does not want lychee to waste the retry budget on it now.
Activity
mre commentedon May 12, 2026
I would not mind skipping retries on these status codes by default.
afalhambra-hivemq commentedon May 13, 2026
uh, that would be even nicer! Although I still see this as a valid option to have as well
costajohnt commentedon Jul 10, 2026
I started implementing this (subtractive
retry_skip_status_codesconfig + CLI flag) and hit something worth checking before I open a PR.lychee doesn't actually retry 5xx status-code responses in the normal request path today, so subtracting 521/523/525/530 from the retry set would be a no-op for the exact case in the proposal. Quick empirical check with a mock server and
max_retries = 3:The reason is that a 5xx response comes back as
Status::Error(ErrorKind::RejectedStatusCode(code)), andErrorKind::should_retryonly returns true for 429 there. Theis_server_error()branch inRetryExt for StatusCodeonly runs forreqwest::Errorcases (transport-level), not for status-code responses. Soshould_retry()is already false for 521/523/525/530 and the retry loop returns before any skip list is consulted.One nuance: real-world Cloudflare 52x that manifest as a connection timeout do get retried (
Status::Timeout), but those carry no HTTP status code, so a numeric skip list can't target them either.So before I build this out, which direction do you want?
Happy to implement whichever you prefer. I have the plumbing for a
StatusCodeSelector-based option ready either way.docs(dev-log): correct the 2026-10-05 link-check breakdown
ci(links): re-check transient link failures instead of failing on them
konard commentedon Oct 11, 2026
Additional reproducible evidence for the status-response distinction raised above: with Lychee 0.24.2 and
--max-retries 3, a local endpoint returning 503 once and 200 thereafter receives exactly one request and Lychee exits 2. A second invocation succeeds; a persistent 404 control still fails.Finite reproduction (three invocations, localhost only; no Docker):
The complete standalone script is in link-foundation/links-notation PR 341, at
experiments/issue-339/probe_lychee_retry.py. Output:{"path": "/flaky", "exit_code": 2, "cumulative_requests": 1, "successful": 0, "errors": 1} {"path": "/flaky", "exit_code": 0, "cumulative_requests": 2, "successful": 1, "errors": 0} {"path": "/gone", "exit_code": 2, "cumulative_requests": 1, "successful": 0, "errors": 1}Our CI run failed twice on GitHub HTML file URLs returning 503. The first attempt had 21 occurrences across 13 distinct files. Live probes found all 13 raw-content equivalents returned 200, while a missing-file raw URL returned 404. A user-agent change alone did not prevent recurrence.
Workarounds we verified: recheck failures after a pause with caching, and remap only
https://github.com/OWNER/REPO/blob/REF/PATHto the equivalent raw-content URL while retaining 404 failures. The latter now has a local server regression in the PR. Neither workaround reclassifies 503 as success.For a code fix, aligning
RejectedStatusCoderetry classification with the intended bounded 5xx policy would make--max-retriesgovern actual HTTP responses as well as transport errors; regression cases should cover 503-then-200, persistent 503 exhausting the budget, and 404 remaining non-retryable. That alignment should be decided before adding the proposed subtractive selector. I am adding evidence here instead of opening a duplicate.CI: check GitHub blob links against raw.githubusercontent.com (#5187)
CI: check GitHub file links on disk or via raw.githubusercontent.com (#…