fix(http_handler): keep the connect timeout bounded when a request passes a numeric timeout - #47483

TLDR

Problem this solves:

• request_timeout: 600 also became a 600 s connect timeout
• A stopped self-hosted upstream then stalls requests 40-137 s
• The router cannot fail over until that connect gives up

How it solves it:

• AsyncHTTPHandler.post turns a numeric timeout into httpx.Timeout
• The connect part is min(timeout, HTTP_HANDLER_CONNECT_TIMEOUT_SECONDS), 5 s today
• Same for the fresh client that retries the request after a ConnectError
• An explicit httpx.Timeout from the caller is passed through untouched

User Flow

Before: a user whose request lands on a self-hosted backend that just went down waits over a minute

1. The proxy admin runs two OpenAI-compatible backends in one model group with request_timeout: 600 and num_retries: 2
2. One backend container is stopped
3. The user sends POST http://localhost:4100/v1/chat/completions and the request routed to the stopped backend hangs 72 s
4. The answer then comes back 200 from the other backend

After: the same request fails over after about 6 s

1. The proxy admin runs the same two backends with the same config
2. One backend container is stopped
3. The user sends the same POST and the attempt on the stopped backend times out after 5 s
4. The answer comes back 200 from the other backend after 6.1-6.2 s in total

Relevant issues

Refs #47481 (point 2 there, the connect timeout. The stale DNS cache entry and the hidden error are not changed here)

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• Every automated review on my PR's latest commit has finished, and I have addressed or replied to each of its findings before requesting a maintainer review

Screenshots / Proof of Fix

Self-hosted backends here are fake OpenAI-compatible servers, one docker container each, behind a live proxy on localhost:4100. Repro kit: https://gist.github.com/tc3oliver/0a8f98de57a67a85e3d937107148f4a9. Before is the stock v1.106.0 image. After is the same image with this PR's http_handler.py diff applied (patched-http_handler.py in the kit, main and v1.106.0 differ only in unrelated parts of that file)

Config single is one deployment b1 with num_retries: 0 and request_timeout: 600. Config affinity-af0 is two deployments b1/b2 with session_affinity, num_retries: 2, allowed_fails: 0

Before (v1.106.0)

One deployment

1. ./run_issue1.sh single single "" "--after 2" (warms b1 twice, docker stop litellm-repro-b1, sends the next requests)
2. Output, 3 runs: first request after the stop 500 after 72.3 s, 40.0 s and 72.0 s with Cannot connect to host litellm-repro-b1:8000 ... [Temporary failure in name resolution], next request 500 after 3.2 s

Two deployments with session affinity

1. ./run_issue1.sh affinity affinity-af0 "" "--after 3"
2. Output, 2 runs: first request after the stop 200 from the other deployment after 137.1 s and 72.1 s, later requests 0.02 s

After (1165261)

One deployment

1. PATCH=http_handler ./run_issue1.sh single-fixed single "" "--after 2"
2. Output, 3 runs: first request after the stop 408 litellm.Timeout after 5.8 s, 5.8 s and 5.8 s, next request 408 after 5.1 s

Two deployments with session affinity

1. PATCH=http_handler ./run_issue1.sh affinity-fixed affinity-af0 "" "--after 3"
2. Output, 2 runs: first request after the stop 200 from the other deployment after 6.1 s and 6.2 s, later requests 0.02 s

The 600 s case and the replay test fail on main and pass here. make lint passes

Type

Bug Fix

Caveats (if any)

Medium

• A stalled connect now surfaces as 408 Timeout, before it was 500 after the hang
• HTTP_HANDLER_CONNECT_TIMEOUT_SECONDS is a constant, so proxy users cannot raise it
  • SDK callers can pass an httpx.Timeout

Low

• Only AsyncHTTPHandler.post changes, get/put/patch/delete keep the old behavior
• Sync HTTPHandler.post has the same pattern and is unchanged
• The a2a pydantic_ai provider also goes through AsyncHTTPHandler.post, two of its tests pinned connect to the full timeout and now expect the bounded value
• Under allowed_fails_policy a stalled connect now counts as TimeoutErrorAllowedFails, not InternalServerErrorAllowedFails
• The stale DNS entry stays, so each later attempt on the dead deployment costs 5 s until it is cooled down (before it was 3.2 s once the ARP entry had failed)
• fix(http_handler): stop replaying requests when a connection dies #34830 only removes the ConnectError replay, no overlap

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR