tests(proxy_server): surface current behavior in tests#29309
Conversation
* test(proxy): pin proxy_server.py forwarding-route behavior PR2 of the proxy_server.py behavior-pinning project: fills the 12 forwarding-route test files added by the harness PR with happy + error pins for all 52 LLM-facing routes (models, chat/completions, completions, embeddings, moderations, audio, assistants, threads, utils, model-info, model-metrics, queue). Every happy-path test asserts the full response dict via normalize() so the gate enforces real shape pinning rather than status codes. * test(proxy): drop task-plumbing comments from PR2 test files * test(proxy): tighten PR2 error-path status-code pins Apply the same review feedback Greptile gave on PR1 (#28856) and PR3 (#28850) to PR2's forwarding-route tests: - Replace permissive `>= 400` / `in (X, Y)` status assertions with the exact 500/405 the handler actually returns, so a regression that silently shifts the code now fails the pin. - Add a body-presence check alongside each tightened status assertion to satisfy _pin_check.py's no-status-only rule. --------- Co-authored-by: Claude <[email protected]>
…8856) * test(proxy): pin proxy_server.py non-route surface behavior (PR1) Fills the 7 PR1 placeholder files under tests/test_litellm/proxy/proxy_server/ with behavior pins for the non-route surface of proxy_server.py: lifecycle/init/shutdown, ProxyConfig class methods, DB-overlay config scrubbers, spend counters, background-health helpers, OpenAPI customization, exception handlers, and streaming-generator helpers. 233 tests cover 101 pin-list symbols (1+ happy + 1+ error each). New-tests-only coverage on litellm/proxy/proxy_server.py: 32.80% line / 20.91% branch (PR1 gate: 25% line / 18% branch). Full directory runs in ~22s with -n 4. Plan: https://www.notion.so/Plan-Pin-proxy_server-py-behavior-2026-05-25-36c43b8acdab81ee845fd5365128a2fc * test(proxy): address Greptile review comments on test_lifecycle.py - test_initialize_signature_is_async_with_expected_params: hard-code expected_param_count so a signature change actually trips the gate (previously both sides of the comparison were len(sig.parameters)). - test_check_request_disconnection_invalid_when_connected_times_out: patch asyncio.sleep so the test no longer spins for ~1.2 s of real wall-clock; timeout lowered to 0.05 s. --------- Co-authored-by: Claude <[email protected]>
* test(proxy/proxy_server): pin misc routes (PR3, partial) Adds happy + error tests for the misc control-plane routes: GET /, /routes, /adaptive_router/state, /get_logo_url, /get_image, /get_favicon. Also gitignores .pin_list.txt (used by the pin gate). * test(proxy/proxy_server): pin login/SSO routes (PR3, partial) Adds happy + error tests for the 5 login/SSO control-plane routes: GET /fallback/login, POST /login, POST /v2/login, POST /v3/login, POST /v3/login/exchange. Mocks authenticate_user and create_ui_token_object at their imported location. * test(proxy/proxy_server): pin onboarding routes (PR3, partial) Adds happy + error tests for the 2 onboarding control-plane routes: GET /onboarding/get_token, POST /onboarding/claim_token. Wires a MagicMock async context manager for prisma_client.db.tx() and signs the onboarding JWT with the patched master_key. * test(proxy/proxy_server): pin model_cost_map reload routes (PR3, partial) Adds happy + error tests for the 5 model-cost-map control-plane routes: POST /reload/model_cost_map, POST|DELETE|GET /schedule/model_cost_map_reload(/status), GET /model/cost_map/source. Attaches litellm_config to mock_prisma per-test (the table is not in the default _PRISMA_TABLES fixture). * test(proxy/proxy_server): pin anthropic_beta_headers reload routes (PR3, partial) Adds happy + error tests for the 4 anthropic-beta-headers control-plane routes: POST /reload/anthropic_beta_headers, POST|DELETE|GET /schedule/anthropic_beta_headers_reload(/status). Stubs db.litellm_config (not in default _PRISMA_TABLES) and monkeypatches reload_beta_headers_config so no network calls fire. * test(proxy/proxy_server): pin invitation routes (PR3, partial) Adds happy + error tests for the 4 invitation control-plane routes: POST /invitation/new, GET /invitation/info, POST /invitation/update, POST /invitation/delete. Patches _user_has_admin_privileges / _user_has_admin_view to avoid extensive get_user_object mocking. * test(proxy/proxy_server): pin config CRUD routes (PR3, partial) Adds happy + error tests for the 8 config-CRUD control-plane routes: POST /config/update, POST|GET /config/field/update|info, GET /config/list, POST /config/field/delete, POST /config/callback/delete, GET /get/config/callbacks, GET /config/yaml. Attaches litellm_config to mock_prisma per-test. * test(proxy/proxy_server): tighten pin assertions per review - test_routes_misc.py: `b"" in response.content` is trivially true; replace with `len(response.content) > 0` so an empty 405 body trips the gate. - test_routes_login_sso.py: `len(response.content) >= 0` is trivially true; tighten to `> 0`. - test_routes_anthropic_beta.py: replace brittle string-literal checks on the serialized JSON (`'"interval_hours": 12' in payload`) with `json.loads` + dict access so the assertion survives any serializer spacing. - test_routes_config.py: `assert status_code in (404, 500)` was too permissive; the handler re-raises HTTPException(404) verbatim, so pin 404 strictly. --------- Co-authored-by: Claude <[email protected]>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Greptile SummaryThis PR fills in the placeholder test files for
Confidence Score: 4/5Safe to merge — all changes are test-only and do not touch production code paths. The PR is entirely test additions. One test creates a real OS-level socket connection pool (aiohttp TCPConnector) rather than mocking it, and one timeout-driven test uses a 50 ms wall-clock budget that could produce intermittent failures on slow CI runners. Neither affects production behavior. tests/test_litellm/proxy/proxy_server/test_lifecycle.py — the aiohttp session and timeout tests discussed above.
|
| Filename | Overview |
|---|---|
| .gitignore | Adds .cov_new.xml and .pin_list.txt to .gitignore — but .pin_list.txt is added twice (lines 31 and 126), creating a harmless duplicate entry. |
| tests/test_litellm/proxy/proxy_server/test_lifecycle.py | Behavior pins for lifecycle helpers. The timeout-based test for check_request_disconnection uses a 50ms wall-clock limit with a patched sleep; functions as intended but is a potential source of flakiness. All other tests are mock-only with no real network calls. |
| tests/test_litellm/proxy/proxy_server/test_proxy_config.py | Comprehensive behavior pins for ProxyConfig and scrubber helpers. All happy-path and error-path branches covered with strong assertions; entirely mock-based with no real I/O besides tmp_path file operations. |
| tests/test_litellm/proxy/proxy_server/test_spend_counters.py | Thorough behavior pins for all spend-counter helpers including Redis/in-memory interaction paths and edge cases like cache warmth checks, window counters, and invalidation. |
| tests/test_litellm/proxy/proxy_server/test_streaming_helpers.py | Behavior pins for all streaming helper functions covering SSE serialization, chunk restamping, hook application, and generator selection paths. |
| tests/test_litellm/proxy/proxy_server/conftest.py | Shared fixtures: normalize() helper, session-scoped app/client, mock_prisma with all tables stubbed, auth_as context manager, mock_router, and response builder utilities. Well-structured with no issues. |
Reviews (1): Last reviewed commit: "test(proxy/proxy_server): pin control-pl..." | Re-trigger Greptile
| async def test_initialize_shared_aiohttp_session_returns_client_session(): | ||
| from aiohttp import ClientSession | ||
|
|
||
| session = await _initialize_shared_aiohttp_session() | ||
| try: | ||
| observed = { | ||
| "is_client_session": isinstance(session, ClientSession), | ||
| "is_closed": session.closed, | ||
| "has_connector": session.connector is not None, | ||
| } | ||
| assert normalize(observed) == { | ||
| "is_client_session": True, | ||
| "is_closed": False, | ||
| "has_connector": True, | ||
| } | ||
| finally: | ||
| if session is not None: | ||
| await session.close() |
There was a problem hiding this comment.
test_initialize_shared_aiohttp_session_returns_client_session calls the real _initialize_shared_aiohttp_session(), which creates a live aiohttp.TCPConnector and ClientSession — allocating a real OS-level socket/connection pool. While no HTTP requests are made, this is a real network resource allocation. Consider mocking aiohttp.ClientSession and TCPConnector to stay in line with the project rule that tests in this directory must not make real network calls, and to keep the test hermetic in air-gapped CI environments.
Rule Used: What: prevent any tests from being added here that... (source)
| monkeypatch.delenv("WORKER_CONFIG", raising=False) | ||
|
|
||
| save_worker_config(model="gpt-4", config="/tmp/c.yaml", debug=True) | ||
|
|
||
| payload = json.loads(os.environ["WORKER_CONFIG"]) | ||
| assert normalize(payload) == { | ||
| "model": "gpt-4", | ||
| "config": "/tmp/c.yaml", | ||
| "debug": True, | ||
| } | ||
|
|
||
|
|
||
| def test_save_worker_config_invalid_no_kwargs_yields_empty(monkeypatch): | ||
| monkeypatch.delenv("WORKER_CONFIG", raising=False) | ||
|
|
||
| save_worker_config() | ||
| assert os.environ["WORKER_CONFIG"] == "{}" | ||
|
|
||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # initialize | ||
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| def test_initialize_signature_is_async_with_expected_params(): | ||
| sig = inspect.signature(initialize) |
There was a problem hiding this comment.
test_check_request_disconnection_invalid_when_connected_times_out relies on a 50 ms wall-clock budget (timeout=0.05). The patched asyncio.sleep reduces each iteration to a single await asyncio.sleep(0), but the event loop still needs to schedule the timeout callback and process the CancelledError. On a heavily-loaded CI runner the 50 ms budget can be exhausted before the loop even completes one full iteration, causing a spurious asyncio.TimeoutError to arrive inside a different await than expected — or the timeout machinery may race against the next loop iteration, making the test flaky. Consider raising the budget to 0.5 s or replacing the wall-clock timeout with a counter-based stop condition.
| .vscode | ||
| .pin_list.txt |
There was a problem hiding this comment.
.pin_list.txt is added twice in this PR — once near the top (line 31, alongside .cov_new.xml) and again here at the end. The duplicate has no functional effect, but it's unnecessary noise.
| .vscode | |
| .pin_list.txt | |
| .vscode |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
12d29a3
into
litellm_internal_staging
* test(proxy/proxy_server): pin forwarding routes (PR2) (BerriAI#28887) * test(proxy): pin proxy_server.py forwarding-route behavior PR2 of the proxy_server.py behavior-pinning project: fills the 12 forwarding-route test files added by the harness PR with happy + error pins for all 52 LLM-facing routes (models, chat/completions, completions, embeddings, moderations, audio, assistants, threads, utils, model-info, model-metrics, queue). Every happy-path test asserts the full response dict via normalize() so the gate enforces real shape pinning rather than status codes. * test(proxy): drop task-plumbing comments from PR2 test files * test(proxy): tighten PR2 error-path status-code pins Apply the same review feedback Greptile gave on PR1 (BerriAI#28856) and PR3 (BerriAI#28850) to PR2's forwarding-route tests: - Replace permissive `>= 400` / `in (X, Y)` status assertions with the exact 500/405 the handler actually returns, so a regression that silently shifts the code now fails the pin. - Add a body-presence check alongside each tightened status assertion to satisfy _pin_check.py's no-status-only rule. --------- * test(proxy): pin proxy_server.py non-route surface behavior (PR1) (BerriAI#28856) * test(proxy): pin proxy_server.py non-route surface behavior (PR1) Fills the 7 PR1 placeholder files under tests/test_litellm/proxy/proxy_server/ with behavior pins for the non-route surface of proxy_server.py: lifecycle/init/shutdown, ProxyConfig class methods, DB-overlay config scrubbers, spend counters, background-health helpers, OpenAPI customization, exception handlers, and streaming-generator helpers. 233 tests cover 101 pin-list symbols (1+ happy + 1+ error each). New-tests-only coverage on litellm/proxy/proxy_server.py: 32.80% line / 20.91% branch (PR1 gate: 25% line / 18% branch). Full directory runs in ~22s with -n 4. Plan: https://www.notion.so/Plan-Pin-proxy_server-py-behavior-2026-05-25-36c43b8acdab81ee845fd5365128a2fc * test(proxy): address Greptile review comments on test_lifecycle.py - test_initialize_signature_is_async_with_expected_params: hard-code expected_param_count so a signature change actually trips the gate (previously both sides of the comparison were len(sig.parameters)). - test_check_request_disconnection_invalid_when_connected_times_out: patch asyncio.sleep so the test no longer spins for ~1.2 s of real wall-clock; timeout lowered to 0.05 s. --------- * test(proxy/proxy_server): pin control-plane routes (PR3) (BerriAI#28850) * test(proxy/proxy_server): pin misc routes (PR3, partial) Adds happy + error tests for the misc control-plane routes: GET /, /routes, /adaptive_router/state, /get_logo_url, /get_image, /get_favicon. Also gitignores .pin_list.txt (used by the pin gate). * test(proxy/proxy_server): pin login/SSO routes (PR3, partial) Adds happy + error tests for the 5 login/SSO control-plane routes: GET /fallback/login, POST /login, POST /v2/login, POST /v3/login, POST /v3/login/exchange. Mocks authenticate_user and create_ui_token_object at their imported location. * test(proxy/proxy_server): pin onboarding routes (PR3, partial) Adds happy + error tests for the 2 onboarding control-plane routes: GET /onboarding/get_token, POST /onboarding/claim_token. Wires a MagicMock async context manager for prisma_client.db.tx() and signs the onboarding JWT with the patched master_key. * test(proxy/proxy_server): pin model_cost_map reload routes (PR3, partial) Adds happy + error tests for the 5 model-cost-map control-plane routes: POST /reload/model_cost_map, POST|DELETE|GET /schedule/model_cost_map_reload(/status), GET /model/cost_map/source. Attaches litellm_config to mock_prisma per-test (the table is not in the default _PRISMA_TABLES fixture). * test(proxy/proxy_server): pin anthropic_beta_headers reload routes (PR3, partial) Adds happy + error tests for the 4 anthropic-beta-headers control-plane routes: POST /reload/anthropic_beta_headers, POST|DELETE|GET /schedule/anthropic_beta_headers_reload(/status). Stubs db.litellm_config (not in default _PRISMA_TABLES) and monkeypatches reload_beta_headers_config so no network calls fire. * test(proxy/proxy_server): pin invitation routes (PR3, partial) Adds happy + error tests for the 4 invitation control-plane routes: POST /invitation/new, GET /invitation/info, POST /invitation/update, POST /invitation/delete. Patches _user_has_admin_privileges / _user_has_admin_view to avoid extensive get_user_object mocking. * test(proxy/proxy_server): pin config CRUD routes (PR3, partial) Adds happy + error tests for the 8 config-CRUD control-plane routes: POST /config/update, POST|GET /config/field/update|info, GET /config/list, POST /config/field/delete, POST /config/callback/delete, GET /get/config/callbacks, GET /config/yaml. Attaches litellm_config to mock_prisma per-test. * test(proxy/proxy_server): tighten pin assertions per review - test_routes_misc.py: `b"" in response.content` is trivially true; replace with `len(response.content) > 0` so an empty 405 body trips the gate. - test_routes_login_sso.py: `len(response.content) >= 0` is trivially true; tighten to `> 0`. - test_routes_anthropic_beta.py: replace brittle string-literal checks on the serialized JSON (`'"interval_hours": 12' in payload`) with `json.loads` + dict access so the assertion survives any serializer spacing. - test_routes_config.py: `assert status_code in (404, 500)` was too permissive; the handler re-raises HTTPException(404) verbatim, so pin 404 strictly. --------- ---------
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
make test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
CI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes