Skip to content

[Fix] Test Isolation and Path Resolution for GHA Unit Tests#24741

Merged
yuneng-berri merged 2 commits intomainfrom
litellm_gha_p2
Mar 28, 2026
Merged

[Fix] Test Isolation and Path Resolution for GHA Unit Tests#24741
yuneng-berri merged 2 commits intomainfrom
litellm_gha_p2

Conversation

@yuneng-berri
Copy link
Copy Markdown
Collaborator

@yuneng-berri yuneng-berri commented Mar 28, 2026

Summary

Failure Path (Before Fix)

Agent health check tests (4 failures): TestAgentHealthCheck tests returned 500 errors when run with parallel workers (pytest-xdist --dist=loadscope). The get_agents() endpoint imports prisma_client from proxy_server at runtime — when a MagicMock leaked from another test scope on the same worker, the endpoint hit TypeError: object MagicMock can't be used in 'await' expression.

Documentation validation tests: test_router_settings.py found 2 undocumented router settings keys (enable_health_check_routing, health_check_staleness_threshold) that were added to the Router but never documented.

Fix

Agent health check isolation: Added monkeypatch.setattr("litellm.proxy.proxy_server.prisma_client", None) to the TestAgentHealthCheck._setup fixture so the endpoint skips DB queries regardless of leaked module-level state from other test scopes.

Documentation sync: Added 2 missing router_settings keys and 27 missing general_settings keys to config_settings.md.

Documentation workflow: Re-added test-unit-documentation.yml running exactly the 4 tests that CircleCI ran: test_env_keys.py, test_router_settings.py, test_api_docs.py, test_circular_imports.py. Uses direct python execution matching CircleCI behavior. Path resolution updated to use __file__-based paths in test_router_settings.py so tests work regardless of working directory.

Changes

  • tests/test_litellm/proxy/agent_endpoints/test_endpoints.py: Mock prisma_client to None in TestAgentHealthCheck fixture
  • tests/documentation_tests/test_router_settings.py: __file__-based path resolution
  • docs/my-website/docs/proxy/config_settings.md: Add 27 general_settings + 2 router_settings keys
  • .github/workflows/test-unit-documentation.yml: New workflow matching CircleCI's documentation tests

Testing

  • All 32 agent endpoint tests pass locally
  • All 4 CircleCI documentation tests pass: test_env_keys.py, test_router_settings.py, test_api_docs.py, test_circular_imports.py

Type

🐛 Bug Fix
📖 Documentation
🚄 Infrastructure
✅ Test

…ath resolution

Fix agent health check tests failing with 500 errors in parallel CI by
mocking prisma_client to None. Fix documentation validation tests using
CWD-relative paths that break depending on the working directory.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@vercel
Copy link
Copy Markdown

vercel bot commented Mar 28, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
litellm Ready Ready Preview, Comment Mar 28, 2026 6:25pm

Request Review

@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@codspeed-hq
Copy link
Copy Markdown
Contributor

codspeed-hq bot commented Mar 28, 2026

Merging this PR will not alter performance

✅ 16 untouched benchmarks


Comparing litellm_gha_p2 (7851567) with main (428d837)

Open in CodSpeed

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Mar 28, 2026

Greptile Summary

This PR fixes two categories of flaky CI failures: leaked MagicMock state causing TypeError in the TestAgentHealthCheck tests under pytest-xdist, and CWD-dependent path resolution breaking documentation validation tests when run from a non-standard working directory. The fixes are targeted and well-explained.

  • test_endpoints.py: monkeypatch.setattr(\"litellm.proxy.proxy_server.prisma_client\", None) cleanly isolates the health-check tests from cross-scope mock leakage — correct approach that does not weaken coverage.
  • test_router_settings.py: Path resolution replaced with __file__-based _repo_root — correct fix, but the old variable name repo_base is still referenced in the except block (line 64), which will raise a NameError and hide the original error if the docs file cannot be opened.
  • config_settings.md: ~25 new general_settings / router_settings entries backfill documentation that was causing the validation test to fail.
  • test-unit-documentation.yml: Restores the documentation CI workflow now that the path issue is resolved; uses pinned action commit SHAs throughout.

Confidence Score: 4/5

Safe to merge after fixing the stale repo_base reference in the except block of test_router_settings.py.

One P1 bug was introduced: repo_base was removed but is still referenced in the except block, meaning any future failure to open the docs file will surface as a confusing NameError rather than the intended diagnostic message. All other changes are clean and do not weaken any existing assertions or coverage.

tests/documentation_tests/test_router_settings.py — stale repo_base reference on line 64

Important Files Changed

Filename Overview
tests/documentation_tests/test_router_settings.py Path resolution correctly switched to __file__-based _repo_root, but the old variable name repo_base is still referenced in the except block (line 64), causing a NameError if the docs file cannot be opened.
tests/test_litellm/proxy/agent_endpoints/test_endpoints.py Adds monkeypatch.setattr("litellm.proxy.proxy_server.prisma_client", None) to prevent a leaked MagicMock from other pytest-xdist workers causing TypeError in the health-check endpoint. Clean, targeted fix that does not weaken test coverage.
.github/workflows/test-unit-documentation.yml Restores the documentation validation workflow, running four scripts from the repo root with pinned action commit SHAs. Straightforward and safe.
docs/my-website/docs/proxy/config_settings.md Adds ~25 new entries to the general_settings and router_settings tables, filling in previously undocumented keys that caused the documentation validation test to fail.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[pytest-xdist worker starts TestAgentHealthCheck] --> B{prisma_client leaked\nfrom another scope?}
    B -- Before fix: Yes --> C["prisma_client = MagicMock()"]
    C --> D["GET /v1/agents calls find_many()"]
    D --> E["TypeError: MagicMock can't\nbe used in await → 500"]
    B -- After fix: monkeypatch sets None --> F["prisma_client = None"]
    F --> G["GET /v1/agents skips DB query"]
    G --> H["200 OK — uses mock_registry only"]

    subgraph doc_tests["Documentation Test Path Resolution"]
        I["poetry run python ./tests/documentation_tests/test_router_settings.py"]
        I --> J["_test_dir = os.path.dirname(os.path.abspath(__file__))"]
        J --> K["_repo_root = os.path.abspath(join(_test_dir, '..', '..'))"]
        K --> L["docs_path = _repo_root/docs/my-website/..."]
        L --> M{File exists?}
        M -- Yes --> N["Parse & validate docs"]
        M -- No --> O["except block raises Exception\n⚠ repo_base NameError here"]
    end
Loading

Comments Outside Diff (1)

  1. tests/documentation_tests/test_router_settings.py, line 64 (link)

    repo_base removed but still referenced in except block

    The variable repo_base was deleted as part of the path-resolution fix (replaced with _repo_root), but line 64 inside the except block still references the old name. If the docs file ever fails to open (e.g., the path is wrong), Python will raise a NameError: name 'repo_base' is not defined inside the exception handler, completely hiding the original error message.

Reviews (2): Last reviewed commit: "[Fix] Scope documentation workflow to ma..." | Re-trigger Greptile

…router settings

Revert path fixes for documentation tests that CircleCI never ran
(test_exception_types, test_general_setting_keys, test_readme_providers,
test_standard_logging_payload). Update the GHA workflow to run only the
4 tests CircleCI actually executed: test_env_keys, test_router_settings,
test_api_docs, test_circular_imports.

Add 2 missing router_settings keys (enable_health_check_routing,
health_check_staleness_threshold) and 27 missing general_settings keys
to config_settings.md so test_router_settings passes.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@yuneng-berri yuneng-berri merged commit 666a31d into main Mar 28, 2026
59 of 106 checks passed
@yuneng-berri yuneng-berri deleted the litellm_gha_p2 branch March 28, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants