Skip to content

test(e2e): cover add-fallback flow in Router Settings#29069

Merged
ryan-crabbe-berri merged 3 commits into
litellm_internal_stagingfrom
litellm_e2e_router_fallback
May 27, 2026
Merged

test(e2e): cover add-fallback flow in Router Settings#29069
ryan-crabbe-berri merged 3 commits into
litellm_internal_stagingfrom
litellm_e2e_router_fallback

Conversation

@ryan-crabbe-berri

@ryan-crabbe-berri ryan-crabbe-berri commented May 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds an e2e test for the previously-uncovered manual-QA item "Router Settings → Fallbacks → Add fallback". The test opens the Add Fallbacks modal, picks fake-openai-gpt-4 as primary and fake-anthropic-claude as the fallback (both seeded in the e2e config), saves, and verifies both render in the fallback table.

Test plan

  • e2e_tests/tests/settings/routerSettings.spec.ts::Add a fallback and verify it appears in the table

Refs LIT-3093

The Router Settings → Fallbacks → Add Fallbacks flow was an uncovered
manual-QA path. This adds a test that opens the modal, picks a primary
+ fallback from the seeded mock models, saves, and verifies both render
in the fallback table.
@greptile-apps

greptile-apps Bot commented May 27, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds an end-to-end Playwright test for the Router Settings → Fallbacks → Add Fallback flow, covering a previously manual-QA-only path. The test seeds fake-openai-gpt-4 as primary and fake-anthropic-claude as fallback, saves via the UI, and asserts both values appear together in a single table row.

  • beforeEach/afterEach hooks call clearFallbackForPrimary via the LiteLLM API to remove any leftover state, making the test idempotent across retries and local reruns.
  • The waitForResponse listener for /model_group/info is registered before the "Add Fallbacks" button click, correctly avoiding the CI race condition where the modal-mounted request could fire before the listener was attached.
  • The final assertion uses chained .filter({ hasText }) calls to require both model names in the same tbody tr, which is stronger than searching for each name independently across the whole table.

Confidence Score: 5/5

New test-only file with no changes to production code; safe to merge.

The change adds a single new Playwright spec with no impact on production code paths. The teardown logic correctly reads from the live router via /get/config/callbacks (which does return router_settings) and posts back only the filtered fallbacks. All previously flagged weaknesses — missing teardown, weak table assertion, mismatched tab-count comment — have been addressed in this revision.

No files require special attention.

Important Files Changed

Filename Overview
ui/litellm-dashboard/e2e_tests/tests/settings/routerSettings.spec.ts New e2e test covering the Add Fallback flow in Router Settings, with idempotent before/after teardown, race-condition-safe waitForResponse pattern, and strengthened dual-filter table assertion.

Reviews (3): Last reviewed commit: "fix(e2e): keyboard-select fallback model..." | Re-trigger Greptile

Comment thread ui/litellm-dashboard/e2e_tests/tests/settings/routerSettings.spec.ts Outdated
Comment thread ui/litellm-dashboard/e2e_tests/tests/settings/routerSettings.spec.ts Outdated
Comment thread ui/litellm-dashboard/e2e_tests/tests/settings/routerSettings.spec.ts Outdated
@codecov

codecov Bot commented May 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…by text

- Match `.ant-select-item-option` by text instead of `getByTitle(...)` —
  FallbackGroupConfig uses `options=` (not <Select.Option> children), so
  no `title` attribute is emitted and the title-based selector hangs.
- Add before/after hooks that wipe any fallback for fake-openai-gpt-4 via
  /config/update so retries and local reruns don't trip on leftover state.
- Tighten the success assertion to a single tbody row containing BOTH the
  primary and the fallback names — pre-existing rows can no longer
  vacuously satisfy the check.
- Fix the stale "Three tabs" comment to "Four tabs".

Addresses Greptile P2s on PR #29069.
@ryan-crabbe-berri

Copy link
Copy Markdown
Collaborator Author

@greptileai re review

- Replace mouse-based option clicks with click-to-focus + type + Enter.
  FallbackGroupConfig's Selects use `options=` and a custom
  getPopupContainer, so locating options via `.ant-select-dropdown`
  hit several races: DOM-clicks left antd's popup state stale (the
  primary popup then intercepted the fallback click), `getByRole`
  matched always-mounted hidden options, and pointer stability fought
  the open animation. Typing into the showSearch input narrows the
  listbox to one option and Enter selects it cleanly.
- Assert on dialog-side state changes (the active tab adopts the
  primary model name; the chain helper shows "1/10 used") instead of
  popup contents — these reflect the actual selection landing.
- Cleanup helper now hits /get/config/callbacks (the real endpoint;
  /get/callbacks returns 404), so the before/after reset actually
  clears prior router_settings.fallbacks state.
@ryan-crabbe-berri

Copy link
Copy Markdown
Collaborator Author

@greptileai re review

@ryan-crabbe-berri ryan-crabbe-berri merged commit b0ea013 into litellm_internal_staging May 27, 2026
115 of 118 checks passed
fzowl pushed a commit to fzowl/litellm that referenced this pull request Jun 24, 2026
* test(e2e): cover add-fallback flow in Router Settings as proxy admin

The Router Settings → Fallbacks → Add Fallbacks flow was an uncovered
manual-QA path. This adds a test that opens the modal, picks a primary
+ fallback from the seeded mock models, saves, and verifies both render
in the fallback table.

* fix(e2e): make router-fallback test idempotent and pick antd options by text

- Match `.ant-select-item-option` by text instead of `getByTitle(...)` —
  FallbackGroupConfig uses `options=` (not <Select.Option> children), so
  no `title` attribute is emitted and the title-based selector hangs.
- Add before/after hooks that wipe any fallback for fake-openai-gpt-4 via
  /config/update so retries and local reruns don't trip on leftover state.
- Tighten the success assertion to a single tbody row containing BOTH the
  primary and the fallback names — pre-existing rows can no longer
  vacuously satisfy the check.
- Fix the stale "Three tabs" comment to "Four tabs".

Addresses Greptile P2s on PR BerriAI#29069.

* fix(e2e): keyboard-select fallback models + correct cleanup endpoint

- Replace mouse-based option clicks with click-to-focus + type + Enter.
  FallbackGroupConfig's Selects use `options=` and a custom
  getPopupContainer, so locating options via `.ant-select-dropdown`
  hit several races: DOM-clicks left antd's popup state stale (the
  primary popup then intercepted the fallback click), `getByRole`
  matched always-mounted hidden options, and pointer stability fought
  the open animation. Typing into the showSearch input narrows the
  listbox to one option and Enter selects it cleanly.
- Assert on dialog-side state changes (the active tab adopts the
  primary model name; the chain helper shows "1/10 used") instead of
  popup contents — these reflect the actual selection landing.
- Cleanup helper now hits /get/config/callbacks (the real endpoint;
  /get/callbacks returns 404), so the before/after reset actually
  clears prior router_settings.fallbacks state.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants