test(e2e): forward LITELLM_LICENSE to UI e2e proxy by ryan-crabbe-berri · Pull Request #28398 · BerriAI/litellm

ryan-crabbe-berri · 2026-05-20T21:08:36Z

Summary

The UI e2e job (e2e_ui_testing) ran without LITELLM_LICENSE, so the proxy started unlicensed and premium_user was always false in the issued login JWT. That meant premium-gated UI surfaces — Team-BYOK Model switch on the Add Model form, and other premium-only controls — couldn't be driven through the UI in any test.

This PR forwards the env var from run_e2e.sh (local) and the CircleCI e2e_ui_testing job (CI) to the proxy and the Playwright runner. It also adds a sanity test that decodes the admin storage-state JWT and asserts premium_user=true, so any future regression where the env var stops being plumbed through fails loudly with a clear message instead of silently shifting tests onto the unlicensed branch.

The new test gates itself on process.env.LITELLM_LICENSE, so local runs without a license skip cleanly. Also adds an e2e_tests/tsconfig.json so the directory (excluded from the dashboard tsconfig) gets Node types in the editor.

Test plan

tests/proxy-admin/license.spec.ts → "admin session JWT carries premium_user=true when LITELLM_LICENSE is set" — passes locally with license sourced from .env (1 passed, 3.5s); skips cleanly without (1 skipped)
No-license fallback verified: ./run_e2e.sh -g "premium_user=true" with LITELLM_LICENSE unset → 1 skipped
With-license verified: set -a && source .env && set +a && ./run_e2e.sh -g "premium_user=true" → 1 passed
CI run on this branch — should pass with the project-level LITELLM_LICENSE already used by other CircleCI jobs

The UI e2e job ran without LITELLM_LICENSE, so premium_user was always false in the issued login JWT and premium-gated UI surfaces (Team-BYOK Model switch, etc.) couldn't be driven through the UI. Forward the env var from run_e2e.sh and the CircleCI e2e_ui_testing job, and add a sanity test that decodes the admin storage state token and asserts premium_user=true so the wiring fails loudly if it ever regresses. Co-Authored-By: Claude Opus 4.7 <[email protected]>

greptile-apps · 2026-05-20T21:12:01Z

Greptile Summary

This PR fixes a gap where the e2e_ui_testing job launched the proxy without LITELLM_LICENSE, causing the issued JWT to carry premium_user=false and making premium-gated UI surfaces unreachable in Playwright tests. The fix forwards the env var through both the local run_e2e.sh script and the CircleCI job, and adds a lightweight sanity test that decodes the admin JWT and asserts the claim is present.

run_e2e.sh / .circleci/config.yml: LITELLM_LICENSE is now exported/forwarded to both the proxy process and the Playwright runner; the pattern is consistent with how other credentials (LITELLM_MASTER_KEY, etc.) are already handled.
license.spec.ts: New synchronous test reads admin.storageState.json, splits and decodes the JWT payload, and asserts premium_user=true; gates itself on process.env.LITELLM_LICENSE so runs without a license skip cleanly.
tsconfig.json: Editor-only (noEmit: true) config added to give the e2e_tests/ directory Node built-in types; does not affect Playwright's own TypeScript transpilation.

Confidence Score: 5/5

Safe to merge — changes are confined to CI configuration, a shell script, a new test file, and an editor-only tsconfig; no production code paths are touched.

All four changed files are infra/test-only. The env-var forwarding is consistent with existing patterns in the script and CI job. The new test is well-guarded: it skips when unlicensed, asserts the JWT segment count before destructuring, and only checks a claim on a trusted local proxy. No regressions are introduced.

No files require special attention.

Important Files Changed

Filename	Overview
.circleci/config.yml	Forwards LITELLM_LICENSE to both the proxy startup command and the Playwright runner in the e2e_ui_testing job; change is minimal and correct
ui/litellm-dashboard/e2e_tests/run_e2e.sh	Exports LITELLM_LICENSE (defaulting to empty string) so the proxy subprocess and Playwright both inherit it; consistent with how other vars are handled in this script
ui/litellm-dashboard/e2e_tests/tests/proxy-admin/license.spec.ts	New sanity test that reads the admin storage-state file, decodes the JWT payload, and asserts premium_user=true; correctly skips when LITELLM_LICENSE is unset, and the 3-segment length check guards against undefined payloadB64
ui/litellm-dashboard/e2e_tests/tsconfig.json	Adds a noEmit tsconfig scoped to the e2e_tests directory so editors resolve Node built-in types; does not affect Playwright's own TypeScript transpilation

_{Reviews (2): Last reviewed commit: "Update ui/litellm-dashboard/e2e_tests/te..." | Re-trigger Greptile}

codecov · 2026-05-20T21:12:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

ryan-crabbe-berri · 2026-05-20T21:55:27Z

@greptileai re review

mateo-berri

Just a quick question. Otherwise LGTM

mateo-berri · 2026-05-21T00:39:40Z

+{
+  "compilerOptions": {
+    "target": "ES2020",
+    "module": "esnext",
+    "moduleResolution": "bundler",
+    "esModuleInterop": true,
+    "resolveJsonModule": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "noEmit": true,
+    "types": ["node"]
+  },
+  "include": ["**/*.ts"],
+  "exclude": ["node_modules"]
+}


do we need this?

we dont, remove d

Other specs in e2e_tests/tests use the same imports (@playwright/test, relative paths, fs, Buffer) without a local tsconfig — Playwright handles transpilation. The file added in 8a19340 was scope creep. Co-Authored-By: Claude Opus 4.7 <[email protected]>

* feat: add guardrail violation span attributes and fix missing spans on pre-call blocks (#28364) - Fix missing guardrail child spans when a pre-call guardrail blocks the request before reaching the LLM provider; `async_post_call_failure_hook` now calls `_emit_guardrail_spans_from_request_data` to emit spans from `request_data["metadata"]` regardless of whether `_handle_failure` already fired - Add `guardrail_status`, `guardrail_action`, and `guardrail_violation_categories` as queryable top-level OTEL span attributes so trace backends can filter/group by violation type without parsing the redacted `guardrail_response` blob - Introduce `_emit_guardrail_spans_from_request_data` helper that constructs minimal kwargs from `request_data["metadata"]` and routes through `_create_guardrail_span`, sharing the same dedupe state to prevent double-emitting when both failure hooks fire - Extend `BedrockGuardrail` with `_build_tracing_detail` and `_extract_violation_category_names` which flatten BLOCKED assessments into human-readable category labels (topic names, content-filter types, PII entity types, named regex names) before redaction, and surface Bedrock's raw `action` field via `tracing_detail` - Security: violation category extraction deliberately omits `customWords.match` and unnamed regex `match` values because those fields carry the user-submitted content that triggered the rule; only operator-defined `name`/`type` labels are emitted - Add `violation_categories` and `guardrail_action` fields to `StandardLoggingGuardrailInformation` and `GuardrailTracingDetail` TypedDicts to carry the pre-redaction metadata through the logging pipeline - Add comprehensive test suite covering: guardrail span creation on failure, dedupe between `_handle_failure` and `async_post_call_failure_hook`, per-span status attributes for multi-guardrail sequences, Bedrock category extraction for all policy types, security leak prevention, and end-to-end `CustomGuardrail` violation path Co-authored-by: Yassin Kortam <[email protected]> * test(proxy): behavior-pinning matrix for team management endpoints (#28441) * test(proxy): behavior-pinning matrix for team management endpoints PR2 (Team Tier-1) of the management-endpoint behavior-pinning effort. Extends the tests/proxy_behavior/management/ harness PR1 built and adds the actor x target-resource authz matrix for the 7 team endpoints: /team/new, /team/info, /team/list, /team/update, /team/member_add, /team/member_delete, /team/member_update. Tests-only, no production code changes. Harness extensions: - actors.py: ORG_B_ADMIN actor (org admin of ORG_B) and TEAM_GAMMA (an ORG_A team with no actor members), so team-targeting endpoints get a clean own / same-org-other / cross-org target axis. - conftest.py: create_scratch_team() raw-seeds target teams without /team/new side effects; the scratch teardown now also strips dangling scratch-team refs from LiteLLM_UserTable.teams. 156 new scenarios; status codes pinned to observed handler behavior. * test(proxy): record mutmut run blockers in PR2 triage doc Attempted a scoped local mutmut run for G5; it did not complete. Record the three concrete blockers in mutmut_triage/pr2-team-tier1.md so the next attempt has a head start: 1. mutmut's mutants/ sandbox is import-shadowed by the worktree source. 2. the legacy mock suite and the real-DB behavior suite cannot share a pytest session (mock suite globally patches prisma_client). 3. the CI mutation-test.yml workflow starts no Postgres, so its stats phase now aborts on the behavior-suite tests PR1 added to tests_dir. mutmut stays a deferred follow-up (as in PR1); the binding pre-merge signal remains the behavior matrix (G1) and the G4 regression-replay. * test(proxy): drop suite README + triage doc, trim test comments Remove the two prose docs from the behavior suite (README.md and mutmut_triage/pr2-team-tier1.md) and tighten the comment blocks on the team test files + harness down to the load-bearing parts (the gate each matrix pins, plus genuinely surprising results). No behavior change — all 286 scenarios still pass. * test(proxy): remove mutmut tests_dir comment * test(vertex_ai): tolerate transient 500 in google maps grounding test (#28503) test_gemini_google_maps_tool_simple makes live calls to Vertex AI's Google Maps grounding backend, which intermittently returns 500 INTERNAL ("Please retry") — a transient Google-side failure, not a LiteLLM bug. The request LiteLLM emits matches Google's published googleMaps grounding spec field-for-field, and the maps-platform 500 only occurs after Vertex accepts the request. The test already passes on RateLimitError; treat InternalServerError the same way so transient Vertex-side failures don't fail CI. * fix(docker): restore npm to non_root builder image (#28519) The non_root builder stage installs `nodejs` but not `npm`. Without `npm` on PATH, prisma-python falls back to downloading a Node runtime via nodeenv from nodejs.org, and that downloaded binary fails to load `libatomic.so.1` — breaking `prisma generate` and the image build. `npm` was dropped from this apk list in ca52e34. Restoring it lets prisma-python use the system Node + npm, matching docker/Dockerfile which already installs `npm` for the same reason. * build(deps): bump next from 16.2.4 to 16.2.6 in /ui/litellm-dashboard (#27665) (#28524) Bumps [next](https://github.com/vercel/next.js) from 16.2.4 to 16.2.6. - [Release notes](https://github.com/vercel/next.js/releases) - [Changelog](https://github.com/vercel/next.js/blob/canary/release.js) - [Commits](vercel/next.js@v16.2.4...v16.2.6) --- updated-dependencies: - dependency-name: next dependency-version: 16.2.6 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps-dev): bump black to 26.3.1 and apply formatting (#28525) * build(deps-dev): bump black 24.10.0 -> 26.3.1 * style: apply black 26.3.1 formatting * chore: authorize black 26.3.1 license in liccheck.ini * chore(deps): bump deps (#28528) * build(deps): bump next from 16.2.4 to 16.2.6 in /ui/litellm-dashboard (#27665) Bumps [next](https://github.com/vercel/next.js) from 16.2.4 to 16.2.6. - [Release notes](https://github.com/vercel/next.js/releases) - [Changelog](https://github.com/vercel/next.js/blob/canary/release.js) - [Commits](vercel/next.js@v16.2.4...v16.2.6) --- updated-dependencies: - dependency-name: next dependency-version: 16.2.6 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump protobufjs in /tests/pass_through_tests (#28296) Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.5.6 to 7.6.0. - [Release notes](https://github.com/protobufjs/protobuf.js/releases) - [Changelog](https://github.com/protobufjs/protobuf.js/blob/protobufjs-v7.6.0/CHANGELOG.md) - [Commits](protobufjs/protobuf.js@protobufjs-v7.5.6...protobufjs-v7.6.0) --- updated-dependencies: - dependency-name: protobufjs dependency-version: 7.6.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump ws from 8.20.0 to 8.20.1 in /tests/pass_through_tests (#28303) Bumps [ws](https://github.com/websockets/ws) from 8.20.0 to 8.20.1. - [Release notes](https://github.com/websockets/ws/releases) - [Commits](websockets/ws@8.20.0...8.20.1) --- updated-dependencies: - dependency-name: ws dependency-version: 8.20.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * test(e2e): forward LITELLM_LICENSE to UI e2e proxy (#28398) * test(e2e): forward LITELLM_LICENSE to UI e2e proxy The UI e2e job ran without LITELLM_LICENSE, so premium_user was always false in the issued login JWT and premium-gated UI surfaces (Team-BYOK Model switch, etc.) couldn't be driven through the UI. Forward the env var from run_e2e.sh and the CircleCI e2e_ui_testing job, and add a sanity test that decodes the admin storage state token and asserts premium_user=true so the wiring fails loudly if it ever regresses. Co-Authored-By: Claude Opus 4.7 <[email protected]> * Update ui/litellm-dashboard/e2e_tests/tests/proxy-admin/license.spec.ts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 4.7 <[email protected]> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Add granian as a ASGI compliant web server. Provider better throughput stability, (#26027) * Add granian as a ASGI compliant web server. Provides better stability, 10-20 RPS improvement under standard LT conditions. TODO: Verify poetry lock details and add locust numbers to PR * Update granian version in license_cache.json and pyproject.toml to 2.5.7 * Enhance proxy CLI tests by adding SSL initialization checks for Granian server. Remove Python version skip conditions and implement tests to ensure SSL certificate and key are required for server initialization. * update uv lock to fix granian import error --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> Co-authored-by: yuneng-jiang <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <[email protected]> Co-authored-by: Claude Opus 4.7 <[email protected]> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: harish-berri <[email protected]>

* test(e2e): forward LITELLM_LICENSE to UI e2e proxy The UI e2e job ran without LITELLM_LICENSE, so premium_user was always false in the issued login JWT and premium-gated UI surfaces (Team-BYOK Model switch, etc.) couldn't be driven through the UI. Forward the env var from run_e2e.sh and the CircleCI e2e_ui_testing job, and add a sanity test that decodes the admin storage state token and asserts premium_user=true so the wiring fails loudly if it ever regresses. * Update ui/litellm-dashboard/e2e_tests/tests/proxy-admin/license.spec.ts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

ryan-crabbe-berri requested a review from a team May 20, 2026 21:08

greptile-apps Bot reviewed May 20, 2026

View reviewed changes

Comment thread ui/litellm-dashboard/e2e_tests/tests/proxy-admin/license.spec.ts Outdated

Update ui/litellm-dashboard/e2e_tests/tests/proxy-admin/license.spec.ts

5b21b65

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

yuneng-berri approved these changes May 20, 2026

View reviewed changes

ryan-crabbe-berri enabled auto-merge (squash) May 21, 2026 00:22

mateo-berri requested changes May 21, 2026

View reviewed changes

mateo-berri approved these changes May 21, 2026

View reviewed changes

ryan-crabbe-berri merged commit 07bcd2c into litellm_internal_staging May 22, 2026
113 of 114 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

test(e2e): forward LITELLM_LICENSE to UI e2e proxy#28398

test(e2e): forward LITELLM_LICENSE to UI e2e proxy#28398
ryan-crabbe-berri merged 3 commits into
litellm_internal_stagingfrom
litellm_e2e_license_forwarding

ryan-crabbe-berri commented May 20, 2026

Uh oh!

greptile-apps Bot commented May 20, 2026 •

edited

Loading

Important Files Changed

Uh oh!

codecov Bot commented May 20, 2026

Uh oh!

Uh oh!

ryan-crabbe-berri commented May 20, 2026

Uh oh!

mateo-berri left a comment

Uh oh!

mateo-berri May 21, 2026

Uh oh!

ryan-crabbe-berri May 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

ryan-crabbe-berri commented May 20, 2026

Summary

Test plan

Uh oh!

greptile-apps Bot commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

codecov Bot commented May 20, 2026

Codecov Report

Uh oh!

Uh oh!

ryan-crabbe-berri commented May 20, 2026

Uh oh!

mateo-berri left a comment

Choose a reason for hiding this comment

Uh oh!

mateo-berri May 21, 2026

Choose a reason for hiding this comment

Uh oh!

ryan-crabbe-berri May 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

greptile-apps Bot commented May 20, 2026 •

edited

Loading