Repository navigation
feat(proxy): discover metadata and reload selected model offerings - #45023
tudstudent wants to merge 41 commits into
Conversation
|
|
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 5 · PR risk: 0/10 |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
| key: Final = ( | ||
| configured_key.encode() if isinstance(configured_key, str) and configured_key else _MODEL_OFFERING_PROCESS_KEY | ||
| ) | ||
| return hmac.new(key, b"litellm:model-offering-identity:v1\0" + value, hashlib.sha256).hexdigest() |
There was a problem hiding this comment.
This operation derives an opaque, purpose-separated connection identity using HMAC-SHA256 and the existing private server salt/master-key source. It is not password storage, a password verifier, or authentication. Public deployment IDs do not let an observer test credential guesses without the server key; the earlier unkeyed credential fingerprint was replaced for precisely that reason. test_model_offering_identity_uses_server_key_and_not_a_public_credential_hash and credential/account rotation regressions pass. Real HTTP checks also verify stable identities across separate processes with the same configured key. Unconfigured standalone mode uses a private process key and documents its restart-stability limitation. We believe the password-hashing classification does not match this use, but have not suppressed the alert. Leaving it open for maintainer/security assessment; please advise if a different identity design is preferred.
|
Current head: The follow-up revisions address the review findings and the CI quality failures. The current revision uses the native cached HTTP-client factory and reacquires the inventory client for each refresh, so cache eviction cannot leave polling attached to a closed handler. Tests cover startup, invalid configuration, shutdown and actual cache eviction/reacquisition. Local validation passed the full Two CodeQL classifications need maintainer assessment:
We have not suppressed either alert or added an artificial constructor call. Please assess whether these are acceptable intentional uses or whether you prefer a different design. The PR description documents the exact validation scope and limitations. CodeQL is still red; passing functional and quality checks do not constitute CodeQL approval. Greptile's visible 3/5 summary still refers to |
a7e747c to
f7178be
Compare
|
Automatic reviews are disabled for this repository. |
…ing metadata (BerriAI#41041) * fix(proxy): resolve model aliases to their target deployment for listing metadata A model alias (a key's `aliases`, a team's `model_aliases`, or the router's `model_group_alias`) is rewritten at request time and owns no deployment row, so `get_model_listing_info` returns None for it and the alias name is not a cost-map key either. Both metadata candidate sources come up empty and the listing entry carries no `mode`, `max_input_tokens`, or `max_output_tokens`, while the deployment it routes to reports them on the same response. Resolve a listed alias to the deployment it routes to for the metadata lookup, keeping the alias as the response id. Maps are applied in order, each against the result of the last, mirroring how `litellm_pre_call_utils` applies the team map and then the key map, so a chained alias reports the deployment the caller actually reaches rather than an intermediate one. An alias shadowing a same-named deployment follows the alias, because the request rewrite does too. Fixes the model_aliases half of BerriAI#40553. * fix(proxy): look listed aliases up through the shared alias chain Drop the separate alias resolver and retarget a listed alias in alias_listing_entries and alias_target instead, so /v1/models and /v1/models/{id} report the model /chat/completions rewrites it to, and only when the caller can list that model * fix(proxy): keep deployed listed models' own rows and resolve team alias targets at list and retrieval * fix(proxy): resolve router_settings aliases, keep alias labels, and skip paused targets in model listing * fix(proxy): leave an alias to an undiscoverable model bare in model listing A team alias whose target has discoverable: false showed the hidden model's mode and limits on GET /v1/models and retrieved it by alias. Undiscoverable groups now count as hidden for alias resolution, as on main, while admins still see them. * fix(proxy): leave an alias to a model a listing callback hides bare in model listing * fix(proxy): keep aliases bare for models a listing callback never judged or another team owns --------- Co-authored-by: haydster7 <[email protected]> Co-authored-by: mateo-berri <[email protected]>
The gateway publishes selected, authorized models with discovered limits and reasoning options, including supplier-pinned offerings. This update merges current upstream and reuses native implementations without adding a new test suite. Supplier metadata continues to refresh while intentional operator overrides remain available
Details
TLDR
Problem this solves:
How it solves it:
Intentional product change: external-offering mode makes the operator file authoritative. Runtime model-definition mutations, native batches, batch-purpose uploads and caller deployment selectors are rejected in this mode. Pinned policies disable response-cache reads and writes and reject caller tier selectors that would alter endpoint eligibility. Accepted OpenRouter effort values avoid the old unconditional provider rename. Unsupported/default/drop behavior follows native deployment policy. Existing key/team/customer permissions remain authoritative
User Flow
Before: a user with matching wildcard permissions can miss an enabled offering
GET https://gateway.example/v1/modelsafter an offering is enabledAfter: the same user discovers selected offerings within existing permissions
GET https://gateway.example/v1/modelsafter an offering is enabledNative Integration
Source-backed walkthrough, examples and implementation limits are documented in the architecture guide. The guide now distinguishes the latest native reasoning policy from offering metadata and budget defaulting
Canonical reasoning choices/defaults, tool and parallel-tool capabilities, modalities, endpoints, independent token limits and typed request budgets extend native model types and lookup. Unknown metadata is omitted. Caller budgets take precedence and never become physical model maxima
Supplier cards extend the native metadata facet and existing HTTP/cache machinery. Inventory uses native provider-base resolution, preserving explicit/environment/default precedence and avoiding ambient discovery credentials. The obsolete metadata module and fork-only compatibility wrappers are deleted
AUTO/MANUAL configuration supplies operator selection to native Router snapshots. Successful inventories control AUTO availability; MANUAL entries remain handwritten. Invalid edits preserve the last valid generation. Credential/account changes invalidate inventories, and in-flight requests retain their pinned generation
Private OpenRouter policies reside on native deployments rather than parallel snapshot maps. Exact
onlylists andallow_fallbacks: falsecompile into existing request shaping. Ingress/deployment guards reject connection, routing, parameter-dropping, fallback and policy overrides, and isolate caches. Chat, Responses, Responses-to-chat and Messages retain streaming/nonstreaming behavior. No provider ranking or pricing policy is addedPinned AUTO offerings now fetch bounded, credential-scoped endpoint metadata only for selected models. They combine conservative eligible-endpoint bounds and capabilities with OpenRouter's model-level reasoning contract when those endpoints support effort selection. Unknown independent input remains omitted. Explicit fields win individually, MANUAL stays handwritten and unpinned discovery keeps its established model-level meaning. Last-known endpoint metadata survives refresh failure only for the same credential/base identity. The common metadata aggregation now lives with native types and is reused by public listing
Explicit null reasoning enums use the documented gateway effort set; omission, empty choices, mandatory reasoning and disabled defaults retain distinct semantics. Accepted declared OpenRouter
maxandxhighsurvive request shaping rather than the old provider rename. Native deployment policy owns unsupported-value rounding/default/drop behavior. Provider-policy bodies remain unchanged, with no added fallback orrequire_parameterssettingMetadata Authority And Overrides
The external offering file is authoritative for model selection, connection and routing policy; it does not require model capabilities to be handwritten. Automatic supplier metadata is the normal path, not an opt-in exception.
For example, if the sole eligible endpoint advertises a 300,000-token context and 30,000-token output ceiling, an AUTO offering without explicit overrides advertises those values and follows subsequent supplier changes on refresh. An unknown independent input limit remains omitted. Client applications consume the resolved gateway catalog rather than maintaining supplier-specific model exceptions.
Pre-Submission checklist
Local Validation
The source integration at
23ae9fbe177a21abc2b368f5e91df56516b42492merges upstreamdddbaa803d6e9fcc862ea7a5412f771cb1ae040ewith history preservedThe native overlap audit reuses upstream deployment reasoning resolution and deployment-pricing/base-model lookup. The redundant fork key lookup is removed. Offering selection, pinned endpoint inventory/guarantees, request-budget policy and serving-generation guards remain because upstream has no equivalent implementation. New native secret-reference, timeout/admission and shared-capability registration behavior is retained. The registration boundary emits JSON-format arrays so existing native routing consumers accept the frozen metadata facet
The existing-test selection passes 2,811 cases across 30 mapped/native files. No new test case, matrix or test file is authored for this sync. Six private-helper cases are replaced by existing native functional coverage, and duplicate independent-limit lookup coverage is consolidated without its deleted upstream schema fixture, retaining the max_tokens assertion. The upstream merge's test additions are inherited, not a new fork test suite
Canonical
BASE_REF=upstream/main make checkpasses source/test lint, formatting, unchanged rule/type/quality ceilings, imports, e2e typing, dashboard budgets and regenerated API-schema synchronization. Shard/slice/census checks pass for 617 children and 3,925 test files. No threshold, type cap, test selection or coverage guard is weakenedThe architecture guide explains accepted versus unsupported effort behavior and the budget selector's limits. The sync adds no custom compactor, history store, client change or model-family workaround. These are offline/in-process results, not interactive client or real supplier inference proof. No live configuration, image or deployment is changed
CI Remediation
Current head is
3f78bfabd79073673c74c9b26cd1b89e26c1c9b4. Seven files change by +35/-17, with no runtime or documentation changes and no new test cases or filesFilelock is bounded below 3.20.4 in both distribution manifests and resolves to 3.20.3, before the vulnerable soft-lock and SQLite reader/writer features were introduced. This preserves Python 3.10 support. The extension overrides its packaging dependency to patched MSAL Node 7.1.1. Only the full-history DB shard gets eight minutes for checkout instead of three, retaining its history, credentials policy and 60-minute job budget
The unchanged OSV scanner and configuration report no unfiltered vulnerabilities locally. Python 3.10 runtime resolution, the extension's 53 tests, typechecking and packaging, five focused lock-user cases, 15 aggregate-gate cases, 17 telemetry preset cases and the core manifest case pass. Existing workflow startup and job-name checks pass. The canonical pre-commit entry passes test-tree Ruff and the unchanged test-quality gate against upstream/main
Two existing telemetry tests now scope their HTTP mock to the exporter session. The previous process-wide mock recorded unrelated background exports, causing a third request without AgentOps authorization in the assertion history. An in-process reproduction records that extra request before the change and exactly the two authenticated requests afterward. The JWT-fetch count, exact request count and authorization assertions remain unchanged
Remote Validation
Functional, dependency, lint and coverage checks completed on
3f78bfabd79073673c74c9b26cd1b89e26c1c9b4. Only the two policy/security checks described here remain redOSV passes with no unfiltered vulnerabilities. DB/schema passes with 36 passed and 12 skipped. Root passes 7,054 cases, integrations passes 6,307, and the unchanged unit aggregate passes. Runtime resolution and core-distribution checks pass on every supported Python version from 3.10 through 3.14
The two remaining red checks are the unchanged fork dependency guard, which rejects every uv.lock edit from a fork, and CodeQL. Accepting the validated dependency fix through the canonical repository requires maintainer action. The fork guard is not bypassed
The current CodeQL check has exactly one annotation, at encrypt_decrypt_utils.py line 60. It classifies SHA256 as password hashing, while this call computes a server-keyed, domain-separated connection-identity HMAC rather than a password verifier. Python, JavaScript and Actions analyses pass. This single classification still requires upstream security disposition. No query suppression or algorithm workaround is introduced
The upstream Greptile bot reports automatic reviews disabled. Earlier manual requests, including the prior runtime-head request, did not yield a current review. They do not cover this new sync. No duplicate ping, fork-only scan or integration/permission workaround is requested
Type
New Feature, Bug Fix, Refactoring, Test
Caveats
Severe
Medium
Low
Final Attestation