Skip to content

fix(llm): rewrite Vertex rawPredict for any publisher - #2539

Merged
howardjohn merged 1 commit into
agentgateway:mainfrom
piyushbag:fix/2204-vertex-rawpredict-publishers
Jul 16, 2026
Merged

howardjohn merged 1 commit into
agentgateway:mainfrom
piyushbag:fix/2204-vertex-rawpredict-publishers

Conversation

@piyushbag

Copy link
Copy Markdown
Contributor

Summary

  • Vertex :rawPredict / :streamRawPredict virtual-model path rewrites no longer hardcode publishers/anthropic.
  • Parse /publishers/{publisher}/models/{model}:{method}, keep {publisher}, rewrite only the model id.
  • Existing anthropic path behavior is unchanged; added coverage for other publishers (google, meta).

Test plan

  • cargo test -p agentgateway --lib llm::model_router::tests::rewrite_path_model
  • CI on this PR

Fixes #2204

Virtual-model path rewrites for :rawPredict and :streamRawPredict were
hardcoded to publishers/anthropic, so non-anthropic Vertex publishers
never had their model id replaced. Preserve the publisher segment and
rewrite only the model.

Fixes agentgateway#2204

Signed-off-by: Piyush Jagadish Bag <[email protected]>
@piyushbag
piyushbag requested a review from a team as a code owner July 15, 2026 13:35
@howardjohn
howardjohn enabled auto-merge (squash) July 16, 2026 23:03
@howardjohn
howardjohn merged commit ae379d1 into agentgateway:main Jul 16, 2026
9 of 10 checks passed
howardjohn pushed a commit that referenced this pull request Jul 17, 2026
…de (#2571)

## Summary

Detect-mode traffic to Gemini's native `generateContent` API produced
empty telemetry: `gen_ai.request.model` was
blank and no `gen_ai.usage.*` fields were logged, even though the
upstream response contained all of the data. Any
logging, metrics, or usage accounting built on these fields silently
sees nothing for Gemini native traffic.

This matters because some Gemini features are only reachable through
detect mode. For example, Search grounding
(`tools: [{"google_search": {}}]`) is rejected by both AI Studio's and
Vertex's OpenAI-compatible endpoints
(`"Expected the 'type' field of a(n) 'tools' array element to be
'function'; found 'google_search'"`), so grounding
requests must use the native API — where telemetry was silently dropped.

Reproduced against live Vertex traffic before writing the fix: a
successful grounded `generateContent` call logged
  `gen_ai.request.model=` (empty) and no usage fields.

Two root causes in `crates/llm/src/types/detect.rs`, fixed
independently:

1. **Model extraction** — `extract_model_from_path` hardcoded
`/publishers/anthropic/models/` and only matched the
`:rawPredict`/`:streamRawPredict` verbs, so Gemini paths
(`/publishers/google/models/{model}:generateContent`) never
matched. Now splits on the generic `/publishers/` + `/models/` markers
(the same approach #2539 applied to the
path-*rewrite* side) and additionally matches
`:generateContent`/`:streamGenerateContent`. `amend_request_info` also
  treats `:streamGenerateContent` as streaming.
2. **Usage extraction** — the token lookup tables only knew
OpenAI/Anthropic/Bedrock response shapes. Added Gemini's
native shape: `usageMetadata.promptTokenCount` / `candidatesTokenCount`
/ `totalTokenCount`.

Both changes are additive: the new lookup entries are tried only after
all existing ones miss, the
Anthropic-on-Vertex path extraction is a strict generalization (existing
tests untouched and passing), and no other
  provider's behavior changes.

  ## Test plan

- [x] `cargo test -p agent-llm` — 133 passed, including 3 new tests:
model extraction for `:generateContent` and
`:streamGenerateContent` paths (with streaming flag), and usage
extraction from a realistic Gemini native response
  body
  - [x] `make lint` — clean
- [x] Existing Anthropic-on-Vertex tests
(`amend_request_info_extracts_vertex_raw_predict_model`,
`..._stream_raw_predict_model`) left untouched as regression checks for
the publisher generalization

  ## Out of scope / follow-ups

- **`provider_model`**: Gemini native responses report the served model
under `modelVersion`, which isn't in the
`MODEL` lookup, so response-side model attribution remains `None`.
Deferred because that const is also used for
  request-side lookups and deserves its own consideration.
- **Virtual-model path rewriting**: `rewrite_path_model` doesn't handle
the `:generateContent` verbs yet (it
  degrades to a no-op, not an error). Natural follow-up mirroring #2539.

Signed-off-by: emereshub <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

streamRawPredict / rawPredict - support non-anthropic models

2 participants