Skip to content

[Feature][AI Proxy] Add opt-in native Gemini generateContent and streaming support #8389

Open

Description

@Aias00

Feature Request

Add a bounded native Gemini Developer API path for generateContent and SSE streamGenerateContent through the existing AI Proxy Protocol/Provider/Transport architecture. The first delivery should preserve the native wire contract instead of requiring clients to use an OpenAI-compatible endpoint.

Current Baseline and Existing Work

At master 88883dc20fb1f57db6343ba811c1d1f3ee70a6bf, AI Proxy documents Chat/Responses support and excludes native Gemini transport. #7382 explicitly leaves Gemini to future work; #7871 provides provider-native protocol routing foundations. #7955 owns shared usage metering and requires actual supported routes before Gemini coverage can be claimed.

First Delivery

  1. Define an explicit Gemini protocol registration and a provider binding for the direct Developer API. Document the supported API profile, how a routed model maps to the model resource, and how the JSON versus streaming operation is selected.
  2. Support native text content with ordinary JSON and incremental SSE. Preserve candidate structure, finish/block outcomes and supported native fields. Specify unknown-field handling and reject unsupported multimodal/tool modes without silent loss.
  3. Bind credentials and endpoints to administrator-owned targets using existing proxy-key and fallback authorization. Safely encode model/path components; never accept an arbitrary client-selected destination or log credentials.
  4. Reuse WebClient transport, bounded framing, deadlines, backpressure and cancellation ownership. Distinguish protocol completion from HTTP EOF and from OpenAI framing conventions; follow the selected Gemini profile.
  5. Feed native usage into [Task] Integrate shared per-attempt LLM usage metering across supported protocols #7955/[Task][AI Proxy] Integrate AI Proxy attempt lifecycle with shared usage and invocation telemetry #8125 with source/certainty and attempt identity. Publish which existing Prompt/PII/Token stages can inspect the payload, with unknown/unsupported states rather than inferred policy coverage.
  6. Keep the existing Chat/Responses and provider contracts compatible. Native support does not automatically authorize lossy cross-protocol conversion.

Acceptance Criteria

  • A local native-protocol fixture exercises both operations, model/endpoint/authentication binding, and text JSON/SSE preservation.
  • Cover multiple candidates, finish/block outcomes, usage placement, fragmented UTF-8/SSE, malformed/truncated streams and non-success HTTP responses.
  • Request/response/event budgets, deadlines and downstream cancellation work through the original upstream exchange; no full-stream aggregation or post-commit replay is introduced.
  • Unsupported capability profiles are rejected before network execution; no duplicate metering/debit path is added.
  • Document the tested profile, opt-in configuration and governance coverage. Separate real-service smoke evidence from local protocol fixtures.

Scope Boundaries

Limit the first delivery to text generation through the direct Developer API. Vertex authentication/resource naming, Live API/WebSocket, image/audio/video generation, tool execution and protocol conversion are separate increments. Coordinate future shared detection with #8371 instead of adding competing payload guesses.

Protocol Reference

Google documents the separate JSON and streaming operations, native content/candidate representation and usage metadata in Generating content. Pin the supported profile and fixtures during implementation; this link does not establish current ShenYu compatibility.

Activity

  1. hengyuss commented on Nov 10, 2026

    @hengyuss
    Contributor

    /assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions