Skip to content

[Task][AI Proxy] Complete orchestration in AiProxyEngine #7532

Closed

Description

@hengyuss

[Task][AI Proxy] Complete orchestration in AiProxyEngine

Parent issue: #7351

Depends on: #7382, #7383, #7384

Background

#7381 introduces a minimal AiProxyEngine that validates the initial layer
boundaries by decoding an OpenAI Chat request, constructing an OpenAI upstream
request, and delegating it to Transport.

After Protocol response handling, Provider adaptation, and production
Transport behavior are completed, the Engine must coordinate the complete
request and response lifecycle and own cross-layer execution policies.

Goal

Complete AiProxyEngine as the protocol-independent orchestration layer for
both streaming and non-streaming AI exchanges.

Responsibilities

  • resolve the configured or detected Protocol;
  • resolve and validate the selected Provider;
  • coordinate Protocol request decoding and Provider request adaptation;
  • execute the upstream request through Transport;
  • coordinate Provider-specific response adaptation and Protocol response
    encoding;
  • use one orchestration flow for streaming and non-streaming requests;
  • coordinate retry and fallback without moving those policies into Transport;
  • distinguish failures that occur before a streaming response starts from
    failures after downstream data has been emitted;
  • propagate cancellation through the complete reactive chain;
  • expose enough response status, headers, body, and error information for the
    plugin boundary to write the downstream response;
  • keep cross-layer policy out of Protocol, Provider, and Transport
    implementations.

Boundaries

AiProxyEngine must not:

  • depend on ServerWebExchange or write the downstream HTTP response;
  • contain OpenAI-, DeepSeek-, or other Provider-specific field mappings;
  • parse protocol-specific SSE events directly;
  • establish HTTP connections itself;
  • hide upstream failures after streaming data has already been emitted.

Acceptance criteria

  • Streaming and non-streaming requests use the same Engine entry point.
  • The complete Protocol → Provider → Transport → Provider → Protocol flow
    is coordinated by AiProxyEngine.
  • Retry and fallback occur only for eligible failures and do not duplicate
    data after a stream has started.
  • Fallback requests are rebuilt using the fallback Provider configuration.
  • Cancellation reaches the active upstream Transport subscription.
  • Upstream status, headers, bodies, and normalized failures remain
    available at the plugin boundary.
  • Engine code does not depend on Spring WebFlux server response types or
    Spring AI clients and DTOs.
  • Tests cover normal responses, streaming responses, retry, fallback,
    cancellation, unsupported Protocol/Provider combinations, and failures
    before and after stream emission.

Activity

  1. github-actions commented on Oct 10, 2026

    @github-actions
    Contributor

    👋 Hello @yuluo-yx

    A new issue has been opened by @hengyuss:

    Please review when you have time. Thank you! 🙏

  2. yangjj-iso commented on Oct 11, 2026

    @yangjj-iso
    Contributor

    /assign

  3. hengyuss commented on Oct 11, 2026

    @hengyuss
    ContributorAuthor

    Completed by merged PR #7543. AiProxyEngine now coordinates the request and response flow, streaming and non-streaming execution, retry and fallback before body emission, fallback request rebuilding, error/status/header/body propagation, and cancellation. The merged orchestration and compatibility tests cover the Engine migration. The separate Provider and Transport hardening tasks remain tracked by #7383 and #7384.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions