Repository navigation
Closed
Labels
Description
[Task][AI Proxy] Complete orchestration in AiProxyEngine
Parent issue: #7351
Depends on: #7382, #7383, #7384
Background
#7381 introduces a minimal AiProxyEngine that validates the initial layer
boundaries by decoding an OpenAI Chat request, constructing an OpenAI upstream
request, and delegating it to Transport.
After Protocol response handling, Provider adaptation, and production
Transport behavior are completed, the Engine must coordinate the complete
request and response lifecycle and own cross-layer execution policies.
Goal
Complete AiProxyEngine as the protocol-independent orchestration layer for
both streaming and non-streaming AI exchanges.
Responsibilities
- resolve the configured or detected Protocol;
- resolve and validate the selected Provider;
- coordinate Protocol request decoding and Provider request adaptation;
- execute the upstream request through Transport;
- coordinate Provider-specific response adaptation and Protocol response
encoding; - use one orchestration flow for streaming and non-streaming requests;
- coordinate retry and fallback without moving those policies into Transport;
- distinguish failures that occur before a streaming response starts from
failures after downstream data has been emitted; - propagate cancellation through the complete reactive chain;
- expose enough response status, headers, body, and error information for the
plugin boundary to write the downstream response; - keep cross-layer policy out of Protocol, Provider, and Transport
implementations.
Boundaries
AiProxyEngine must not:
- depend on
ServerWebExchangeor write the downstream HTTP response; - contain OpenAI-, DeepSeek-, or other Provider-specific field mappings;
- parse protocol-specific SSE events directly;
- establish HTTP connections itself;
- hide upstream failures after streaming data has already been emitted.
Acceptance criteria
- Streaming and non-streaming requests use the same Engine entry point.
- The complete Protocol → Provider → Transport → Provider → Protocol flow
is coordinated byAiProxyEngine. - Retry and fallback occur only for eligible failures and do not duplicate
data after a stream has started. - Fallback requests are rebuilt using the fallback Provider configuration.
- Cancellation reaches the active upstream Transport subscription.
- Upstream status, headers, bodies, and normalized failures remain
available at the plugin boundary. - Engine code does not depend on Spring WebFlux server response types or
Spring AI clients and DTOs. - Tests cover normal responses, streaming responses, retry, fallback,
cancellation, unsupported Protocol/Provider combinations, and failures
before and after stream emission.
Activity
github-actions commentedon Oct 10, 2026
👋 Hello @yuluo-yx
A new issue has been opened by @hengyuss:
Please review when you have time. Thank you! 🙏
yangjj-iso commentedon Oct 11, 2026
/assign
hengyuss commentedon Oct 11, 2026
Completed by merged PR #7543. AiProxyEngine now coordinates the request and response flow, streaming and non-streaming execution, retry and fallback before body emission, fallback request rebuilding, error/status/header/body propagation, and cancellation. The merged orchestration and compatibility tests cover the Engine migration. The separate Provider and Transport hardening tasks remain tracked by #7383 and #7384.