Repository navigation
Description
[Task] Refactor AI Proxy into protocol, provider, and transport layers
Scope update
The initial Protocol, Provider, and Transport contracts will be established
from a minimal runnable vertical slice instead of attempting to define the
complete architecture up front.
The initial contracts are intentionally small and may evolve compatibly as
response conversion, additional providers, transport policies, and the final
plugin migration are implemented.
Background
The current AI Proxy execution path is tightly coupled to OpenAI Chat
Completions and Spring AI types:
AiProxyPlugindirectly handles OpenAI request/response DTOs and SSE
encoding.AiProxyExecutorServicemixes upstream invocation, retry, and fallback
logic.- Adding protocols such as OpenAI Responses or Anthropic Messages requires
modifying the main execution flow. - Unknown provider-specific fields may be lost during DTO conversion.
This refactor refers to the layered design of Apache APISIX AI Proxy while
adapting it to ShenYu's reactive Java architecture.
Target architecture
Client request
↓
Protocol: detect, validate, and decode the client protocol
↓
Provider: construct the provider-specific upstream request
↓
Transport: execute the raw reactive HTTP exchange
↓
Provider: handle provider-specific response differences
↓
Protocol: encode the client protocol response
AiProxyEngine coordinates these layers. Retry and fallback policies belong
to the orchestration layer rather than Transport.
The final AI Proxy data path should not depend on Spring AI clients or DTOs.
Other AI plugins may continue using Spring AI.
Delivery plan
- [Task][AI Proxy] Establish initial layered contracts with a runnable OpenAI path #7381 establishes the initial contracts with a minimal runnable OpenAI Chat
→ OpenAI Provider → WebClient Transport vertical slice. It does not migrate
AiProxyPlugin. - [Task][AI Proxy] Complete and extend native AI Protocol support #7382 completes client protocol response handling, streaming semantics, and
normalized response/event/error models. - [Task][AI Proxy] Extend and harden the AI Provider layer #7383 adds provider capability handling and additional OpenAI-compatible and
DeepSeek provider behavior. - [Task][AI Proxy] Harden the reactive AI Transport for production #7384 hardens the WebClient/Reactor Netty transport with production
configuration and verification. - [Task][AI Proxy] Complete orchestration in AiProxyEngine #7532 completes
AiProxyEngineorchestration, including response flow,
retry, fallback, and cancellation. - [Task][AI Proxy] Migrate AiProxyPlugin to AiProxyEngine #7533 migrates
AiProxyPlugintoAiProxyEngine, removes the legacy AI
Proxy execution path, and verifies compatibility.
Compatibility requirements
The completed refactor must preserve:
- existing
AiProxyHandleconfiguration; - OpenAI Chat Completions behavior;
- streaming and non-streaming requests;
- proxy API key validation;
- retry and fallback behavior;
- existing Selector and Rule data;
- unknown protocol- and provider-specific fields whenever no transformation
is required.
Acceptance criteria
- Protocol, Provider, and Transport have independent contracts and
registries. - Adding a Protocol does not require changing Provider or Transport.
- Adding a Provider does not require changing the plugin execution flow.
- Streaming responses are forwarded without buffering the complete body.
- Upstream status, headers, and error bodies are preserved.
- Retry and fallback are coordinated outside Transport.
-
AiProxyPluginno longer depends on OpenAI-specific DTOs,OpenAiApi,
or other Spring AI clients after the final migration. - Existing OpenAI-compatible behavior remains covered by compatibility
tests. - Extension documentation and unit tests are provided.
Activity
github-actions commentedon Sep 28, 2026
👋 Hello @yuluo-yx
A new issue has been opened by @hengyuss:
Please review when you have time. Thank you! 🙏
hengyuss commentedon Sep 28, 2026
/assign
hengyuss commentedon Oct 11, 2026
Progress update after merged PR #7543:
Keeping this parent task open until the remaining Protocol, Provider, and Transport hardening work is completed or explicitly descoped.