Skip to content

[Task] Refactor AI Proxy into protocol, provider, and transport layers #7351

Open
6 of 8 issues completed

Description

@hengyuss

[Task] Refactor AI Proxy into protocol, provider, and transport layers

Scope update

The initial Protocol, Provider, and Transport contracts will be established
from a minimal runnable vertical slice instead of attempting to define the
complete architecture up front.

The initial contracts are intentionally small and may evolve compatibly as
response conversion, additional providers, transport policies, and the final
plugin migration are implemented.

Background

The current AI Proxy execution path is tightly coupled to OpenAI Chat
Completions and Spring AI types:

  • AiProxyPlugin directly handles OpenAI request/response DTOs and SSE
    encoding.
  • AiProxyExecutorService mixes upstream invocation, retry, and fallback
    logic.
  • Adding protocols such as OpenAI Responses or Anthropic Messages requires
    modifying the main execution flow.
  • Unknown provider-specific fields may be lost during DTO conversion.

This refactor refers to the layered design of Apache APISIX AI Proxy while
adapting it to ShenYu's reactive Java architecture.

Target architecture

Client request
      ↓
Protocol: detect, validate, and decode the client protocol
      ↓
Provider: construct the provider-specific upstream request
      ↓
Transport: execute the raw reactive HTTP exchange
      ↓
Provider: handle provider-specific response differences
      ↓
Protocol: encode the client protocol response

AiProxyEngine coordinates these layers. Retry and fallback policies belong
to the orchestration layer rather than Transport.

The final AI Proxy data path should not depend on Spring AI clients or DTOs.
Other AI plugins may continue using Spring AI.

Delivery plan

Compatibility requirements

The completed refactor must preserve:

  • existing AiProxyHandle configuration;
  • OpenAI Chat Completions behavior;
  • streaming and non-streaming requests;
  • proxy API key validation;
  • retry and fallback behavior;
  • existing Selector and Rule data;
  • unknown protocol- and provider-specific fields whenever no transformation
    is required.

Acceptance criteria

  • Protocol, Provider, and Transport have independent contracts and
    registries.
  • Adding a Protocol does not require changing Provider or Transport.
  • Adding a Provider does not require changing the plugin execution flow.
  • Streaming responses are forwarded without buffering the complete body.
  • Upstream status, headers, and error bodies are preserved.
  • Retry and fallback are coordinated outside Transport.
  • AiProxyPlugin no longer depends on OpenAI-specific DTOs, OpenAiApi,
    or other Spring AI clients after the final migration.
  • Existing OpenAI-compatible behavior remains covered by compatibility
    tests.
  • Extension documentation and unit tests are provided.

Activity

  1. github-actions commented on Sep 28, 2026

    @github-actions
    Contributor

    👋 Hello @yuluo-yx

    A new issue has been opened by @hengyuss:

    Please review when you have time. Thank you! 🙏

  2. hengyuss commented on Sep 28, 2026

    @hengyuss
    ContributorAuthor

    /assign

  3. hengyuss commented on Oct 11, 2026

    @hengyuss
    ContributorAuthor

    Progress update after merged PR #7543:

    Keeping this parent task open until the remaining Protocol, Provider, and Transport hardening work is completed or explicitly descoped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions