Skip to content

[Task][AI Proxy] Harden the reactive AI Transport for production #7384

Closed

Description

@hengyuss

[Task][AI Proxy] Harden the reactive AI Transport for production

Parent issue: #7351

Depends on: #7381

Background

#7381 provides a minimal WebClientAiProxyTransport that sends a raw upstream
request and returns status, headers, and a reactive byte stream. WebClient and
Reactor Netty already provide most low-level transport capabilities; this task
configures, exposes, and verifies them for ShenYu's AI traffic rather than
reimplementing them.

Goal

Make the WebClient/Reactor Netty transport production-ready while keeping it
independent of Protocol and Provider semantics.

Scope

  • distinguish connection timeout, response-header timeout, and streaming idle
    timeout;
  • enforce request and response size policies without buffering streaming
    responses;
  • verify cancellation reaches the upstream exchange;
  • verify reactive backpressure and avoid full-body aggregation;
  • reuse or expose ShenYu proxy, TLS, connection-pool, and HTTP client settings;
  • preserve all upstream status codes, headers, and error bodies;
  • translate low-level connection, timeout, cancellation, and codec failures
    into transport-level errors that AiProxyEngine can classify;
  • ensure WebClient and Reactor Netty resources are reused correctly;
  • add tests for normal and streaming exchanges.

Boundaries

Transport must not:

  • parse OpenAI, Anthropic, or other protocol payloads;
  • map Provider-specific fields;
  • parse SSE event semantics;
  • select a Provider;
  • decide or execute retry and fallback policy.

Acceptance criteria

  • Normal and streaming bodies use the shared upstream models.
  • Streaming responses are not fully buffered.
  • Downstream cancellation propagates to the upstream exchange.
  • Connect, response-header, and streaming idle timeout behavior is
    configurable and tested.
  • Size limits do not require aggregating an entire streaming response.
  • ShenYu proxy, TLS, and connection-pool configuration can be reused.
  • Upstream status, headers, and error bodies are preserved.
  • Network failures are exposed as transport-level errors without deciding
    retry or fallback.
  • Transport has no Spring AI, Protocol-specific, or Provider-specific
    dependency.
  • Tests cover normal responses, streaming chunks, cancellation,
    backpressure, timeout, size limits, upstream errors, and metadata.

Activity

  1. changed the title [-][Task][AI Proxy] Implement protocol-independent reactive AI Transport[/-] [+][Task][AI Proxy] Harden the reactive AI Transport for production[/+] on Oct 10, 2026
  2. hengyuss commented on Oct 11, 2026

    @hengyuss
    ContributorAuthor

    Status after #7543: the merged WebClient transport uses shared upstream models, streams response chunks without full aggregation, preserves status/headers/error bodies, maps low-level failures to transport errors, and has HTTP/cancellation compatibility coverage. Keeping this issue open because the production-hardening scope remains: separately configurable connect/header/idle timeouts, reusable ShenYu proxy/TLS/connection-pool settings, streaming-safe size policies, explicit backpressure/resource-reuse verification, and broader timeout/metadata tests.

  3. hengyuss commented on Oct 11, 2026

    @hengyuss
    ContributorAuthor

    /assign

  4. deleted a comment from github-actions on Oct 11, 2026
  5. added 2 commits that reference this issue on Oct 23, 2026
  6. added a commit that references this issue on Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions