Repository navigation
Closed
Labels
Description
[Task][AI Proxy] Harden the reactive AI Transport for production
Parent issue: #7351
Depends on: #7381
Background
#7381 provides a minimal WebClientAiProxyTransport that sends a raw upstream
request and returns status, headers, and a reactive byte stream. WebClient and
Reactor Netty already provide most low-level transport capabilities; this task
configures, exposes, and verifies them for ShenYu's AI traffic rather than
reimplementing them.
Goal
Make the WebClient/Reactor Netty transport production-ready while keeping it
independent of Protocol and Provider semantics.
Scope
- distinguish connection timeout, response-header timeout, and streaming idle
timeout; - enforce request and response size policies without buffering streaming
responses; - verify cancellation reaches the upstream exchange;
- verify reactive backpressure and avoid full-body aggregation;
- reuse or expose ShenYu proxy, TLS, connection-pool, and HTTP client settings;
- preserve all upstream status codes, headers, and error bodies;
- translate low-level connection, timeout, cancellation, and codec failures
into transport-level errors thatAiProxyEnginecan classify; - ensure WebClient and Reactor Netty resources are reused correctly;
- add tests for normal and streaming exchanges.
Boundaries
Transport must not:
- parse OpenAI, Anthropic, or other protocol payloads;
- map Provider-specific fields;
- parse SSE event semantics;
- select a Provider;
- decide or execute retry and fallback policy.
Acceptance criteria
- Normal and streaming bodies use the shared upstream models.
- Streaming responses are not fully buffered.
- Downstream cancellation propagates to the upstream exchange.
- Connect, response-header, and streaming idle timeout behavior is
configurable and tested. - Size limits do not require aggregating an entire streaming response.
- ShenYu proxy, TLS, and connection-pool configuration can be reused.
- Upstream status, headers, and error bodies are preserved.
- Network failures are exposed as transport-level errors without deciding
retry or fallback. - Transport has no Spring AI, Protocol-specific, or Provider-specific
dependency. - Tests cover normal responses, streaming chunks, cancellation,
backpressure, timeout, size limits, upstream errors, and metadata.
Activity
[-][Task][AI Proxy] Implement protocol-independent reactive AI Transport[/-][+][Task][AI Proxy] Harden the reactive AI Transport for production[/+]hengyuss commentedon Oct 11, 2026
Status after #7543: the merged WebClient transport uses shared upstream models, streams response chunks without full aggregation, preserves status/headers/error bodies, maps low-level failures to transport errors, and has HTTP/cancellation compatibility coverage. Keeping this issue open because the production-hardening scope remains: separately configurable connect/header/idle timeouts, reusable ShenYu proxy/TLS/connection-pool settings, streaming-safe size policies, explicit backpressure/resource-reuse verification, and broader timeout/metadata tests.
hengyuss commentedon Oct 11, 2026
/assign
[ISSUE apache#7384] Harden reactive AI transport for production
[ISSUE apache#7384] Harden reactive AI transport for production
[ISSUE #7384] Harden reactive AI transport for production (#8034)