Skip to content

[Task][AI Proxy] Harden configurable retry and fallback policy in AiProxyEngine #7667

Closed

Description

@hengyuss

Background

Follow up on #7532 to make retry and fallback decisions explicit, configurable, and independently testable. Retry and fallback remain orchestration responsibilities of AiProxyEngine; Transport only reports classified failures.

Scope

  • Extract HTTP status and transport failure decisions into AiProxyRetryPolicy.
  • Define explicit retry and fallback eligibility for upstream HTTP statuses.
  • Define explicit retry and fallback eligibility for classified transport failures.
  • Configure retry limits independently for non-streaming and streaming requests.
  • Support configurable exponential backoff for non-streaming retries.
  • Retry the primary Provider before attempting fallback, and attempt fallback at most once.
  • Rebuild fallback requests using the fallback Provider configuration.
  • Prevent retry or fallback after response data has been emitted.
  • Add unit tests for decision matrices, configuration binding, backoff, fallback, and streaming boundaries.

Out of Scope

  • Connection pool, timeout, TLS verification, HTTP proxy, and transport lifecycle configuration.
  • Health-based multi-provider load balancing or automatic Provider selection.
  • Mapping terminal transport failures to gateway HTTP 502/503/504 responses.
  • Protocol and Provider request/response adaptation.

Acceptance Criteria

  • Retry and fallback decisions are centralized and independently testable.
  • Only explicitly eligible statuses and transport failures are retried or switched to fallback.
  • Retry counts and initial backoff can be configured through Spring Boot configuration.
  • Streaming retries happen only before the first response chunk is emitted.
  • No retry or fallback occurs after downstream response emission starts.
  • The fallback Provider is attempted at most once.

Related Issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions