Repository navigation
proposal(client): let embedded LLM clients classify routing failures #865
Description
Problem
An embedded RoutedLlmClient may already know why a model call failed and whether another Switchyard candidate can safely serve it. Some of this is now covered by existing LlmClientError variants. #887 added TemporarilyUnavailable, and #922 added Host while preserving the original error. Their fallback behavior is fixed by the variant.
What is still missing is a way to report the failure kind separately from whether routing should stop or try the next target.
Proposed direction
There is a contributor API proposal in the comments. It has not landed.
Agree on the smallest provider-neutral contract that lets a client return:
- a bounded, stable failure kind for telemetry; and
- an explicit routing disposition such as
StoporNextTarget.
Existing built-in errors should keep their current behavior. Focused coverage should prove that NextTarget tries the next candidate and Stop terminates routing.
Non-goals
- adopting a large provider failure taxonomy;
- provider-exhaustion partitions or summaries;
- implementing circuit-breaker state;
- changing retry policy;
- defining new server error bodies;
- judge fail-open behavior; or
- reclassifying connection timeouts.
LlmClientError is already #[non_exhaustive], so a new variant would be an additive Rust API change for external callers.
Related work
- feat(libsy): evict targets that reject a request as unservable #345 covers built-in capability rejection and whether it should be remembered across turns.
- [feature] Skip repeatedly unavailable routing candidates with circuit breakers #594 covers a Switchyard-owned circuit-breaker state machine.
- [feature] let a classifier judge fail open on client failures and deadlines (judge_failure = "fail_open") #850 covers judge fail-open policy.
- [bug] stage_router no longer falls back to the capable target when the connection to the efficient upstream times out (0.3.0; 0.2.0 fell back) #853 covers connection-timeout fallback behavior.
ConductorOne has a working version of the broader contract in ConductorOne/Switchyard#1: typed failure contract and routing/status integration. Its full taxonomy should be treated as prior art rather than adopted wholesale. @pquerna
Activity
ting-hong-shieh commentedon Sep 30, 2026
I'd like to take this on. Before opening a PR, I'd like to confirm this minimal API:
The kind supplies a fixed telemetry label. The disposition alone controls whether the completed routing outcome tries another candidate. Stop returns the original failure. NextTarget advances if a candidate remains; otherwise it returns the last failure. The boxed source preserves the client's original error.
Existing error variants keep their fallback behavior. This leaves retries, judge failures, response-stream failures, connection-timeout classification, and server error bodies unchanged. It adds no circuit-breaker state or provider-exhaustion summary.
I have a local prototype with tests covering both dispositions across the three kinds, source preservation, and bounded telemetry labels. Rust workspace tests and workspace clippy pass.
The names and three failure kinds are proposed. Would this API and scope work for this issue?