Concurrent sessions retry a failing provider independently, causing error storms #52513

Description

When a provider starts failing (429/5xx, connection resets, unreachable endpoint), every concurrent opencode session retries it independently on its own schedule: retry state lives per session processor, with no shared per-provider health/backoff state. N parallel sessions hit the same failing endpoint at overlapping times until it recovers, and every retry/exhausted cycle publishes session.status / session.error events to all connected clients — a storm of futile requests plus a UI event flood. In daily parallel-subagent workflows this is the biggest reliability problem we run into: sessions stall, the UI looks hung, and quota is burned on requests that cannot succeed.

The numbers below come from ~/.local/share/opencode/log/opencode.log (filter message="stream error") on one machine running heavy parallel workloads — a parent session with parallel subagents, across several 1.18.x versions, with opencode's own gateway providers (opencode / opencode-go):

• 2,610 stream error entries; worst single day: 864
• one observed minute: 5+ concurrent sessions each failing every 2–5 s against the same provider

Per-session backoff works as designed (initial 2s, ×2, jitter, max 5 attempts). The missing piece is cross-session coordination:

• packages/opencode/src/session/retry.ts — the policy's state lives inside each SessionProcessor.process() call (RETRY_INITIAL_DELAY at retry.ts#L26, RETRY_MAX_RETRIES at retry.ts#L31)
• Effect.retry(SessionRetry.policy(...)) is attached per processor: processor.ts#L674 (v1.18.34). Every new processor/run re-arms the full retry budget.

Related issues — all about the per-session side, none cover cross-session coordination: #41848, #45514, #33955, #45315.

Suggested direction — shared state keyed by provider + endpoint + credential:

• 429: keep respecting Retry-After (the delay() helper already parses it) but share the resulting next-allowed time across sessions
• 5xx / connection errors: circuit breaker (consecutiveFailures, closed/open/half-open, nextAllowedAt, one half-open probe), opening after ~one exhausted policy — the logs show a full 5-attempt cycle takes ~30–60s, so the breaker would open roughly when per-session retries give up anyway; cooldown should be configurable, default open to discussion
• in-flight requests when the circuit opens: let already-streaming requests finish naturally; the breaker only gates new attempts
• while open: one deduplicated provider.status event (e.g. max 1 per provider per N seconds) instead of per-session retry events, keeping the client message clear (Provider unavailable, retrying in Ns)
• scope note: v1 should share state within one server process; cross-instance sharing needs external storage and is better left out of a first pass

Happy to attempt a PR if you can point me to where provider-level shared state would best live.

Plugins

None

OpenCode version

1.18.34 (desktop app)

Steps to reproduce

1. Configure a provider that returns 429/503 (or point it at a mock server that does).
2. Start one parent session and 3–5 parallel subagents using that provider.
3. Watch the server log and client events while the provider is failing.

Expected: a shared backoff/circuit opens; only one probe request in flight; one throttled provider status event.
Actual: each session retries on its own schedule, attempts overlap, and clients receive per-session retry/error events from every session.

Operating System

Windows 11 (build 26200)