Skip to content

fix(agent): a 403 whose error body self-identifies as server_error should be retried, not treated as a credential refusal #117869

New issue
New issue
Open
#134714
#117874
Open
fix(agent): a 403 whose error body self-identifies as server_error should be retried, not treated as a credential refusal#117869
#134714
#117874
Labels
P2Medium — degraded but workaround existsarea/authAuthentication, OAuth, credential poolscomp/agentCore agent runtime: loop, agent_init, prompt builder, context-compression, responses endpointprovider/otherCalling models through a small provider without its own label (for example Kilo Code, Hugging Face Itype/bugSomething isn't working

Description

@LuisLiuwh
LuisLiuwh
opened on Sep 21, 2026

fix(agent): a 403 whose error body self-identifies as server_error should be retried, not treated as a credential refusal

Summary

OpenCode Go (an OpenAI-compatible relay at https://opencode.ai/zen/go/v1) wraps transient upstream failures as:

HTTP 403
{"error": {"type": "server_error", "code": "server_error",
           "message": "Upstream request failed: [server_error] Upstream response was not valid JSON"}}

classify_api_error maps this to FailoverReason.auth / retryable=False, so the turn aborts on attempt 1 and never uses its retry budget. Observed twice on Sep 21 (UTC), both times killing a scheduled cron job mid-run:

00:40:28 ERROR Streaming failed before delivery: Error code: 403 - {'error': {'type': 'server_error', ...}}
00:40:29 WARNING API call failed (attempt 1/3) error_type=PermissionDeniedError
00:40:29 ERROR Non-retryable client error: ...
00:40:29 WARNING Job '…': session ended without a final assistant message (lifecycle=interrupted)

The same relay serves the same request successfully seconds later — it's a blip, not a credential problem. (The Go docs themselves label this error class "Upstream response was not valid JSON".)

Root cause

agent/error_classifier.py::_status_403 has a transient-code escape hatch:

_403_TRANSIENT_CODES = frozenset({"upstream_unavailable"})

added by #115796 (merged 2026-09-19, resolving case 1 of #75388). OpenCode's structured code for the same situation is server_error — a different literal, so it falls through to the generic auth default.

Proposed fix (one line)

_403_TRANSIENT_CODES = frozenset({"upstream_unavailable", "server_error"})

Reproduction & verification (live, on the installed v0.21.3 tree)

classify_api_error(err, provider="opencode-go", model="glm-5.3-flash", base_url="https://opencode.ai/zen/go/v1"):

403 body before after (proposed)
{"type": "server_error", "code": "server_error", "message": "Upstream request failed: …"} auth, retryable=False overloaded, retryable=True
{"message": "Forbidden", "type": "permission_error"} (real credential refusal — control) auth, retryable=False auth, retryable=False (unchanged)
{"code": "insufficient_quota"} (billing — control) billing, retryable=False billing, retryable=False (unchanged)
{"code": "upstream_unavailable"} (existing transient — control) overloaded, retryable=True overloaded, retryable=True (unchanged)

So the change widens the retry to exactly the class #75388/#115796 already established (structured server-side error on a 403 = transient), while real credential refusals (permission_error, access denied, WAF blocks) and billing 403s keep their existing verdicts.

Why this matters beyond one relay

The failure mode is not OpenCode-specific: any OpenAI-compatible aggregator that fronts flaky upstreams can legitimately answer 403 with a server-side error body. Today the agent (a) burns the whole cron/agent turn, (b) benches/rotates the credential via the credential pool (marking … exhausted (status=403), rotating), and (c) skips fallback_providers, because non-retryable client errors settle before the fallback walk. All three are wrong for a transient upstream hiccup.

Environment

Activity

  1. kokhlo commented on Sep 21, 2026

    @kokhlo
    kokhlo
    on Sep 21, 2026

    Confirmed on main: agent/error_classifier.py:1000 keeps _403_TRANSIENT_CODES = frozenset({"upstream_unavailable"}), and _status_403 (line 1003) returns _V_OVERLOADED only for that literal. OpenCode's server_error code falls through the billing scan and the WAF check into _V_AUTH_FALLBACK — which is what aborts the turn on attempt 1 and benches a healthy credential.

    The fix widens exactly the class #115796 established for #75388: a structured server-side error code on a 403 means "upstream is down, retry later", so server_error joins the transient set. Real credential refusals (permission_error, access denied, WAF blocks) and billing 403s (insufficient_quota, key limit exceeded) never carry that code, so all the control cases in the report keep their verdicts.

    Plan: one-line set extension + a regression case next to the existing upstream_unavailable tests (403 body {"type": "server_error", "code": "server_error"} → overloaded/retryable; permission_error and insufficient_quota controls unchanged). PR within the hour.

  2. alt-glitch
    added
    type/bugSomething isn't working
    P2Medium — degraded but workaround exists
    comp/agentCore agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint
    area/authAuthentication, OAuth, credential pools
    on Sep 21, 2026

  3. Sahilvishnaliya commented on Sep 21, 2026

    @Sahilvishnaliya
    Sahilvishnaliya
    on Sep 21, 2026

    Claiming this — working on a PR. @Sahilvishnaliya

  4. Sahilvishnaliya commented on Sep 21, 2026

    @Sahilvishnaliya
    Sahilvishnaliya
    on Sep 21, 2026

    Claiming this — working on a PR. @Sahilvishnaliya

  5. foxhoundhcm commented on Sep 21, 2026

    @foxhoundhcm
    foxhoundhcm
    on Sep 21, 2026

    Independent confirmation from a Windows desktop install — provider opencode-go (https://opencode.ai/zen/go/v1, model deepseek-v4.1-flash), same body every time:

    HTTP 403 {'error': {'type': 'server_error', 'code': 'server_error',
  'message': 'Error from provider (Console Go): Upstream request failed:
              [server_error] Upstream response was not valid JSON'}}
API call failed (attempt 1/3, not retryable) error_type=PermissionDeniedError
Non-retryable client error: ...

    8 occurrences in one day (2026-09-21, UTC+8) in 6 clusters — 09:32, 09:38, 10:07, 19:46, 20:51, 20:56 — so this is a steady background rate, not a one-off. Two user-visible consequences that the fix should account for:

    1. The UI blames the credential. _V_AUTH_FALLBACK routes the turn to the canned auth copy (agent/turn_failure_copy.py:183, _AUTH_COPY["api_key"]), so the desktop reports "OpenCode Go rejected your API key — The key saved for OpenCode Go is invalid or was revoked. Update it, then retry." and offers an Update API key button for what is a relay blip. Users are being told to rotate a healthy key.
    2. The pool bench is why it looks like it heals itself, and why it can outlive the blip. credential_pool._mark_exhausted persists last_status=exhausted, last_error_code=403, last_error_reason=server_error, failure_reason=auth into auth.json, and _exhausted_ttl_seconds gives a non-billing 403 on a sole credential EXHAUSTED_TTL_SOLE_CREDENTIAL_SECONDS (60s) — so the retry a minute later succeeds and the operator sees a random failure that "goes away by itself" (as reported here), while every intervening turn on that attempt dies outright.

    A/B on the installed tree, real body, classify_api_error(..., provider="opencode-go", base_url="https://opencode.ai/zen/go/v1"):

    _403_TRANSIENT_CODES verdict retryable rotate_credential
    {"upstream_unavailable"} (current) auth False False
    + "server_error" overloaded True False

    Same direction as the report, and the retry budget is then spent as intended. One note for whoever lands the fix: whether the code is honoured globally or only for this relay is your call, but the ordering matters — in the current _status_403 the transient-code check runs before the billing and WAF checks, so a global set extension would also shadow a 403 that carries billing prose alongside code=server_error. Both open PRs (#117871 provider-scoped, #117874 wrapper-marker-gated) sidestep that by reading the transient code last; either shape fixes our case, so we're only asking for the ordering to stay deliberate.

    We're running the one-line set extension as a local patch until this lands, and will test a merged build against the same relay — happy to report back.

  6. weidzhou commented on Sep 21, 2026

    @weidzhou
    weidzhou
    on Sep 21, 2026

    Independent user-side reproduction on v0.21.3 (2026.9.14), plus one observation the merged fix does not cover.

    The gap: #115796 (merged 2026-09-19) retries a 403 stamped code: upstream_unavailable. What this relay actually sends is code: server_error:

    HTTP 403 {'error': {'type': 'server_error', 'code': 'server_error',
                    'message': 'Upstream request failed: [server_error] Upstream response was not valid JSON'}}

    So the merged wording doesn't fire here, and the healthy credential gets benched.

    Evidence chain — 2026-09-21 22:52:39, platform=webui, provider=opencode-go, model=deepseek-v4.1-flash

    1. The 403 is surfaced as non-retryable:
      ERROR agent.conversation_loop: Non-retryable client error: Error code: 403 - {...}
    2. The pool benches the healthy primary credential immediately: 
      $ hermes -p <profile> auth list
opencode-go (2 credentials):
  #1  <primary-key>   api_key priority=0 manual    auth failed server_error (403) (re-auth may be required)
  #2  <fallback-key>  api_key priority=1 env:…      rate-limited (429) (52m 28s left)
      The 403 body carries no reset_at, so _exhausted_ttl(403) → EXHAUSTED_TTL_DEFAULT_SECONDS = 3600 s of being skipped — and auth list prints “re-auth may be required” while hiding the window (hermes_cli/auth_commands.py::_EXHAUSTED_CLASSES maps {401,403} → label auth failed, show_retry_window=False).
    3. Traffic then falls through to the genuinely exhausted fallback credential:
      429 GoUsageLimitError … metadata={'limitName': 'monthly'} → Retrying API call in 600s
      ⇒ the UI shows “processing” with no explanation for up to an hour.
    4. The benched credential was healthy all along: the same key + model returned 200 OK minutes later, and the copy of the same key in another profile's pool remained healthy throughout.
    5. Reproduced twice in one day (14:20 and 22:52 local), both on long-lived sessions.

    Ask — beyond retryability (#117874 / #117871 / #117911), it would help to also:

    • (a) treat code: server_error as transient in credential_pool so the credential is not benched for 3600 s, and
    • (b) stop rendering that case as auth failed … (re-auth may be required) (it sends operators hunting a key problem that doesn't exist).

    Filed as #118286 (closed as duplicate of this issue).

  7. Fission21 commented on Sep 22, 2026

    @Fission21
    Fission21
    on Sep 22, 2026

    Third independent reproduction — desktop install, plus a direct proof the credential was never the problem

    Same relay, same body, provider=opencode-go / model=deepseek-v4.1-flash, on v0.21.3 (2026.9.14), commit fbc4ea8b96 (macOS, desktop app).

    Our timeline (UTC+8): 2026-09-21 14:58 / 15:04 / 17:11 — three clusters, 15 log hits, zero occurrences before or after in the retained logs. In UTC that is 06:58 / 07:04 / 09:11, filling the gap between the clusters already reported in this thread.

    The chain is exactly as described:

    403 -> PermissionDeniedError (attempt 1/3, not retryable)
agent.credential_pool: marking OPENCODE_GO_API_KEY exhausted (status=403), rotating
agent.credential_pool: no available entries (all exhausted or empty)   # single-key pool
tui turn finished: status=error failure_reason=auth

    What I'd add: a direct check of the credential. Using the same key, minutes after the last failure and again the next morning:

    GET https://opencode.ai/zen/go/v1/models   ->   HTTP 200, 33 models listed

    So the "bench the credential" half of this bug is demonstrably wrong: a single-key pool goes to no available entries after a blip, and the user-facing advice ("Update API key") recommends the one action that cannot help.

    User-visible impact beyond the aborted turn: failure_reason=auth drives the desktop banner "The key saved for OpenCode Go is invalid or was revoked. Update it, then retry." — copy in apps/desktop/src/i18n/en.ts:4217, agent side agent/turn_failure_copy.py:185 — with an Update API key action button. Retrying the turn works; rotating the key does not.

    Happy to re-test once a PR lands. Our tree already carries the _403_TRANSIENT_CODES escape hatch from #115796 (agent/error_classifier.py:1000), so I can flip the set locally, re-run the classifier cases, and run a live turn against the relay.

  8. Hamakaze1s commented on Oct 18, 2026

    @Hamakaze1s
    Hamakaze1s
    on Oct 18, 2026

    Independent field evidence for the same relay envelope, plus one body in that family this change would swallow.

    Environment: hermes-agent local-patches on a 3068d1c177-era main (Linux, opencode-go via https://opencode.ai/zen/go/v1, model deepseek-v4.1-flash). No keys or account details below.

    The relay reuses this envelope for a second, unrelated condition. Same HTTP 403, same error.type/error.code == "server_error", same Upstream request failed: wrapper — only the message text differs:

    1) {"error":{"type":"server_error","code":"server_error","message":"Upstream request failed: [server_error] Upstream response was not valid JSON"}}
2) {"error":{"type":"server_error","message":"Upstream request failed: An active OpenCode Go subscription is required to use Go models."}}

    (1) is the blip this issue is about. (2) is an account that has no Go subscription — that credential cannot serve Go traffic at all, so it must be benched and rotated away.

    What (2) does today, measured through the production entry points (classify_api_error for the verdict; recover_with_credential_pool with a real two-entry CredentialPool for the pool reaction, the subscription-less key at priority 0):

    body verdict pool reaction
    (1) blip auth, retryable=False entry marked exhausted (403, failure_reason=auth) → rotated to entry #2
    (2) no subscription auth, retryable=False same — benched and rotated
    401 This API key has expired… (control) auth, should_rotate=True benched (401) → rotated

    So today both bodies bench a credential for an hour — the behaviour you report for (1) — and (2) happens to land on the correct recovery (rotate away from a key that cannot serve Go traffic) purely because core's 403 default benches.

    Why the fix shape matters for (2). A check keyed on code == "server_error" alone — or on code plus the Upstream request failed wrapper — claims (2) as well and makes it overloaded: recover_with_credential_pool has no branch for it (agent/agent_runtime_helpers.py:1098-1133), so the key is never benched and stays selected, spending the retry budget on every call. That is strictly worse than today. Note _code_from_payload (agent/error_classifier.py:1461) falls back to error.type, so body (2) — which carries no code — still arrives as server_error at any code-only predicate.

    Suggested shape for whoever lands this: keep the transient classification for (1), but claim (2) in a provider-scoped branch ordered before the code branch. The ProviderProfile.classify_api_error seam already runs ahead of _status_403 (_profile_verdict is the second stage of _STAGES), e.g.

    if status_code == 403 and "subscription is required" in message.lower():
    return {"reason": "billing", "retryable": False,
            "should_rotate_credential": True, "should_fallback": True}
return None  # (1) keeps whatever the built-in pipeline decides

    billing (rather than model_entitlement, which benches the credential+model pair for a year and clears only through hermes auth reset, or a DEAD row, which drops a manual entry after 24h) keeps the one-hour bench on single-key pools too and re-arms the per-turn revert hook, so a renewed subscription recovers without manual intervention.

    Zero-code workaround for anyone hitting (2) today: hermes auth priority opencode-go <subscribed-key> 0 puts the working key first.

    Repro is pure-local (no network): feed the two bodies to classify_api_error(err, provider="opencode-go", model=<id>, base_url=<relay>), and for the pool half drive recover_with_credential_pool with a two-entry CredentialPool. Happy to share the probe script if useful.

  9. alt-glitch
    added
    provider/otherCalling models through a small provider without its own label (for example Kilo Code, Hugging Face I
    on Oct 18, 2026

  10. alt-glitch commented on Oct 18, 2026

    @alt-glitch
    alt-glitch
    on Oct 18, 2026

    #117871 aims to fix this: it adds the server error code to the list of transient 403 codes, so a relay 403 marked as a server error gets retried instead of being treated as a credential refusal.

  11. kvnloo
    added a commit that references this issue on Nov 9, 2026

    fix(opencode): preserve Go upstream 429 when retrying relay 403

    1db3faf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existsarea/authAuthentication, OAuth, credential poolscomp/agentCore agent runtime: loop, agent_init, prompt builder, context-compression, responses endpointprovider/otherCalling models through a small provider without its own label (for example Kilo Code, Hugging Face Itype/bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions