Skip to content

fix: classify openrouter json 404 model errors#51573

Merged
altaywtf merged 1 commit into
openclaw:mainfrom
mrdear:fix/openrouter-404-model-fallback
Apr 13, 2026
Merged

fix: classify openrouter json 404 model errors#51573
altaywtf merged 1 commit into
openclaw:mainfrom
mrdear:fix/openrouter-404-model-fallback

Conversation

@mrdear

@mrdear mrdear commented Mar 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Problem: OpenRouter can return a JSON-wrapped HTTP 404 payload for unavailable models, and we want a stable regression guard that proves fallback still treats that payload as model_not_found.
  • Why it matters: this exact payload is easy to regress during future matcher/refactor work, and a regression would stop model selection on a dead OpenRouter candidate instead of continuing through the configured fallback chain.
  • What changed: rewrote the stale branch on top of current main and kept only regression coverage for the exact payload in the shared matcher, failover classifier, and fallback loop.
  • What did NOT change (scope boundary): no production behavior changes in this branch; current main already handles the payload correctly.

Change Type (select all)

  • Bug fix
  • Tests / regression coverage

Scope (select all touched areas)

  • Agents / fallback
  • Tests

Linked Issue/PR

User-visible / Behavior Changes

  • No new runtime behavior in this branch; it locks in existing current-main behavior with regression tests.

Security Impact (required)

  • New permissions/capabilities? No
  • Secrets/tokens handling changed? No
  • New/changed network calls? No
  • Command/tool execution surface changed? No
  • Data access scope changed? No
  • If any Yes, explain risk + mitigation: None.

Repro + Verification

Environment

  • OS: macOS
  • Runtime/container: local workspace
  • Model/provider: OpenRouter
  • Integration/channel (if any): model fallback path
  • Relevant config (redacted): model: "openrouter/healer-alpha"

Steps

  1. Construct the JSON 404 payload from OpenRouter JSON 404 payloads do not trigger model fallback #51571.
  2. Run the shared model-not-found matcher on current main.
  3. Run the failover classifier and fallback loop against the same payload.

Expected

  • The payload is recognized as model_not_found and fallback continues.

Actual

  • Current main already behaves correctly; this PR preserves that behavior with regression coverage.

Evidence

  • Targeted runtime probe on current main
  • Regression tests added and passing locally
  • Live production rollout behavior

Human Verification (required)

  • Verified scenarios: node_modules/.bin/vitest run src/agents/live-model-errors.test.ts src/agents/failover-error.test.ts src/agents/model-fallback.test.ts
  • Edge cases checked: the exact OpenRouter stealth-model JSON 404 payload is recognized; a generic JSON 404 payload is not over-classified in the shared matcher.
  • What you did not verify: full repo CI is currently noisy on unrelated current-main checks.

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

  • Backward compatible? Yes
  • Config/env changes? No
  • Migration needed? No
  • If yes, exact upgrade steps: None.

Failure Recovery (if this breaks)

  • How to disable/revert this change quickly: revert this commit
  • Files/config to restore: src/agents/live-model-errors.test.ts, src/agents/failover-error.test.ts, src/agents/model-fallback.test.ts
  • Known bad symptoms reviewers should watch for: future matcher changes stop recognizing the JSON payload or start over-classifying generic JSON 404 payloads.

Risks and Mitigations

  • Risk: the branch no longer matches the original stale implementation diff.
    • Mitigation: the branch was intentionally rebuilt on current main after verifying the underlying runtime behavior already landed upstream.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels Mar 21, 2026
@greptile-apps

greptile-apps Bot commented Mar 21, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a gap in the OpenRouter model-fallback path: JSON-wrapped HTTP 404 payloads (where the 404 is carried as a numeric error.code field rather than a leading HTTP status prefix) were not previously classified as model_not_found, so the fallback chain could stall on a dead OpenRouter candidate. The fix has two parts:

  • parseApiErrorInfo now extracts a numeric (or purely-numeric string) error.code as httpCode, so 404 inner codes are surfaced to callers.
  • classifyFailoverReason adds a branch: if the parsed httpCode is "404" and the raw string is a structured API error payload, the error is classified as model_not_found.

Key points:

  • The core fix is correct and the three test suites (unit, failover, and end-to-end fallback) together give good confidence.
  • All numeric error.code values — not just those in the HTTP 1xx–5xx range — are currently promoted to httpCode. A value like 1001 would cause formatRawAssistantErrorForUi to emit "HTTP 1001: …", which could be confusing for debugging. Guarding the range to 100–599 would be a cheap improvement.
  • The classifyFailoverReason 404 guard applies to any provider that returns a structured JSON error with a numeric code: 404, not only OpenRouter. The PR description acknowledges this risk; in practice the blast radius is limited (incorrect model_not_found classification on a non-model 404 would just trigger an unnecessary fallback rather than a hard failure).

Confidence Score: 4/5

  • Safe to merge; the fix is well-scoped and well-tested with minor style concerns remaining.
  • The core logic is correct and all three test suites pass the stated scenarios. The two P2 suggestions (HTTP status range guard, narrower 404 classification) are non-blocking improvements rather than bugs. Worst-case behaviour from the remaining concerns is an unnecessary model fallback, not a failure or data loss.
  • src/shared/assistant-error-format.ts (numeric code range guard) and src/agents/pi-embedded-helpers/errors.ts (breadth of the 404 → model_not_found rule)
Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/shared/assistant-error-format.ts
Line: 148-152

Comment:
**Non-HTTP numeric codes treated as HTTP status**

`error.code` is a provider-defined field; its value is not guaranteed to be an HTTP status code. For example, a custom code like `1001` would cause `formatRawAssistantErrorForUi` to render `"HTTP 1001: ..."`, which is misleading. Only well-known HTTP status code ranges (100–599) should be promoted to `httpCode`:

```suggestion
    if (typeof err.code === "number" && Number.isFinite(err.code) && err.code >= 100 && err.code < 600) {
      errHttpCode = String(err.code);
    } else if (typeof err.code === "string" && /^\d+$/.test(err.code.trim()) && Number(err.code.trim()) >= 100 && Number(err.code.trim()) < 600) {
      errHttpCode = err.code.trim();
    }
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/agents/pi-embedded-helpers/errors.ts
Line: 872-877

Comment:
**Broad 404 → `model_not_found` mapping may over-classify**

The guard `isRawApiErrorPayload` confirms the payload is a structured API error, but it does not scope the rule to OpenRouter (or to model-related 404s). Any provider that returns a JSON error payload with a numeric `code: 404` — even for unrelated 404s such as a missing resource, endpoint, or user — would now be classified as `model_not_found` and silently trigger a fallback.

This is mentioned in the PR's risk section. If the triggering context (provider string, a hint in the error message such as `"model"`, or a `model_not_found` type field) can be threaded through to this call site, a narrower guard would reduce the risk of false fallbacks from non-OpenRouter providers or non-model 404 responses.

How can I resolve this? If you propose a fix, please make it concise.

Last reviewed commit: "fix: classify openro..."

Comment thread src/shared/assistant-error-format.ts Outdated
Comment on lines +148 to +152
if (typeof err.code === "number" && Number.isFinite(err.code)) {
errHttpCode = String(err.code);
} else if (typeof err.code === "string" && /^\d+$/.test(err.code.trim())) {
errHttpCode = err.code.trim();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Non-HTTP numeric codes treated as HTTP status

error.code is a provider-defined field; its value is not guaranteed to be an HTTP status code. For example, a custom code like 1001 would cause formatRawAssistantErrorForUi to render "HTTP 1001: ...", which is misleading. Only well-known HTTP status code ranges (100–599) should be promoted to httpCode:

Suggested change
if (typeof err.code === "number" && Number.isFinite(err.code)) {
errHttpCode = String(err.code);
} else if (typeof err.code === "string" && /^\d+$/.test(err.code.trim())) {
errHttpCode = err.code.trim();
}
if (typeof err.code === "number" && Number.isFinite(err.code) && err.code >= 100 && err.code < 600) {
errHttpCode = String(err.code);
} else if (typeof err.code === "string" && /^\d+$/.test(err.code.trim()) && Number(err.code.trim()) >= 100 && Number(err.code.trim()) < 600) {
errHttpCode = err.code.trim();
}
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/shared/assistant-error-format.ts
Line: 148-152

Comment:
**Non-HTTP numeric codes treated as HTTP status**

`error.code` is a provider-defined field; its value is not guaranteed to be an HTTP status code. For example, a custom code like `1001` would cause `formatRawAssistantErrorForUi` to render `"HTTP 1001: ..."`, which is misleading. Only well-known HTTP status code ranges (100–599) should be promoted to `httpCode`:

```suggestion
    if (typeof err.code === "number" && Number.isFinite(err.code) && err.code >= 100 && err.code < 600) {
      errHttpCode = String(err.code);
    } else if (typeof err.code === "string" && /^\d+$/.test(err.code.trim()) && Number(err.code.trim()) >= 100 && Number(err.code.trim()) < 600) {
      errHttpCode = err.code.trim();
    }
```

How can I resolve this? If you propose a fix, please make it concise.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in commit 29bbe28a: numeric error.code is now limited to real HTTP status ranges (100-599), with a regression test covering a custom 1001 code.

Comment on lines +872 to +877
const apiInfo = parseApiErrorInfo(raw);
if (apiInfo?.httpCode === "404" && isRawApiErrorPayload(raw)) {
// Some providers, including OpenRouter, wrap model-not-found responses in
// JSON error payloads that only expose a numeric 404 code.
return "model_not_found";
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Broad 404 → model_not_found mapping may over-classify

The guard isRawApiErrorPayload confirms the payload is a structured API error, but it does not scope the rule to OpenRouter (or to model-related 404s). Any provider that returns a JSON error payload with a numeric code: 404 — even for unrelated 404s such as a missing resource, endpoint, or user — would now be classified as model_not_found and silently trigger a fallback.

This is mentioned in the PR's risk section. If the triggering context (provider string, a hint in the error message such as "model", or a model_not_found type field) can be threaded through to this call site, a narrower guard would reduce the risk of false fallbacks from non-OpenRouter providers or non-model 404 responses.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/pi-embedded-helpers/errors.ts
Line: 872-877

Comment:
**Broad 404 → `model_not_found` mapping may over-classify**

The guard `isRawApiErrorPayload` confirms the payload is a structured API error, but it does not scope the rule to OpenRouter (or to model-related 404s). Any provider that returns a JSON error payload with a numeric `code: 404` — even for unrelated 404s such as a missing resource, endpoint, or user — would now be classified as `model_not_found` and silently trigger a fallback.

This is mentioned in the PR's risk section. If the triggering context (provider string, a hint in the error message such as `"model"`, or a `model_not_found` type field) can be threaded through to this call site, a narrower guard would reduce the risk of false fallbacks from non-OpenRouter providers or non-model 404 responses.

How can I resolve this? If you propose a fix, please make it concise.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in commit 29bbe28a: the 404 fallback rule is now narrowed to OpenRouter-shaped JSON payloads with an OpenRouter URL in the message, and I added a generic JSON 404 regression test to keep the scope tight.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 86d850b421

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/shared/assistant-error-format.ts Outdated
Comment on lines +145 to +150
if (typeof err.code === "string" && !errType) {
errType = err.code;
}
if (typeof err.code === "number" && Number.isFinite(err.code)) {
errHttpCode = String(err.code);
} else if (typeof err.code === "string" && /^\d+$/.test(err.code.trim())) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Do not treat numeric-string error.code as both type and status

parseApiErrorInfo() still assigns any string error.code to errType, and this new branch also promotes purely numeric string codes to httpCode. That makes already-supported payloads like {"error":{"code":"404","message":"..."}} format as HTTP 404 404: ... via formatRawAssistantErrorForUi(), and observation fields now record providerErrorType: "404" even though the code is being used as the HTTP status. This regression is reachable because parseApiErrorPayload() explicitly recognizes string-valued error.code payloads today.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 29bbe28a0e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +874 to +876
apiInfo?.httpCode === "404" &&
isRawApiErrorPayload(raw) &&
typeof apiInfo.message === "string" &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Accept status-prefixed JSON before testing for raw payload

When OpenRouter's 404 body arrives in the same 404 {...} form that we already handle elsewhere (for example the 429 {...} shape covered by formatRawAssistantErrorForUi() tests), parseApiErrorInfo() extracts httpCode, but isRawApiErrorPayload(raw) still returns false because it only parses strings that start with {. In that case isOpenRouterModel404 stays false, so resolveFailoverReasonFromError() still treats the failure as unknown and the fallback chain stops on the unavailable model.

Useful? React with 👍 / 👎.

@mrdear
mrdear force-pushed the fix/openrouter-404-model-fallback branch from 29bbe28 to b272626 Compare April 4, 2026 05:38

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b27262621b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/shared/assistant-error-format.ts Outdated
Comment on lines +162 to +165
} else if (typeof err.code === "string" && /^\d+$/.test(err.code.trim())) {
const normalizedCode = Number(err.code.trim());
if (normalizedCode >= 100 && normalizedCode < 600) {
errHttpCode = err.code.trim();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid double-classifying numeric string error codes

This change promotes digit-only string error.code values to httpCode, but the existing errType = err.code path still runs for the same payloads, so inputs like {"error":{"code":"404","message":"..."}} now format as HTTP 404 404: ... and telemetry records providerErrorType: "404" even though the value is being treated as status. That is a behavior regression for already-supported payload shapes and makes both UI copy and error observation fields misleading.

Useful? React with 👍 / 👎.

Comment on lines +1189 to +1190
if (!isRawApiErrorPayload(raw)) {
return false;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Parse status-prefixed JSON before OpenRouter 404 gating

isOpenRouterModel404Payload currently bails out unless isRawApiErrorPayload(raw) succeeds, but that predicate rejects strings with a leading status token (for example 404 {"error":...}), even though parseApiErrorInfo can parse that exact shape. In this case the new OpenRouter classifier never runs, so resolveFailoverReasonFromError can still miss model_not_found and stop fallback on an unavailable model.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5fca3951db

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/shared/assistant-error-format.ts Outdated
if (typeof err.message === "string") {
errMessage = err.message;
}
}

return {
httpCode,
httpCode: errHttpCode ?? httpCode,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep prefixed HTTP status authoritative over payload code

parseApiErrorInfo now returns errHttpCode before the already-parsed leading status (httpCode: errHttpCode ?? httpCode), so supported <status> <json> errors can be re-labeled with a different code from error.code. In practice, an input like 429 {"error":{"code":"404","message":"..."}} is now formatted/observed as HTTP 404, which can mislead users and downstream classification that relies on parseApiErrorInfo. The leading transport status should remain authoritative, and error.code should only be used when no prefix status was present.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: db230a283b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +1268 to +1271
apiInfo?.httpCode === "404" &&
typeof apiInfo.message === "string" &&
apiInfo.message.toLowerCase().includes("openrouter.ai")
) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restrict OpenRouter 404 URL heuristic to model-not-found cases

isModelNotFoundErrorMessage() now treats any parsed 404 payload whose message contains openrouter.ai as model_not_found, but this check is not tied to provider context or model-specific wording. As a result, non-model OpenRouter 404s (for example account/route/docs links in error text) can be misclassified and drive fallback down the wrong path instead of surfacing the real failure reason.

Useful? React with 👍 / 👎.

@altaywtf altaywtf self-assigned this Apr 13, 2026
@altaywtf
altaywtf force-pushed the fix/openrouter-404-model-fallback branch from db230a2 to 250a6fe Compare April 13, 2026 18:09
@altaywtf
altaywtf merged commit 95ee120 into openclaw:main Apr 13, 2026
37 of 46 checks passed
@altaywtf

Copy link
Copy Markdown
Member

Merged as 95ee120.

This landed as the rewritten current-main regression-only version of the PR after verifying the runtime behavior was already fixed upstream and confirming the added regression coverage locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling size: XS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenRouter JSON 404 payloads do not trigger model fallback

2 participants