fix: classify openrouter json 404 model errors#51573
Conversation
Greptile SummaryThis PR fixes a gap in the OpenRouter model-fallback path: JSON-wrapped HTTP 404 payloads (where the 404 is carried as a numeric
Key points:
Confidence Score: 4/5
Prompt To Fix All With AIThis is a comment left during a code review.
Path: src/shared/assistant-error-format.ts
Line: 148-152
Comment:
**Non-HTTP numeric codes treated as HTTP status**
`error.code` is a provider-defined field; its value is not guaranteed to be an HTTP status code. For example, a custom code like `1001` would cause `formatRawAssistantErrorForUi` to render `"HTTP 1001: ..."`, which is misleading. Only well-known HTTP status code ranges (100–599) should be promoted to `httpCode`:
```suggestion
if (typeof err.code === "number" && Number.isFinite(err.code) && err.code >= 100 && err.code < 600) {
errHttpCode = String(err.code);
} else if (typeof err.code === "string" && /^\d+$/.test(err.code.trim()) && Number(err.code.trim()) >= 100 && Number(err.code.trim()) < 600) {
errHttpCode = err.code.trim();
}
```
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: src/agents/pi-embedded-helpers/errors.ts
Line: 872-877
Comment:
**Broad 404 → `model_not_found` mapping may over-classify**
The guard `isRawApiErrorPayload` confirms the payload is a structured API error, but it does not scope the rule to OpenRouter (or to model-related 404s). Any provider that returns a JSON error payload with a numeric `code: 404` — even for unrelated 404s such as a missing resource, endpoint, or user — would now be classified as `model_not_found` and silently trigger a fallback.
This is mentioned in the PR's risk section. If the triggering context (provider string, a hint in the error message such as `"model"`, or a `model_not_found` type field) can be threaded through to this call site, a narrower guard would reduce the risk of false fallbacks from non-OpenRouter providers or non-model 404 responses.
How can I resolve this? If you propose a fix, please make it concise.Last reviewed commit: "fix: classify openro..." |
| if (typeof err.code === "number" && Number.isFinite(err.code)) { | ||
| errHttpCode = String(err.code); | ||
| } else if (typeof err.code === "string" && /^\d+$/.test(err.code.trim())) { | ||
| errHttpCode = err.code.trim(); | ||
| } |
There was a problem hiding this comment.
Non-HTTP numeric codes treated as HTTP status
error.code is a provider-defined field; its value is not guaranteed to be an HTTP status code. For example, a custom code like 1001 would cause formatRawAssistantErrorForUi to render "HTTP 1001: ...", which is misleading. Only well-known HTTP status code ranges (100–599) should be promoted to httpCode:
| if (typeof err.code === "number" && Number.isFinite(err.code)) { | |
| errHttpCode = String(err.code); | |
| } else if (typeof err.code === "string" && /^\d+$/.test(err.code.trim())) { | |
| errHttpCode = err.code.trim(); | |
| } | |
| if (typeof err.code === "number" && Number.isFinite(err.code) && err.code >= 100 && err.code < 600) { | |
| errHttpCode = String(err.code); | |
| } else if (typeof err.code === "string" && /^\d+$/.test(err.code.trim()) && Number(err.code.trim()) >= 100 && Number(err.code.trim()) < 600) { | |
| errHttpCode = err.code.trim(); | |
| } |
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/shared/assistant-error-format.ts
Line: 148-152
Comment:
**Non-HTTP numeric codes treated as HTTP status**
`error.code` is a provider-defined field; its value is not guaranteed to be an HTTP status code. For example, a custom code like `1001` would cause `formatRawAssistantErrorForUi` to render `"HTTP 1001: ..."`, which is misleading. Only well-known HTTP status code ranges (100–599) should be promoted to `httpCode`:
```suggestion
if (typeof err.code === "number" && Number.isFinite(err.code) && err.code >= 100 && err.code < 600) {
errHttpCode = String(err.code);
} else if (typeof err.code === "string" && /^\d+$/.test(err.code.trim()) && Number(err.code.trim()) >= 100 && Number(err.code.trim()) < 600) {
errHttpCode = err.code.trim();
}
```
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
Fixed in commit 29bbe28a: numeric error.code is now limited to real HTTP status ranges (100-599), with a regression test covering a custom 1001 code.
| const apiInfo = parseApiErrorInfo(raw); | ||
| if (apiInfo?.httpCode === "404" && isRawApiErrorPayload(raw)) { | ||
| // Some providers, including OpenRouter, wrap model-not-found responses in | ||
| // JSON error payloads that only expose a numeric 404 code. | ||
| return "model_not_found"; | ||
| } |
There was a problem hiding this comment.
Broad 404 →
model_not_found mapping may over-classify
The guard isRawApiErrorPayload confirms the payload is a structured API error, but it does not scope the rule to OpenRouter (or to model-related 404s). Any provider that returns a JSON error payload with a numeric code: 404 — even for unrelated 404s such as a missing resource, endpoint, or user — would now be classified as model_not_found and silently trigger a fallback.
This is mentioned in the PR's risk section. If the triggering context (provider string, a hint in the error message such as "model", or a model_not_found type field) can be threaded through to this call site, a narrower guard would reduce the risk of false fallbacks from non-OpenRouter providers or non-model 404 responses.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/pi-embedded-helpers/errors.ts
Line: 872-877
Comment:
**Broad 404 → `model_not_found` mapping may over-classify**
The guard `isRawApiErrorPayload` confirms the payload is a structured API error, but it does not scope the rule to OpenRouter (or to model-related 404s). Any provider that returns a JSON error payload with a numeric `code: 404` — even for unrelated 404s such as a missing resource, endpoint, or user — would now be classified as `model_not_found` and silently trigger a fallback.
This is mentioned in the PR's risk section. If the triggering context (provider string, a hint in the error message such as `"model"`, or a `model_not_found` type field) can be threaded through to this call site, a narrower guard would reduce the risk of false fallbacks from non-OpenRouter providers or non-model 404 responses.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
Fixed in commit 29bbe28a: the 404 fallback rule is now narrowed to OpenRouter-shaped JSON payloads with an OpenRouter URL in the message, and I added a generic JSON 404 regression test to keep the scope tight.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 86d850b421
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (typeof err.code === "string" && !errType) { | ||
| errType = err.code; | ||
| } | ||
| if (typeof err.code === "number" && Number.isFinite(err.code)) { | ||
| errHttpCode = String(err.code); | ||
| } else if (typeof err.code === "string" && /^\d+$/.test(err.code.trim())) { |
There was a problem hiding this comment.
Do not treat numeric-string
error.code as both type and status
parseApiErrorInfo() still assigns any string error.code to errType, and this new branch also promotes purely numeric string codes to httpCode. That makes already-supported payloads like {"error":{"code":"404","message":"..."}} format as HTTP 404 404: ... via formatRawAssistantErrorForUi(), and observation fields now record providerErrorType: "404" even though the code is being used as the HTTP status. This regression is reachable because parseApiErrorPayload() explicitly recognizes string-valued error.code payloads today.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 29bbe28a0e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| apiInfo?.httpCode === "404" && | ||
| isRawApiErrorPayload(raw) && | ||
| typeof apiInfo.message === "string" && |
There was a problem hiding this comment.
Accept status-prefixed JSON before testing for raw payload
When OpenRouter's 404 body arrives in the same 404 {...} form that we already handle elsewhere (for example the 429 {...} shape covered by formatRawAssistantErrorForUi() tests), parseApiErrorInfo() extracts httpCode, but isRawApiErrorPayload(raw) still returns false because it only parses strings that start with {. In that case isOpenRouterModel404 stays false, so resolveFailoverReasonFromError() still treats the failure as unknown and the fallback chain stops on the unavailable model.
Useful? React with 👍 / 👎.
29bbe28 to
b272626
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b27262621b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| } else if (typeof err.code === "string" && /^\d+$/.test(err.code.trim())) { | ||
| const normalizedCode = Number(err.code.trim()); | ||
| if (normalizedCode >= 100 && normalizedCode < 600) { | ||
| errHttpCode = err.code.trim(); |
There was a problem hiding this comment.
Avoid double-classifying numeric string error codes
This change promotes digit-only string error.code values to httpCode, but the existing errType = err.code path still runs for the same payloads, so inputs like {"error":{"code":"404","message":"..."}} now format as HTTP 404 404: ... and telemetry records providerErrorType: "404" even though the value is being treated as status. That is a behavior regression for already-supported payload shapes and makes both UI copy and error observation fields misleading.
Useful? React with 👍 / 👎.
| if (!isRawApiErrorPayload(raw)) { | ||
| return false; |
There was a problem hiding this comment.
Parse status-prefixed JSON before OpenRouter 404 gating
isOpenRouterModel404Payload currently bails out unless isRawApiErrorPayload(raw) succeeds, but that predicate rejects strings with a leading status token (for example 404 {"error":...}), even though parseApiErrorInfo can parse that exact shape. In this case the new OpenRouter classifier never runs, so resolveFailoverReasonFromError can still miss model_not_found and stop fallback on an unavailable model.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5fca3951db
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (typeof err.message === "string") { | ||
| errMessage = err.message; | ||
| } | ||
| } | ||
|
|
||
| return { | ||
| httpCode, | ||
| httpCode: errHttpCode ?? httpCode, |
There was a problem hiding this comment.
Keep prefixed HTTP status authoritative over payload code
parseApiErrorInfo now returns errHttpCode before the already-parsed leading status (httpCode: errHttpCode ?? httpCode), so supported <status> <json> errors can be re-labeled with a different code from error.code. In practice, an input like 429 {"error":{"code":"404","message":"..."}} is now formatted/observed as HTTP 404, which can mislead users and downstream classification that relies on parseApiErrorInfo. The leading transport status should remain authoritative, and error.code should only be used when no prefix status was present.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: db230a283b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| apiInfo?.httpCode === "404" && | ||
| typeof apiInfo.message === "string" && | ||
| apiInfo.message.toLowerCase().includes("openrouter.ai") | ||
| ) { |
There was a problem hiding this comment.
Restrict OpenRouter 404 URL heuristic to model-not-found cases
isModelNotFoundErrorMessage() now treats any parsed 404 payload whose message contains openrouter.ai as model_not_found, but this check is not tied to provider context or model-specific wording. As a result, non-model OpenRouter 404s (for example account/route/docs links in error text) can be misclassified and drive fallback down the wrong path instead of surfacing the real failure reason.
Useful? React with 👍 / 👎.
db230a2 to
250a6fe
Compare
|
Merged as 95ee120. This landed as the rewritten current-main regression-only version of the PR after verifying the runtime behavior was already fixed upstream and confirming the added regression coverage locally. |
Summary
model_not_found.mainand kept only regression coverage for the exact payload in the shared matcher, failover classifier, and fallback loop.mainalready handles the payload correctly.Change Type (select all)
Scope (select all touched areas)
Linked Issue/PR
User-visible / Behavior Changes
Security Impact (required)
Yes, explain risk + mitigation: None.Repro + Verification
Environment
model: "openrouter/healer-alpha"Steps
main.Expected
model_not_foundand fallback continues.Actual
mainalready behaves correctly; this PR preserves that behavior with regression coverage.Evidence
mainHuman Verification (required)
node_modules/.bin/vitest run src/agents/live-model-errors.test.ts src/agents/failover-error.test.ts src/agents/model-fallback.test.tsReview Conversations
Compatibility / Migration
Failure Recovery (if this breaks)
src/agents/live-model-errors.test.ts,src/agents/failover-error.test.ts,src/agents/model-fallback.test.tsRisks and Mitigations
mainafter verifying the underlying runtime behavior already landed upstream.