Skip to content

fix(agents): warn on unrecognised model ID instead of silently falling back#38631

Closed
ademczuk wants to merge 3 commits intoopenclaw:mainfrom
ademczuk:fix/model-id-silent-fallback
Closed

fix(agents): warn on unrecognised model ID instead of silently falling back#38631
ademczuk wants to merge 3 commits intoopenclaw:mainfrom
ademczuk:fix/model-id-silent-fallback

Conversation

@ademczuk
Copy link
Copy Markdown
Contributor

@ademczuk ademczuk commented Mar 7, 2026

Summary

  • Problem: When a user specifies a model ID that doesn't match any configured provider/model (e.g. typo like openai/gpt-5.4-pro), the agent silently falls back to the primary default model with no warning.
  • Why it matters: Users unknowingly use the wrong model, with billing implications, capability differences, and hard-to-debug "off" responses.
  • What changed: Added warning logs in two places:
    1. src/agents/model-fallback.ts: Runtime fallback. When runWithModelFallback succeeds on a later candidate after the user's primary model fails with model_not_found, a warning is logged.
    2. src/agents/model-selection.ts: Config-level. When a malformed model string (e.g. "openai/") can't be parsed into a valid ModelRef, a warning is logged before falling back to defaults.
  • What did NOT change: Fallback behaviour itself is unchanged. This is strictly additive logging.

Change Type

  • Bug fix

Scope

  • Agents / subagents

Linked Issue/PR

User-visible / Behavior Changes

Users will now see a warning log when their specified model ID is not found and the system falls back to the default model: Model "openai/gpt-6" not found. Fell back to "openai/gpt-4.1-mini".

Security Impact

  • None of the above

Evidence

  • Unit/integration test added or updated (2 new tests, 97/97 passing)

Human Verification

  • Verified scenarios: Unrecognised model ID triggers warning; valid model ID produces no warning; malformed config string triggers config-level warning.
  • Edge cases checked: attempts[0] precision. Only warns when user's primary model (always candidate 0) has model_not_found, not when a fallback model is unrecognised.
  • What was NOT verified: runWithImageModelFallback (same pattern but doesn't extract reason into attempts, separate concern and out of scope).

Compatibility / Migration

  • Backward compatible: yes
  • Config changes: none
  • Migration needed: none

Failure Recovery

  • Revert: git revert <sha>
  • Bad symptoms: Unexpected warning logs appearing in normal model resolution (would indicate the check is too broad, but the model_not_found reason gating prevents this).

Risks and Mitigations

Risk Mitigation
Duplicate warning for malformed config strings (resolveConfiguredModelRef called twice) Minor noise, not worth adding dedup complexity for an uncommon edge case
Warning not emitted for runWithImageModelFallback Out of scope. Image model fallback is a separate concern

@openclaw-barnacle openclaw-barnacle bot added agents Agent runtime and tooling size: S labels Mar 7, 2026
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Mar 7, 2026

Greptile Summary

This PR adds warning logs in two places when a configured model ID cannot be found: at runtime when runWithModelFallback successfully falls back after the primary model returns model_not_found, and at config-resolution time when a malformed model string (e.g. "openai/") cannot be parsed. Fallback behavior is unchanged — this is purely additive observability. The implementation is clean and the gating condition (attempts[0]?.reason === "model_not_found") correctly avoids false positives for cooldown-skip scenarios.

  • src/agents/model-fallback.ts: Warning is correctly guarded by i > 0 && attempts[0]?.reason === "model_not_found", ensuring it only fires when the user's primary model was actually rejected with that specific error.
  • src/agents/model-selection.ts: Warning is placed after the if (resolved) return guard so it only fires on the unresolvable-string fallback path.
  • src/agents/model-selection.test.ts: warnSpy.mockRestore() is missing from the finally block, causing the suppressing mock implementation to leak into subsequent tests (see inline comment).

Confidence Score: 4/5

  • Safe to merge; the only issue is a test-isolation bug in model-selection.test.ts that could suppress warnings in later tests.
  • The production code changes are minimal, correctly gated, and do not alter any fallback behavior. The single bug found (missing warnSpy.mockRestore() in model-selection.test.ts) is a test-isolation issue, not a production correctness problem, which keeps the score high but not perfect.
  • src/agents/model-selection.test.ts — missing warnSpy.mockRestore() in the finally block will leak the suppression mock to subsequent tests.

Last reviewed commit: aeb6181

@ademczuk
Copy link
Copy Markdown
Contributor Author

ademczuk commented Mar 7, 2026

@steipete — small agents fix (4 files, size:S). When an unrecognised model ID is configured, the current code silently falls back to the default model. This adds a console.warn() so users know the configured model wasn't found.

CI all green, greptile finding (missing mockRestore) already fixed in 963846d.

@ademczuk
Copy link
Copy Markdown
Contributor Author

ademczuk commented Mar 7, 2026

Fixed in da08ac8. Both warn sites now sanitize model strings via stripAnsi() + control char removal before interpolation (CWE-117).

Uses the existing src/terminal/ansi.ts:stripAnsi utility — no new dependencies.

@ademczuk ademczuk force-pushed the fix/model-id-silent-fallback branch from da08ac8 to 1c9e452 Compare March 7, 2026 16:16
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1c9e4524b2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ademczuk ademczuk force-pushed the fix/model-id-silent-fallback branch from 1c9e452 to c9c8b1b Compare March 7, 2026 16:56
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9c8b1bd71

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ademczuk ademczuk force-pushed the fix/model-id-silent-fallback branch from c9c8b1b to 7c8c8fb Compare March 7, 2026 17:25
@ademczuk
Copy link
Copy Markdown
Contributor Author

ademczuk commented Mar 7, 2026

@steipete -- would appreciate your eyes on this when you get a chance.

Context: Adds a warning log when an unrecognised model ID silently falls back to defaults (closes #37813). Size S, 4 files in src/agents/, +90/-0, all 98 tests green.

CI note: The extensions/diffs test failure is a pre-existing issue from your 44881b02 commit (missing headers in index.test.ts mock). Separate fix submitted in #39041 -- trivial 1-line alignment with http.test.ts.

The "Changes at a glance" table in the description has the exact lines to review.

@ademczuk ademczuk force-pushed the fix/model-id-silent-fallback branch from 7c8c8fb to b3fa786 Compare March 7, 2026 20:08
@aisle-research-bot
Copy link
Copy Markdown

aisle-research-bot bot commented Mar 7, 2026

🔒 Aisle Security Analysis

We found 1 potential security issue(s) in this PR:

# Severity Title
1 🔵 Low Incomplete log sanitization allows terminal control / log injection via C1 control characters

1. 🔵 Incomplete log sanitization allows terminal control / log injection via C1 control characters

Property Value
Severity Low
CWE CWE-117
Location src/agents/model-fallback.ts:534-545

Description

runWithModelFallback() adds a warning log that includes provider/model values derived from user-controlled sources (CLI args, config values, session model overrides). The new sanitize() helper strips ANSI SGR + OSC-8 and removes C0 controls (U+0000–U+001F) and DEL (U+007F), but does not remove C1 control characters (U+0080–U+009F).

Because many terminals/log viewers treat C1 characters as control codes (e.g. U+009B CSI, U+009D OSC, U+0085 NEL), an attacker who can influence provider/model strings can still:

  • inject terminal control sequences (screen clearing, cursor movement, OSC sequences)
  • potentially forge/mislead operators by manipulating how the log line renders

Vulnerable code:

const sanitize = (v: string) => {
  let out = stripAnsi(v);
  for (let c = 0; c <= 0x1f; c++) {
    out = out.replaceAll(String.fromCharCode(c), "");
  }
  return out.replaceAll(String.fromCharCode(0x7f), "");
};
log.warn(`Model "${sanitize(attempts[0].provider)}/${sanitize(attempts[0].model)}" ...`);

This leaves C1 controls intact (e.g. \u009b, \u009d), which undermines the intended CWE-117 mitigation.

Recommendation

Use a dedicated sanitizer that removes both C0 and C1 control characters before logging, and consider making separators visible (escaping) instead of deleting.

This repo already has a helper for this purpose: sanitizeTerminalText (src/terminal/safe-text.ts). Prefer reusing it:

import { sanitizeTerminalText } from "../terminal/safe-text.js";

log.warn(
  `Model "${sanitizeTerminalText(attempts[0].provider)}/${sanitizeTerminalText(attempts[0].model)}" not found. ` +
  `Fell back to "${sanitizeTerminalText(candidate.provider)}/${sanitizeTerminalText(candidate.model)}".`,
);

If you keep a local sanitizer, extend it to strip U+007F–U+009F at minimum (and optionally other Unicode separators like U+2028/U+2029 if you want to prevent multi-line rendering across environments).


Analyzed PR: #38631 at commit 4da48cb

Last updated on: 2026-03-07T21:12:23Z

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b3fa786c4c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ademczuk
Copy link
Copy Markdown
Contributor Author

ademczuk commented Mar 7, 2026

CI Status Update (post-rebase)

After rebasing onto latest main, 5 CI failures surfaced. Analysis:

Fixed in this PR (commit 4da48cb):

  • check (TS2741): Two test files (models-config.fills-missing-provider-apikey-from-env-var.test.ts, models-config.providers.normalize-keys.test.ts) were missing baseUrl in ModelProviderConfig objects after upstream made baseUrl required. Added baseUrl: "" to both. All 117 tests pass locally.

Pre-existing on main (not fixed here):

Our PR touches only 4 files in src/agents/ (model-fallback and model-selection) with 90 lines of changes. The unfixed failures all exist on main independent of this PR.

@ademczuk
Copy link
Copy Markdown
Contributor Author

ademczuk commented Mar 7, 2026

Closing this one. The merge conflicts and CI failures have stacked up, and the extensions test regressions on main make it hard to get a clean baseline here. The underlying idea (warning on unrecognised model IDs) still has merit - I'll revisit it as a smaller, more focused fix once the test landscape on main settles down.

@ademczuk ademczuk closed this Mar 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Unrecognised model IDs silently fall back to primary default — bypasses configured fallback chain and tool permissions

1 participant