Skip to content

fix(cli): canonicalize infer model run refs#73717

Open
ai-hpc wants to merge 1 commit intoopenclaw:mainfrom
ai-hpc:fix/infer-model-case-canonicalization
Open

fix(cli): canonicalize infer model run refs#73717
ai-hpc wants to merge 1 commit intoopenclaw:mainfrom
ai-hpc:fix/infer-model-case-canonicalization

Conversation

@ai-hpc
Copy link
Copy Markdown
Contributor

@ai-hpc ai-hpc commented Apr 28, 2026

Summary

  • Problem: infer model run --model forwarded case-mismatched catalog model ids such as Anthropic/CLAUDE-OPUS-4-7 to the model runtime, which could surface as a misleading empty-output provider error.
  • Why it matters: users can paste natural mixed-case model refs and get a provider-looking failure instead of the canonical known model behavior.
  • What changed: explicit model-run overrides now resolve through the model catalog when there is a unique case-insensitive provider/model match, then dispatch the canonical provider/model ref for both local and gateway transports.
  • What did NOT change (scope boundary): custom mixed-case refs are preserved when no unique catalog match exists; this does not change model catalog listing, provider auth, or non-infer model run commands.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

Root Cause (if applicable)

  • Root cause: infer model run parsed and forwarded explicit model overrides without canonicalizing case-only matches against the known model catalog.
  • Missing detection / guardrail: no regression test covered case-mismatched explicit model refs before local or gateway dispatch.
  • Contributing context (if known): provider ids are normalized case-insensitively, while model ids were forwarded with user-supplied casing.

Regression Test Plan (if applicable)

  • Coverage level that should have caught this:
    • Unit test
    • Seam / integration test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file: src/cli/capability-cli.test.ts
  • Scenario the test should lock in: Anthropic/CLAUDE-OPUS-4-7 is canonicalized to anthropic/claude-opus-4-7 before local and gateway model-run dispatch.
  • Why this is the smallest reliable guardrail: the bug is in the CLI override-to-dispatch seam, so mocked dispatch tests catch the raw value before any provider call.
  • Existing test that already covers this (if any): none.
  • If no new test is added, why not: N/A.

User-visible / Behavior Changes

openclaw infer model run --model <provider/model> now accepts case-only mismatches for catalog models when the match is unique, dispatching the canonical model id instead of the raw casing.

Diagram (if applicable)

Before:
--model Anthropic/CLAUDE-OPUS-4-7 -> raw runtime dispatch -> misleading provider/no-output error

After:
--model Anthropic/CLAUDE-OPUS-4-7 -> unique catalog match -> anthropic/claude-opus-4-7 -> normal dispatch

Security Impact (required)

  • New permissions/capabilities? (Yes/No): No
  • Secrets/tokens handling changed? (Yes/No): No
  • New/changed network calls? (Yes/No): No
  • Command/tool execution surface changed? (Yes/No): No
  • Data access scope changed? (Yes/No): No
  • If any Yes, explain risk + mitigation: N/A

Repro + Verification

Environment

  • OS: Ubuntu 24.04.4 LTS
  • Runtime/container: Node 22, pnpm dev checkout
  • Model/provider: Anthropic catalog model refs
  • Integration/channel (if any): CLI infer model run
  • Relevant config (redacted): default source-checkout runtime config; no secrets included

Steps

  1. Run openclaw infer model run --model Anthropic/CLAUDE-OPUS-4-7 --prompt "hello" --json.
  2. Observe the model-run dispatch receives the canonical ref.
  3. Run the same path with a custom mixed-case ref that has no catalog match.

Expected

  • Known catalog model refs with case-only mismatches dispatch as their canonical provider/model id.
  • Custom mixed-case refs without a unique catalog match remain unchanged.

Actual

  • Before this PR, the raw mixed-case catalog model id was forwarded.
  • After this PR, the CLI canonicalizes only unique catalog matches before dispatch.

Evidence

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording
  • Perf numbers (if relevant)
pnpm test src/cli/capability-cli.test.ts -- --reporter=verbose
Test Files  1 passed (1)
Tests  47 passed (47)

pnpm check:changed
passed

git diff --check
passed

Human Verification (required)

  • Verified scenarios: local dispatch canonicalization, gateway dispatch canonicalization, custom mixed-case ref preservation.

  • Edge cases checked: no catalog match leaves the explicit model override unchanged.

  • What you did not verify: live Anthropic provider call.

  • pnpm test src/cli/capability-cli.test.ts -- --reporter=verbose passed: 51 tests.

  • pnpm build passed after refreshing the local install with pnpm install.

  • Live CLI smoke with existing local OpenRouter auth profile passed:

    • Command shape: pnpm --silent openclaw infer model run --model OpenRouter/OPENROUTER/AUTO --prompt 'Reply with OK only.' --json
    • Result: exit 0, parseable JSON, ok=true, provider=openrouter, model=openrouter/auto, output text OK.
  • Additional live smoke with OpenRouter/ANTHROPIC/CLAUDE-3-HAIKU exited 1 because the provider returned no text, but the error reported canonicalized provider "openrouter" model "anthropic/claude-3-haiku", not the uppercase input.

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

  • Backward compatible? (Yes/No): Yes
  • Config/env changes? (Yes/No): No
  • Migration needed? (Yes/No): No
  • If yes, exact upgrade steps: N/A

Risks and Mitigations

  • Risk: A custom provider intentionally using mixed-case ids could be changed unexpectedly.
    • Mitigation: canonicalization only happens when the loaded model catalog has exactly one case-insensitive provider/model match; otherwise the raw ref is preserved.

Real behavior proof

Behavior or issue addressed: openclaw infer model run --model <ref> previously forwarded the user-supplied model ref to the simple-completion transport without case canonicalization. A user typing DeepSeek/DeepSeek-V4-Flash (matching the docs / catalog display name) hit a "model not found" / wrong-routing path because catalog ids are stored lowercased. After this patch, mixed-case refs are looked up against the model catalog and rewritten to the canonical provider/id before transport setup, so case-insensitive references just work.

Real environment tested: macOS Darwin 25.2.0 (arm64), Node 24.15.0 (Homebrew node@24), local OpenClaw source checkout at /Users/a1111/openclaw/ running directly via node --import tsx against the patched src/cli/capability-cli.ts and the production findModelInCatalog / loadModelCatalog helpers. Real configured providers (DeepSeek native + OpenAI catalog entries) loaded from ~/.openclaw/openclaw.json — no mocked catalog.

Exact steps or command run after this patch:

  1. Checked out the patched branch fix/infer-model-case-canonicalization (HEAD 2d491c11f6, rebased onto upstream b8f9137d31).
  2. Ran a small node script that loads the user's real config snapshot, loads the model catalog, and exercises findModelInCatalog with several mixed-case provider/id inputs — the same lookup path that the patched canonicalizeModelRunRef calls.
  3. Captured the canonical output for each input.

Evidence after fix:

Live terminal capture from the patched source running on my Mac, exercising the canonical lookup with mixed-case inputs (the exact code path that runModelRun now uses before transport setup):

$ node --import tsx --input-type=module -e "
import { loadModelCatalog } from './src/agents/model-catalog.ts';
import { findModelInCatalog } from './src/agents/model-catalog-lookup.ts';
import { readConfigFileSnapshot } from './src/config/io.ts';
const snap = await readConfigFileSnapshot();
const catalog = await loadModelCatalog({ config: snap.config });
for (const r of [
  { provider: 'DeepSeek', model: 'DeepSeek-V4-Flash' },
  { provider: 'DEEPSEEK', model: 'DEEPSEEK-V4-PRO' },
  { provider: 'deepseek', model: 'deepseek-v4-flash' },
  { provider: 'OpenAI', model: 'GPT-4o-mini' },
]) {
  const e = findModelInCatalog(catalog, r.provider, r.model);
  console.log(JSON.stringify(r) + ' => ' + (e ? e.provider + '/' + e.id : 'NOT FOUND'));
}
"
{"provider":"DeepSeek","model":"DeepSeek-V4-Flash"} => deepseek/deepseek-v4-flash
{"provider":"DEEPSEEK","model":"DEEPSEEK-V4-PRO"} => deepseek/deepseek-v4-pro
{"provider":"deepseek","model":"deepseek-v4-flash"} => deepseek/deepseek-v4-flash
{"provider":"OpenAI","model":"GPT-4o-mini"} => openai/gpt-4o-mini

The patched canonicalizeModelRunRef in src/cli/capability-cli.ts runs only when the input has uppercase characters and a parsable provider/id shape, then reuses the catalog lookup above and substitutes the canonical pair before invoking transport setup. Already-canonical lowercase inputs short-circuit and are returned untouched.

Observed result after fix:

Mixed-case --model inputs (e.g., DeepSeek/DeepSeek-V4-Flash, OpenAI/GPT-4o-mini) resolve to the lowercased catalog id (deepseek/deepseek-v4-flash, openai/gpt-4o-mini) before the local simple-completion transport or the gateway lane is invoked. Already-canonical lowercase refs are returned unchanged, so this patch is a no-op for the dominant call shape and only changes behavior for previously-failing mixed-case input.

What was not tested:

  • Live one-shot inference round-trip (i.e., actually completing a model turn with a mixed-case ref) — exercise here was scoped to the canonicalization step that immediately precedes transport setup, since that is where the case bug lives.
  • Refs containing non-ASCII characters or unicode case folding edge cases — the helper compares model !== model.toLowerCase() against ASCII catalog ids and was not exercised against non-ASCII names.
  • Gateway transport lane (only the local simple-completion branch was followed in code; the patch wires the canonicalized ref through both branches but the gateway path was not separately exercised live).

@openclaw-barnacle openclaw-barnacle Bot added cli CLI command changes size: S labels Apr 28, 2026
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Apr 28, 2026

Greptile Summary

This PR fixes a bug in infer model run where mixed-case --model refs like Anthropic/CLAUDE-OPUS-4-7 were forwarded raw to the runtime, producing misleading provider errors. The fix adds resolveCanonicalModelRunOverride, which performs a case-insensitive catalog lookup and replaces the ref with the canonical form before both local and gateway dispatch.

Confidence Score: 5/5

This PR is safe to merge; the canonicalization is narrowly scoped, well-guarded, and fully covered by the new tests.

The change is minimal and targeted: it only fires when both a provider and model are present in the ref and at least one has non-lowercase characters, avoiding unnecessary catalog loads for the common case. All four dispatch variants (local canonical, local no-match, gateway canonical, gateway no-match) are now tested. No pre-existing patterns are altered, and custom mixed-case refs without a catalog match are explicitly preserved.

No files require special attention.

Reviews (2): Last reviewed commit: "fix(cli): canonicalize infer model refs" | Re-trigger Greptile

Comment thread src/cli/capability-cli.test.ts Outdated
@clawsweeper
Copy link
Copy Markdown
Contributor

clawsweeper Bot commented Apr 28, 2026

Codex review: needs changes before merge.

Summary
The PR adds catalog-backed canonicalization for mixed-case infer model run --model refs before local and gateway dispatch, plus CLI tests for canonical matches and no-match preservation.

Reproducibility: yes. by source inspection and linked report. The related issue gives concrete infer model run --model anthropic/CLAUDE-OPUS-4-7 steps, and current main still forwards the raw model ref into local and gateway dispatch paths.

Real behavior proof
Sufficient (terminal): The PR body contains after-fix terminal output from a patched checkout, including live CLI output showing canonicalized provider/model behavior.

Next step before merge
A repair worker can make a narrow update on this PR or a replacement branch: preserve auth-profile suffixes, add the missing regression coverage, and add the changelog entry.

Security
Cleared: The diff only changes CLI model-ref normalization and colocated tests, with no dependency, CI, permission, secret-handling, or supply-chain surface change.

Review findings

  • [P2] Preserve auth-profile suffixes before catalog lookup — src/cli/capability-cli.ts:570-576
  • [P3] Add the required changelog entry — src/cli/capability-cli.ts:666
Review details

Best possible solution:

Land this PR or an equivalent catalog-backed fix after preserving trailing auth-profile suffixes, adding focused regression coverage, and recording the user-facing CLI fix in the changelog.

Do we have a high-confidence way to reproduce the issue?

Yes, by source inspection and linked report. The related issue gives concrete infer model run --model anthropic/CLAUDE-OPUS-4-7 steps, and current main still forwards the raw model ref into local and gateway dispatch paths.

Is this the best way to solve the issue?

No, not as-is. Catalog-backed canonicalization before dispatch is the right narrow solution, but it should split and reattach supported @profile suffixes and add the required changelog entry before merge.

Full review comments:

  • [P2] Preserve auth-profile suffixes before catalog lookup — src/cli/capability-cli.ts:570-576
    canonicalizeModelRunRef looks up the catalog entry before removing a trailing auth-profile suffix. A supported ref such as Anthropic/CLAUDE-OPUS-4-7@work searches for CLAUDE-OPUS-4-7@work, misses the catalog entry, and still dispatches the uppercase base model; split the existing auth-profile suffix first, canonicalize the base ref, then reattach it.
    Confidence: 0.92
  • [P3] Add the required changelog entry — src/cli/capability-cli.ts:666
    This is a user-facing CLI bug fix, but the PR changes only source and tests. OpenClaw policy requires a CHANGELOG.md entry under the active Unreleased Fixes section before landing.
    Confidence: 0.86

Overall correctness: patch is incorrect
Overall confidence: 0.9

Acceptance criteria:

  • pnpm test src/cli/capability-cli.test.ts -- --reporter=verbose
  • pnpm exec oxfmt --check --threads=1 src/cli/capability-cli.ts src/cli/capability-cli.test.ts CHANGELOG.md
  • git diff --check
  • pnpm check:changed

What I checked:

  • current_main_raw_dispatch: Current main passes params.model directly to the local simple-completion path and resolves gateway provider/model from the raw override, matching the linked case-mismatch bug. (src/cli/capability-cli.ts:647, 0b88d6286c4b)
  • pr_head_canonicalization: The PR head adds canonicalizeModelRunRef, loads the model catalog for mixed-case provider/model refs, and passes the canonicalized value into both local and gateway dispatch. (src/cli/capability-cli.ts:569, 38aca3853a90)
  • auth_profile_suffix_contract: Current model selection supports trailing auth-profile suffixes by splitting model@profile before model resolution and retaining profileId for credential lookup; tests cover anthropic/claude-opus-4-6@work. (src/agents/simple-completion-runtime.ts:77, 0b88d6286c4b)
  • pr_head_auth_suffix_miss: The PR helper resolves the provider/model and calls findModelInCatalog before stripping any trailing @profile, so Anthropic/CLAUDE-OPUS-4-7@work searches for a catalog model id containing @work and falls back to the raw uppercase ref. (src/cli/capability-cli.ts:570, 38aca3853a90)
  • pr_tests_cover_main_cases: The PR adds local and gateway tests for case-mismatched catalog refs and no-match preservation, including the earlier Greptile gateway no-match gap. (src/cli/capability-cli.test.ts:818, 38aca3853a90)
  • changelog_missing: The live PR files API reports only src/cli/capability-cli.ts and src/cli/capability-cli.test.ts changed; the PR head changelog has no entry for this user-facing CLI fix under the active Unreleased Fixes section. (CHANGELOG.md:111, 38aca3853a90)

Likely related people:

  • steipete: Recent commits repeatedly touched infer model run, local probe behavior, simple-completion runtime, catalog lookup, and trailing auth-profile model-ref parsing around the exact surface this PR changes. (role: recent maintainer and likely reviewer; confidence: high; commits: 42dddbbe785a, 12ee7f696f05, 8c8dfa768a0d; files: src/cli/capability-cli.ts, src/agents/simple-completion-runtime.ts, src/agents/model-catalog-lookup.ts)
  • vincentkoc: Recent and co-authored history covers the capability CLI gateway/client seams and auth-profile parsing behavior that this PR must preserve. (role: adjacent maintainer; confidence: medium; commits: 0f7d9c957093, 4b259ab81b2f, 420c96e7aab2; files: src/cli/capability-cli.ts, src/agents/model-ref-profile.ts, src/agents/simple-completion-runtime.ts)

Remaining risk / open question:

  • No tests were executed in this read-only review; contributor-reported pnpm test, pnpm check:changed, and diff-check output still need normal merge-gate confirmation on the exact head.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 0b88d6286c4b.

@ai-hpc ai-hpc force-pushed the fix/infer-model-case-canonicalization branch from 9da6425 to 550dec8 Compare April 28, 2026 18:16
@ai-hpc
Copy link
Copy Markdown
Contributor Author

ai-hpc commented Apr 28, 2026

Human verification update for bbdcee6d5ca27156971ad1fe46eb091647caa2c4:

Verified scenarios:

  • pnpm test src/cli/capability-cli.test.ts -- --reporter=verbose passed before the final rebase: 1 file, 52 tests.
  • pnpm check:changed passed before the final clean rebase. It selected lanes=all because the rebased fork branch history made the changed-lane detector conservative; completed checks included conflict markers, changelog attributions, guarded/plugin-sdk wildcard re-export guards, runtime sidecar loader guard, tsgo:all, oxlint shards, and runtime import cycles.
  • Final clean rebase onto current main completed without conflicts.
  • Re-ran pnpm test src/cli/capability-cli.test.ts -- --reporter=verbose after the final rebase: 1 file, 52 tests passed.
  • git diff --check upstream/main...HEAD passed.
  • GitHub currently reports mergeable: true, mergeable_state: clean.

Edge cases checked:

  • Case-mismatched model refs canonicalize before local dispatch.
  • Case-mismatched model refs canonicalize before gateway dispatch.
  • Custom mixed-case refs are preserved when the catalog has no unique match.
  • Gateway model overrides still request the backend/admin lane.
  • Existing image-file model probe behavior remains covered.

What I did not verify:

  • I did not rerun live Anthropic/OpenRouter provider calls after this rebase; the PR behavior is covered by local CLI seam tests.
  • I did not test on macOS or Windows locally.

Re-review progress:

@ai-hpc
Copy link
Copy Markdown
Contributor Author

ai-hpc commented Apr 29, 2026

Hi @steipete, could you take a look at this PR when you have a moment? Thanks!

@ai-hpc ai-hpc force-pushed the fix/infer-model-case-canonicalization branch 4 times, most recently from c947bc1 to 2bdc4fa Compare April 29, 2026 14:10
@hxy91819 hxy91819 self-assigned this May 4, 2026
@ai-hpc ai-hpc force-pushed the fix/infer-model-case-canonicalization branch from 2bdc4fa to 2d491c1 Compare May 5, 2026 07:41
@openclaw-barnacle openclaw-barnacle Bot added triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 5, 2026
@ai-hpc ai-hpc force-pushed the fix/infer-model-case-canonicalization branch from 2d491c1 to a0456d4 Compare May 5, 2026 07:54
@ai-hpc ai-hpc force-pushed the fix/infer-model-case-canonicalization branch from a0456d4 to 38aca38 Compare May 6, 2026 08:48
@openclaw-barnacle openclaw-barnacle Bot added the proof: supplied External PR includes structured after-fix real behavior proof. label May 6, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli CLI command changes proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. size: S

Projects

None yet

2 participants