Skip to content

fix(agents): avoid inviting provider swaps in model alias guidance#85155

Closed
extrasmall0 wants to merge 5 commits into
openclaw:mainfrom
extrasmall0:fix/model-alias-guidance-exact-ref
Closed

fix(agents): avoid inviting provider swaps in model alias guidance#85155
extrasmall0 wants to merge 5 commits into
openclaw:mainfrom
extrasmall0:fix/model-alias-guidance-exact-ref

Conversation

@extrasmall0

@extrasmall0 extrasmall0 commented May 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • stop telling the model to prefer aliases when a full provider/model override is already present
  • clarify that aliases are only shortcuts for unqualified model requests
  • update the prompt test to lock in the exact-ref guidance

Real behavior proof

Behavior or issue addressed: with model aliases configured, the generated system prompt was telling the model to prefer an alias even when an exact provider/model override was already specified.

Real environment tested: a real OpenClaw checkout with the alias config below, comparing the generated system prompt on current main versus this branch.

Config used:

agents:
  defaults:
    models:
      github-copilot/gpt-4.1:
        alias: copilot-4.1
      openai/gpt-4.1:
        alias: openai-4.1

Exact steps or command run after this patch:

  1. generated the system prompt with the config above on current main
  2. generated it again on this branch
  3. compared the model-alias guidance line and alias entries

Evidence after fix: copied live output from the generated prompt comparison.
Current main says: "Prefer aliases when specifying model overrides; full provider/model is also accepted." It then lists copilot-4.1: github-copilot/gpt-4.1 and openai-4.1: openai/gpt-4.1.
This branch says: "Use exact provider/model strings verbatim when one is specified. Aliases are shortcuts for unqualified model requests." It then lists the same alias mappings.

Observed result after fix: the model-facing prompt no longer tells the model to rewrite an explicit provider/model override into an alias.

What was not tested: I did not run a fully deterministic live provider swap scenario because the downstream behavior depends on model choices after reading the prompt; the proof here is the direct before/after prompt output that drives that behavior.

Testing

  • pnpm vitest src/agents/system-prompt.test.ts

Closes #61949

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: XS triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 22, 2026
@clawsweeper

clawsweeper Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 1, 2026, 5:48 AM ET / 09:48 UTC.

Summary
The PR rewrites Model Aliases prompt guidance and changes subagent spawn model alias resolution/tests to support slash-form aliases while trying to preserve exact provider/model overrides.

PR surface: Source +32, Tests +82. Total +114 across 4 files.

Reproducibility: yes. at source level. Current main and v2026.6.11 still render the reported Model Aliases guidance, and current main's resolver preserves slash-form exact refs before alias lookup.

Review metrics: 1 noteworthy metric.

  • Hard-coded exact-ref providers: 6 listed. The safety of the new resolver precedence depends on this list, while official provider plugins can route without models.providers entries.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #61949
Summary: This PR is the active fix candidate for the canonical Model Aliases provider-substitution report; nearby model-alias work overlaps but targets other routing surfaces or only adjacent exact-ref invariants.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🦪 silver shellfish
Patch quality: 🧂 unranked krab
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Preserve exact refs for all official provider-plugin IDs or replace the hard-coded list with a generic known-provider check.
  • [P1] Add redacted terminal output, logs, copied live output, or a linked artifact proving final-head sessions_spawn exact-ref and slash-alias routing.

Proof guidance:

  • [P1] Needs stronger real behavior proof before merge: The PR body includes copied live prompt output, but final-head sessions_spawn resolver behavior still needs redacted terminal output, logs, copied live output, or a linked artifact proving exact-ref and slash-alias routing; redact private data before posting. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

  • [P1] Exact refs for official provider plugins that do not require models.providers entries, such as github-copilot or openrouter, can still fall through to slash-form alias lookup and be rewritten to a different provider/model.
  • [P1] A rewritten model ref can select a different provider runtime or auth profile for spawned sessions, which is the same class of provider-swap failure this PR is trying to prevent.
  • [P1] The PR body proves prompt wording before/after, but the final head also changes sessions_spawn model routing and still needs redacted terminal/live output or logs proving that runtime path.

Maintainer options:

  1. Complete Exact-Ref Detection Before Merge (recommended)
    Update the resolver so every official or otherwise known provider ID that can route without a models.providers row stays authoritative before slash-form alias lookup, then add focused coverage for an omitted official provider.
  2. Require Runtime Proof On Final Head
    Ask the contributor for redacted terminal output, logs, copied live output, or a linked artifact showing sessions_spawn preserving an exact provider ref and resolving a slash-form alias after the final resolver change.
  3. Accept Narrow Provider Coverage
    Maintainers may intentionally accept the hard-coded provider set, but that owns the upgrade risk for official provider-plugin refs outside the list.

Next step before merge

  • [P1] The blocker combines a contributor-owned final-head proof gap with a maintainer-visible provider resolver fix, so this is not a safe repair-only lane right now.

Security
Cleared: The diff changes TypeScript prompt/model-selection logic and tests only; no workflow, dependency, lockfile, secret, package, publishing, or new code-execution surface is touched.

Review findings

  • [P1] Preserve all provider plugin refs before slash aliases — src/agents/model-selection.ts:93-100
Review details

Best possible solution:

Land this only after exact provider/model refs are preserved for every known provider route through a generic provider-known check, plus final-head sessions_spawn proof for exact refs and slash-form aliases.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level. Current main and v2026.6.11 still render the reported Model Aliases guidance, and current main's resolver preserves slash-form exact refs before alias lookup.

Is this the best way to solve the issue?

No, not yet. The prompt wording is the right layer, but the resolver repair should preserve exact refs for all known provider routes and the final runtime path needs real behavior proof.

Full review comments:

  • [P1] Preserve all provider plugin refs before slash aliases — src/agents/model-selection.ts:93-100
    This exact-ref guard only treats six hard-coded providers plus cfg.models.providers keys as authoritative. Official provider plugins such as github-copilot and openrouter can route without user models.providers rows, so an exact override like openrouter/... can still be captured by a matching slash-form alias and rewritten to another provider, regressing current passthrough behavior and the new verbatim-ref contract.
    Confidence: 0.86

Overall correctness: patch is incorrect
Overall confidence: 0.88

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against cf2efbe5d515.

Label changes

Label justifications:

  • P2: This is a normal-priority agent/provider routing bug fix with real but limited blast radius for users with model aliases and multiple providers.
  • merge-risk: 🚨 compatibility: The PR changes sessions_spawn slash-form model override precedence and can alter existing exact provider/model routing on upgrade.
  • merge-risk: 🚨 auth-provider: If an exact ref is rewritten through an alias, spawned sessions can use a different provider runtime and auth profile than requested.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🦪 silver shellfish and patch quality is 🧂 unranked krab.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR body includes copied live prompt output, but final-head sessions_spawn resolver behavior still needs redacted terminal output, logs, copied live output, or a linked artifact proving exact-ref and slash-alias routing; redact private data before posting. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +32, Tests +82. Total +114 across 4 files.

View PR surface stats
Area Files Added Removed Net
Source 2 40 8 +32
Tests 2 83 1 +82
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 4 123 9 +114

What I checked:

  • Repository policy read and applied: Root AGENTS.md treats provider routing, model choice, auth profiles, and compatibility paths as merge-sensitive, and the scoped agents guide was also read for this touched path. (AGENTS.md:20, cf2efbe5d515)
  • Current main still has reported prompt wording: Current main still tells agents to prefer aliases while also accepting full provider/model refs, so the central issue is not already implemented on main. (src/agents/system-prompt.ts:1150, cf2efbe5d515)
  • Latest release still has old prompt wording: The latest release tag v2026.6.11 still contains the same Model Aliases sentence, so the fix has not shipped. (src/agents/system-prompt.ts:1158, e085fa1a3ffd)
  • Current main preserves slash-form exact refs before alias lookup: The existing resolver returns any slash-containing value before alias lookup, so this PR changes an existing exact-ref passthrough invariant for sessions_spawn model selection. (src/agents/model-selection.ts:352, cf2efbe5d515)
  • PR head hard-codes exact-ref provider IDs: The PR treats only six built-in provider IDs plus configured provider rows as authoritative before slash-form alias lookup. (src/agents/model-selection.ts:93, ec1a94017f3c)
  • Official provider plugins exist outside that hard-coded set: Bundled provider plugin manifests declare provider IDs such as github-copilot, openrouter, and amazon-bedrock, which are not in the PR's six-item exact-ref set. (extensions/github-copilot/openclaw.plugin.json:7, cf2efbe5d515)

Likely related people:

  • HowdyDooToYou: Commit 9cdfe0e added resolveModelThroughAliases and nearby sessions_spawn alias tests that this PR changes. (role: introduced sessions_spawn alias resolution; confidence: high; commits: 9cdfe0ebc27a; files: src/agents/model-selection.ts, src/agents/model-selection.test.ts)
  • vincentkoc: The sessions_spawn alias-resolution commit was committed by vincentkoc, making this a useful routing signal for the provider/model selection surface. (role: committer and adjacent area contributor; confidence: high; commits: 9cdfe0ebc27a; files: src/agents/model-selection.ts, src/agents/model-selection.test.ts)
  • steipete: Earlier Model Aliases prompt and alias-line builder history is tied to Peter Steinberger's commits in the prompt path changed by this PR. (role: feature-history contributor; confidence: high; commits: 75c66acfd828, cb2f978ed508, 5c8e1b6eeff8; files: src/agents/system-prompt.ts, src/agents/system-prompt.test.ts, src/agents/model-alias-lines.ts)
  • stevenepalmer: Merged PR history shows Steven fixed exact configured provider refs before aliases in the shared model-selection resolver, the same invariant this PR needs in the subagent path. (role: adjacent exact-ref precedence contributor; confidence: medium; commits: 86e33d6985c9; files: src/agents/model-selection-shared.ts, src/agents/model-selection.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. labels May 22, 2026
@clawsweeper

clawsweeper Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper PR egg

✨ Hatched: 🥚 common Frosted Signal Puff

Hatch command

Comment @clawsweeper hatch when this PR is hatchable.

Hatchability rules:

  • Merged PRs are hatchable.
  • Open PRs are hatchable when they are status: 👀 ready for maintainer look, status: 🚀 automerge armed, or labeled clawsweeper:automerge.
  • Closed unmerged PRs are hatchable only when one of those hatchable labels is still present in the durable record.

Rarity: 🥚 common.
Trait: watches the merge queue.
Image traits: location proof lagoon; accessory proof snapshot camera; palette sunrise gold and clean white; mood mischievous; pose standing beside its cracked shell; shell frosted glass shell; lighting warm desk-lamp glow; background tiny artifact crates.
Share on X: post this hatch
Copy: My PR egg hatched a 🥚 common Frosted Signal Puff in ClawSweeper.

What is this egg doing here?
  • Eggs appear after the PR passes real-behavior proof. It is here for vibes, not verdicts: it does not change labels, ratings, merge decisions, or automation.
  • The shell reacts to review momentum: open follow-up work warms it up, re-review makes it wobble, and a clean final review lets it hatch.
  • Hatchability usually comes from sufficient real-behavior proof, no blocking P0/P1/P2 findings, no security attention needed, and clean correctness. A merged PR is already final, so merge makes the egg hatchable independently.
  • The hatch is seeded from this repository and PR number, so the same PR keeps the same creature; the reviewed head SHA can only change safe visual details.
  • Rarity is just collectible sparkle: 🥚 common, 🌱 uncommon, 💎 rare, ✨ glimmer, and 🌈 legendary.

@extrasmall0

extrasmall0 commented May 22, 2026

Copy link
Copy Markdown
Contributor Author

Updated the PR body again so the proof fields now use the gate's exact accepted labels, and the evidence line explicitly points at the terminal output shown below.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels May 22, 2026
@martingarramon

Copy link
Copy Markdown
Contributor

This tightens the model-alias guidance in the right layer. The old phrasing, "full provider/model is also accepted," was ambiguous about priority and left room for the model to treat a provider swap as an acceptable first move. The replacement makes the invariant explicit: qualified provider/model strings pass through verbatim, and aliases are only shortcuts for unqualified requests. The guard conditions (modelAliasLines present, non-minimal mode) are unchanged, so there is no expansion of when this section appears.

@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 26, 2026
@clawsweeper

clawsweeper Bot commented May 26, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Command router queued. I will update this comment with the next step.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels May 26, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label May 26, 2026
@clawsweeper clawsweeper Bot added rating: 🦀 challenger crab Exceptional PR readiness: strong proof, clean patch, and convincing validation. and removed rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. labels May 26, 2026
@openclaw-barnacle openclaw-barnacle Bot added triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. labels May 27, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🦀 challenger crab Exceptional PR readiness: strong proof, clean patch, and convincing validation. labels May 27, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label May 27, 2026
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. and removed rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. proof: sufficient ClawSweeper judged the real behavior proof convincing. labels Jun 5, 2026
@extrasmall0

Copy link
Copy Markdown
Contributor Author

Updated this to keep slash-form aliases valid while preserving exact provider/model passthrough. Added the focused slash-form alias prompt case.

Validated:

  • node scripts/run-vitest.mjs src/agents/system-prompt.test.ts
  • git diff --check

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@extrasmall0

Copy link
Copy Markdown
Contributor Author

Updated this to resolve exact configured aliases before treating a slash-containing value as a provider/model ref. Unknown provider/model overrides still pass through unchanged, but aliases like xiaomi/mimo-v2-pro-mit now resolve to the configured target first.

Validated:

  • node scripts/run-vitest.mjs src/agents/model-selection.test.ts
  • node scripts/run-vitest.mjs src/agents/system-prompt.test.ts
  • node scripts/run-vitest.mjs src/agents/openclaw-tools.subagents.sessions-spawn.model.test.ts
  • git diff --check

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@extrasmall0

extrasmall0 commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Pushed a small follow-up for the resolver concern.

The subagent spawn path now keeps exact built-in provider/model refs like openai/gpt-5.4 ahead of slash-form aliases, and still keeps exact configured provider/model refs ahead of aliases. Non-provider slash aliases such as xiaomi/mimo-v2-pro-mit continue to resolve through the alias table.

Validation:
pnpm test src/agents/model-selection.test.ts
pnpm format -- src/agents/model-selection.ts src/agents/model-selection.test.ts
git diff --check

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper

clawsweeper Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@extrasmall0

Copy link
Copy Markdown
Contributor Author

please review thanks

@clawsweeper

clawsweeper Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix(agents): avoid inviting provider swaps in model alias guidance This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

@openclaw-barnacle

Copy link
Copy Markdown

Closing due to inactivity.
If you believe this PR should be revived, post in #clawtributors on Discord to talk to a maintainer.
That channel is the escape hatch for high-quality PRs that get auto-closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: S stale Marked as stale due to inactivity status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Model Aliases in system prompt cause LLM to substitute providers unexpectedly

3 participants