fix: user-pinned model falls back to global chain on quota exhaustion#88329
fix: user-pinned model falls back to global chain on quota exhaustion#88329Knowcheng wants to merge 4 commits into
Conversation
When a user explicitly selects a model via `/model <name>`, the session stores `modelOverride` with `modelOverrideSource: "user"`. Previously, `resolveEffectiveModelFallbacks` returned `[]` for user-sourced overrides, disabling the entire fallback chain for that session. This causes a complete outage when the pinned model hits a long-term limit: - Weekly/monthly quota exhaustion (error 1310 from zhipuai/GLM) - Subscription limits (openai-codex) In these cases, `hasFallbackCandidates` becomes `false`, the runner sets `fallbackConfigured: false`, and every message surfaces: "⚠️ All models are temporarily rate-limited. Please try again in a few minutes." The user's intent when pinning a model is to *prefer* it, not to permanently disable recovery when it becomes unavailable. The pinned model is always the first candidate and continues to be tried first — this change only affects what happens when it fails. Fix: when `canUseConfiguredFallbacks` is false (user-sourced pin), return the global fallback chain (`agentFallbacksOverride ?? defaultFallbacks`) instead of `[]`. This is consistent with how `modelOverrideSource === "auto"` already behaves — it falls through to the same return at lines 551-552.
openclaw-barnacle
Bot
added
agents
|
Codex review: needs real behavior proof before merge. Reviewed July 5, 2026, 5:07 PM ET / 21:07 UTC. Summary PR surface: Source +80, Tests +128, Docs 0. Total +208 across 10 files. Reproducibility: yes. at source level: current main returns an empty fallback list for user-sourced model overrides, and the PR discussion includes before-fix logs showing one-candidate exhaustion. I did not run a live quota-exhausted provider setup. Review metrics: 1 noteworthy metric.
Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Proof guidance:
Risk before merge
Maintainer options:
Next step before merge
Maintainer decision needed
Security Review findings
Review detailsBest possible solution: Land a rebased quota-only fallback fix only after maintainers accept the pinned-model semantics change, quota candidates are available before cooldown/suspend decisions, current OpenClaw and upstream Codex usage-limit strings are covered, and redacted after-fix proof shows fallback progression. Do we have a high-confidence way to reproduce the issue? Yes at source level: current main returns an empty fallback list for user-sourced model overrides, and the PR discussion includes before-fix logs showing one-candidate exhaustion. I did not run a live quota-exhausted provider setup. Is this the best way to solve the issue? No, not yet. The quota-only direction is plausible, but the expansion point is too late for cooldown-precheck exhaustion, the Codex matcher is too narrow, and the persisted-pin semantics change needs maintainer acceptance. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 51771c3a1485. Label changesLabel justifications:
Evidence reviewedPR surface: Source +80, Tests +128, Docs 0. Total +208 across 10 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
Review history (1 earlier review cycle)
|
clawsweeper
Bot
added
rating: 🧂 unranked krab
|
I think this needs a bit more design/coverage before landing. This change flips |
… preference semantics - In the !canUseConfiguredFallbacks branch, check isSubagentSessionKey first so user-pinned subagent sessions use the subagent fallback chain (same as auto-selected subagents) rather than the generic agent/default chain. - Update agent-scope tests: user-pinned and legacy-source sessions now return the configured fallback list instead of []. - Update models.md and model-failover.md to describe user pins as a preference (pinned model tried first, falls back on long-term quota exhaustion) rather than a strict lock. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
…on B) Revert the earlier broad fix that allowed user-pinned sessions to use the configured fallback chain for any failure type. Instead, keep resolveEffectiveModelFallbacks returning [] for user-pinned sessions (strict pin for transient errors), and introduce a targeted recovery path in runWithModelFallback that activates only on permanent periodic quota signals (weekly/monthly limits detected via isPeriodicUsageLimitErrorMessage). Changes: - resolveEffectiveModelFallbacks: reverts !canUseConfiguredFallbacks branch back to return [] - resolveQuotaExhaustionFallbacks: new exported function providing the fallback chain for quota-exhaustion recovery (subagent-aware, mirrors the configured chain logic) - runWithModelFallback: adds quotaExhaustionFallbacksOverride param; on primary failure, if isPrimary && !hasFallbackCandidates and the error matches the periodic quota pattern, candidates is expanded in-place so the loop continues with the configured fallback chain - agent-command.ts: pre-computes quotaExhaustionFallbacksOverride when effectiveFallbacksOverride is [] (user-pinned sessions only) and passes it to runWithModelFallback - agent-scope.test.ts: reverts 4 assertions back to toStrictEqual([]), adds new test block for resolveQuotaExhaustionFallbacks Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
|
@clawsweeper review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
…nner
Addresses two P1/P2 findings from the Option B review:
1. Extend isPeriodicUsageLimitErrorMessage to cover missing signals:
- Z.ai Chinese weekly/monthly quota message: 每[周月].*使用上限
(real error from logs: 您已达到每周/每月使用上限)
- Codex usage limit: You have hit your (ChatGPT )?usage limit
(produced by openai-codex-responses.ts for error 1310/usage_limit_reached)
Adds matcher tests covering all three patterns.
2. Wire quotaExhaustionFallbacksOverride through resolveModelFallbackOptions
so channel/auto-reply runners (agent-runner-execution.ts, followup-runner.ts)
also expand the candidate chain on permanent quota for user-pinned sessions.
The agent-runner-run-params.ts function already owns fallbacksOverride
resolution; quotaExhaustionFallbacksOverride follows the same pattern.
Adds test cases covering user-pin expansion and non-expansion for auto.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
|
@clawsweeper review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
Live repro: 7 consecutive
|
|
@Knowcheng thanks for the PR. ClawSweeper is still waiting on real behavior proof before this can move forward. Useful proof can be a screenshot, short video, terminal output, copied live output, linked artifact, or redacted logs that show the changed behavior after the fix. Please redact private tokens, phone numbers, private endpoints, customer data, and anything else sensitive. Once proof is added to the PR body or a comment, ClawSweeper or a maintainer can re-check it. |
clawsweeper
Bot
added
the
merge-risk: 🚨 availability
openclaw-barnacle
Bot
added
the
triage: needs-pr-context
|
This pull request has been automatically marked as stale due to inactivity. |
|
@Knowcheng thanks for the PR. ClawSweeper is still waiting on real behavior proof before this can move forward. Useful proof can be a screenshot, short video, terminal output, copied live output, linked artifact, or redacted logs that show the changed behavior after the fix. Please redact private tokens, phone numbers, private endpoints, customer data, and anything else sensitive. Once proof is added to the PR body or a comment, ClawSweeper or a maintainer can re-check it. |
4 participants
Problem
When a user selects a model via `/model `, `modelOverride` is stored in `sessions.json` with `modelOverrideSource: "user"`. The `resolveEffectiveModelFallbacks` function returns `[]` for user-sourced overrides, making `hasFallbackCandidates = false`.
When the pinned model hits a long-term failure (weekly/monthly quota exhaustion, subscription limit), every subsequent message fails with:
```
⚠️ All models are temporarily rate-limited. Please try again in a few minutes.
```
This is a complete outage — no recovery path short of manually editing `sessions.json`.
Root cause (`src/agents/agent-scope.ts`):
```typescript
if (!canUseConfiguredFallbacks) {
return []; // disables entire fallback chain for user-pinned sessions
}
```
Reproduction:
Fix
Return the global fallback chain instead of `[]`. The user's intent when pinning a model is to prefer it, not to disable recovery when it's unavailable. The pinned model remains the first candidate and is always tried first.
For subagent sessions, the subagent-specific fallback chain is used (same as auto-selected subagents), rather than the generic agent/default chain.
Behavior change
Real behavior proof
Two occurrences captured in local gateway logs on 2026-05-30 (before this fix), both showing the same subagent `zai/glm-5` session hitting the weekly quota with zero fallback candidates:
Occurrence 1 — 2026-05-30T13:14 CST
Session `b640381c` (subagent `a093b44d`), run `f6f7e336`:
```
trajectory event: model.fallback_step
{
"type": "model.fallback_step",
"ts": "2026-05-30T05:14:09.631Z",
"sessionKey": "agent:main:subagent:a093b44d-8405-4e1d-a3bd-00ccb1c2b7e5",
"provider": "zai",
"modelId": "glm-5",
"data": {
"fallbackStepFromModel": "zai/glm-5",
"fallbackStepFromFailureReason": "rate_limit",
"fallbackStepFromFailureDetail": "429 您已达到每周/每月使用上限，您的限额将在 2026-05-31 10:00:31 重置。",
"fallbackStepChainPosition": 1,
"fallbackStepFinalOutcome": "chain_exhausted" // ← chain position 1 + exhausted = 0 candidates
}
}
```
Gateway log at the same moment:⚠️ API rate limit reached. Please try again later.
```
2026-05-30T13:14:09.778+08:00 [ws] ⇄ res ✗ agent errorCode=UNAVAILABLE
errorMessage=FailoverError:
runId=f6f7e336-85e2-4db7-825e-59ec76b32a07
```
Occurrence 2 — 2026-05-30T15:23 CST (2 hours later, same quota, still no fallback)
Session `f455173a` (subagent `90f819dc`), run `625ffb86`:
```
trajectory event: model.fallback_step
{
"type": "model.fallback_step",
"ts": "2026-05-30T07:23:05.922Z",
"sessionKey": "agent:main:subagent:90f819dc-fdfa-4ad3-9b3f-8c16b3cb250a",
"provider": "zai",
"modelId": "glm-5",
"data": {
"fallbackStepFromModel": "zai/glm-5",
"fallbackStepFromFailureReason": "rate_limit",
"fallbackStepFromFailureDetail": "429 您已达到每周/每月使用上限，您的限额将在 2026-05-31 10:00:31 重置。",
"fallbackStepChainPosition": 1,
"fallbackStepFinalOutcome": "chain_exhausted"
}
}
```
Gateway log:⚠️ API rate limit reached. Please try again later.
```
2026-05-30T15:23:06.062+08:00 [ws] ⇄ res ✗ agent errorCode=UNAVAILABLE
errorMessage=FailoverError:
runId=625ffb86-eee4-4e97-8b7b-7bcdad2bb55c
```
Key signal: `fallbackStepChainPosition: 1` with `fallbackStepFinalOutcome: "chain_exhausted"` means the fallback chain had zero candidates — the bug. After the fix, the same quota-exhaustion event would advance to the next model in the configured chain instead of immediately declaring `chain_exhausted`.