Skip to content

fix(failover): skip same-model retries for Chinese quota exhaustion - #172173

Merged
steipete merged 6 commits into
openclaw:mainfrom
skyzhao1223:fix/failover-cjk-periodic-usage-limit
Oct 15, 2026
Merged

steipete merged 6 commits into
openclaw:mainfrom
skyzhao1223:fix/failover-cjk-periodic-usage-limit

Conversation

@skyzhao1223

@skyzhao1223 skyzhao1223 commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #172165

What Problem This Solves

MiniMax Token Plan's Chinese quota-exhaustion response (已达到 Token Plan 用量上限, error 2056) without Retry-After enters ten same-model attempts before eligible fallback, delaying every turn by about 2.5 minutes as reported in the issue.

User Impact

The reported MiniMax message and Chinese exhausted-quota wording now bypass same-model retries. Temporary 请求过于频繁 throttles remain retryable, and unrelated context-length caps do not become exhausted usage windows. No configuration or migration is needed.

Why This Change Was Made

The canonical periodic message matcher owns the exact MiniMax wording and existing 额度已用尽 / 配额已用尽 phrases. Retry evidence consumes that predicate instead of maintaining a second Chinese regex. This keeps classification and the retry controller aligned without broad matching of arbitrary 上限 text.

Original fix by @skyzhao1223; maintainer cleanup removes the duplicate matcher and helper-only test file, covers the issue's exhausted-quota wording, and adds consumer-level regression and negative cases.

Evidence

  • Source/history confirmation: main 339bc009284962661e3bcf3fa31ae280e677300c has English-only long-window evidence; the retry controller checks it before spending its rate-limit budget. Historical owner changes preserve the distinction between general classification and same-request retryability.
  • Exact-head CI run 38683910309 passed on 9a3644f978655d51bb732056f6c03bc335bdb052: 48 successful jobs, 29 skipped, zero failures, including openclaw/ci-gate, routed Node tests, production/test typechecks, lint, docs, and boundary checks. All applicable PR checks passed; no reruns or inherited-failure exception were needed.
  • Regression: recovers Chinese rate limits without retrying exhausted quotas: %s checks the exact MiniMax 2056 body without Retry-After, exhausted quota phrases, and two temporary-throttle controls. Exhaustion must produce proceed, zero sleeps, and zero retries; temporary throttles must produce retry, one sleep call, and one retry.
  • Classification corpus regression billing-minimax-chinese-usage-limit supplies bare MiniMax wording so HTTP status/type evidence cannot hide a missing matcher; its ID is included in representativeCorpusIds. keeps transient throttles and context caps out of long windows: %s checks short/unknown window results. These replace the standalone matcher tests.
  • Local static checks: changed-file oxfmt and core oxlint pass. The shared install lacks the unrelated Solid plugin; lint used a temporary copy of the repository config excluding only its ui/**/*.tsx override, retaining all core rules. git diff --check passes.
  • Deslop: duplicate regex and explanatory boilerplate removed; no new classifier, runtime state, or public API. Final Codex branch autoreview: scoped-clean, no actionable P0–P2 findings. Existing reply sanitization is unchanged.
  • Live Gateway/Telegram proof, leases, local tests/typechecks, and production-reverted negative controls were intentionally skipped per maintainer policy. CI is the proof; no current-head local unit or live result is claimed. Tests add no timers, sleeps, networking, or Gateway startup; backoff is mocked. Local test wall time was not measured under this policy. CI logs confirm all five new recovery cases passed in 2–5 ms each (job 116254672552) and the new MiniMax classification case passed in 4 ms (job 116254672751). The added retry-window table has no separate case-level result in the published log, so no individual timing is claimed for it.
  • ClawSweeper review 6169980696: historical P2 retry-owner finding addressed; current code findings none. Helper-only test concern addressed by consumer coverage. Live/Telegram proof-only requests are superseded by the maintainer's PR-CI proof policy.

LOC against merged main: production +8/-3 (net +5), tests +45/-0, docs +2/-0. The five production lines wire the canonical predicate into retry evidence; further deletion would be unrelated cleanup. Compared with the contributor head, the duplicate regex and helper-only test file are removed. Main 2529bba0484e51a4e961eed87700351cd238ba8f was merged immediately before push without rebase or conflict. The subsequent main merge did not change the reviewed owner files or patch. No detached/background effects were added; stale-fact revalidation is not applicable.

A provider that reports an exhausted usage window in Chinese (MiniMax Token Plan
2056: "已达到 Token Plan 用量上限:请升级 Token Plan 套餐或购买积分补充用量。", no
Retry-After header) was classified as an ordinary transient rate limit, so the run
burned the same-model retry spiral on a provider that cannot recover until the
window resets. The English periodic wording already classified correctly.

Extend the periodic-usage predicate with the Chinese usage/quota cap phrasing,
keeping length/context caps and transient rate-limit messages out of that path.
@clawsweeper

clawsweeper Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: XS labels Oct 14, 2026
@clawsweeper clawsweeper Bot added P2 Normal backlog priority with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Oct 14, 2026
@clawsweeper

clawsweeper Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge.

What this changes

This PR recognizes Chinese quota-exhaustion messages and skips futile same-model retries.

Example: MiniMax returns HTTP 429 with “已达到 Token Plan 用量上限” and error 2056, without Retry-After.

  • Before: OpenClaw spends up to nine same-model retries before eligible fallback.
  • After: The exhausted quota bypasses same-model retries while temporary Chinese throttles remain retryable.

Review scores

Measure Result What it means
Overall readiness 🧂 unranked krab (1/6) The focused patch is useful and has successful CI with no identified defect, but required real-behavior evidence remains mock-only.
Proof confidence 🧂 unranked krab (1/6) Real behavior proof is necessary before merge. See Before merge.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: Every turn can wait through repeated retries against an exhausted MiniMax quota before reaching a configured fallback.
Reason: The patch restores the documented exhausted-window recovery contract and preserves temporary throttling.

Merge readiness

⛔ Blocked before merge - 1 item remains

Keep this PR open: it fixes a concrete current-main quota-classification gap, resolves the earlier retry-owner finding, and introduces no identified correctness defect.

Priority: P2
Reviewed head: 9a3644f978655d51bb732056f6c03bc335bdb052

Before merge

  • Add real behavior proof - The new Chinese scenarios exercise production recovery with injected error strings, mocked surrounding runtime, and mocked backoff; successful exact-head CI does not show the MiniMax transport response reaching eligible fallback. Supply redacted terminal output or logs showing MiniMax 2056 without Retry-After reaching fallback with zero same-model retries, and Telegram evidence for the changed terminal quota reply. Redact API keys, IPs, phone numbers, and non-public endpoints; then request re-review. After adding proof, update the PR body, then request a new review with @clawsweeper re-review. The PR author or someone with repository write access can request it.

Findings

None.

Tests

  • Missing end-to-end proof: Provide an after-fix real-transport recovery trace for MiniMax 2056 without Retry-After, including eligible fallback and zero same-model retries, plus the affected terminal reply in Telegram; current-head pre-fix failure results are not supplied.
Agent review details

How this fits together

Provider errors enter shared classification and retry evidence; the embedded recovery controller consumes those facts to choose retries or eligible fallback, and reply formatting consumes the same message matcher.

flowchart TD
A[Provider error] --> B[Canonical message matcher]
B --> C[Failover classification]
B --> D[Retry evidence]
C --> E[Attempt recovery]
D --> E
E --> F[Retry or eligible fallback]
B --> G[Terminal failure reply]
Loading

Technical review

Best possible solution:

Land the focused canonical-matcher repair after verifying the reported response through production recovery.

Do we have a high-confidence way to reproduce the issue?

Current-main source shows the reported Chinese 429 missing long-window evidence and entering the retry budget, consistent with the linked reporter's logs; no target code was executed during this review.

Is this the best way to solve the issue?

Reusing the existing periodic matcher fixes the owning retry decision without adding competing classification logic, configuration, or storage.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 02e70aba4762.

Provenance checked

  • Periodic usage-limit classification keeps the original intent (a65d70f: Recognize Zhipu weekly/monthly exhaustion so configured fallback can proceed.)
  • Shared retry-evidence responsibility keeps the original intent (refactor(agents): absorb overflow, retry, and matcher duplicates into the failover substrate #121817: Centralize retry evidence while keeping general classification broader than same-attempt retryability.)
  • Long-window retry rejection keeps the original intent (cb1b29d: Preserve recovery ownership and distinguish exhausted usage windows from temporary retry floors.)
  • Recovery regression contract keeps the original intent (b8ddad0: Avoid doomed same-model retries for exhausted daily, weekly, or monthly limits.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

Likely related people:

  • Peter Steinberger: Raw commit 9e01174 adds src/agents/failover/retry-evidence.ts:149 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 9e01174cb9ab; files: src/agents/failover/retry-evidence.ts)
  • Mitchell Etzel: Raw commit b8ddad0 adds src/agents/failover/retry-evidence.ts:29 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: medium; commits: b8ddad0e8037; files: src/agents/failover/retry-evidence.ts)
  • zhouhe-xydt: Raw commit a65d70f adds src/agents/pi-embedded-helpers/failover-matches.ts:3 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: medium; commits: a65d70f84b4e; files: src/agents/pi-embedded-helpers/failover-matches.ts)
  • Zhang Xiaofeng: Raw commit a090092 adds src/agents/pi-embedded-helpers/failover-matches.ts:87 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: medium; commits: a0900926c388; files: src/agents/pi-embedded-helpers/failover-matches.ts)

PR surface

Source +13, Tests +37, Docs +2. Total +52 across 7 files. Added test files: 0.

View PR surface stats
Area Files Added Removed Net
Source 3 16 3 +13
Tests 3 37 0 +37
Docs 1 2 0 +2
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 7 55 3 +52

Review metrics

Metric Value Why it matters
Production versus test growth Production +8/-3 lines; tests +45/-0 lines The small production increase reuses the canonical matcher; consumer regressions cover the reported response and preserve transient-throttle behavior.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #172165
Summary: This PR is the candidate repair for the concrete MiniMax Chinese quota-exhaustion report.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Labels

Label changes:

No label changes.

Label justifications:

  • P2: A limited provider-message mismatch causes repeated avoidable delays before eligible fallback.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask.
  • proof: telegram-e2e: This PR needs Telegram Test Server proof with the repository E2E skill. The periodic matcher changes terminal quota-reply formatting through renderRateLimitReplyCopy; Telegram can observe that reply when fallback does not succeed.

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Provide the real-transport recovery and Telegram terminal-reply evidence specified in the proof assessment.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

History

Review history (2 earlier review cycles)
  • reviewed 2026-10-14T13:51:33.555Z sha 9522803 :: needs real behavior proof before merge. :: [P2] Apply Chinese exhaustion recognition to the retry-window owner
  • reviewed 2026-10-14T14:37:44.150Z sha f36e490 :: needs real behavior proof before merge. :: none

Reviewed October 15, 2026, 2:45 AM ET / 06:45 UTC (Revision 3).

Review finding (P2): classifying the MiniMax Token Plan 2056 message as a periodic
usage limit did not stop the same-model retry spiral, because the retry-window owner
(retry-evidence.ts) kept its English-only long-window evidence and the controller
therefore still entered the retry budget.

hasLongWindowRateLimitEvidence now also matches the Chinese usage/quota cap phrasing;
CJK text has no word boundaries, so it checks separately from the ASCII pattern, the
same way the short-window owner already handles its Chinese phrases.
@skyzhao1223

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review — the P2 finding is addressed at the owner it named: hasLongWindowRateLimitEvidence in retry-evidence.ts now matches the Chinese usage-cap phrasing (checked separately from the ASCII pattern because CJK has no word boundaries), and the new regression drives the recovery controller with the exact reported 429 body — red before it returned { action: 'retry' }, green after with zero same-model retries.

@clawsweeper

clawsweeper Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
Exact review queued.

Re-review progress:

@clawsweeper clawsweeper Bot added the proof: telegram-e2e This PR needs Telegram Test Server proof with the repository E2E skill. label Oct 14, 2026
steipete and others added 4 commits October 14, 2026 21:50
Recognize the reported MiniMax Token Plan 2056 wording and exhausted Chinese quotas at the canonical message-pattern owner. Reuse that predicate for retry decisions and retain transient-throttle negative controls at the recovery boundary.

Fixes openclaw#172165. Original fix by @skyzhao1223; maintainer cleanup removes the duplicate matcher and replaces helper-only assertions with consumer coverage.

Co-authored-by: skyzhao1223 <[email protected]>
Replace incorrect verbatim reply expectations with the reported MiniMax wording in the classification corpus, including its representative ID. Existing reply sanitization remains unchanged; the fix concerns retry and classification behavior.
@openclaw-barnacle openclaw-barnacle Bot added the docs Improvements or additions to documentation label Oct 15, 2026
@steipete steipete changed the title fix(failover): treat Chinese usage-cap messages as periodic limits fix(failover): skip same-model retries for Chinese quota exhaustion Oct 15, 2026
@steipete

Copy link
Copy Markdown
Contributor

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
Exact review queued.

Re-review progress:

@steipete
steipete merged commit e0e08a6 into openclaw:main Oct 15, 2026
143 of 145 checks passed
@steipete

Copy link
Copy Markdown
Contributor

Merged via squash.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Oct 16, 2026
…penclaw#172173)

* fix(failover): treat Chinese usage-cap messages as periodic limits

A provider that reports an exhausted usage window in Chinese (MiniMax Token Plan
2056: "已达到 Token Plan 用量上限:请升级 Token Plan 套餐或购买积分补充用量。", no
Retry-After header) was classified as an ordinary transient rate limit, so the run
burned the same-model retry spiral on a provider that cannot recover until the
window resets. The English periodic wording already classified correctly.

Extend the periodic-usage predicate with the Chinese usage/quota cap phrasing,
keeping length/context caps and transient rate-limit messages out of that path.

* fix(failover): recognize the Chinese usage cap in the retry window

Review finding (P2): classifying the MiniMax Token Plan 2056 message as a periodic
usage limit did not stop the same-model retry spiral, because the retry-window owner
(retry-evidence.ts) kept its English-only long-window evidence and the controller
therefore still entered the retry budget.

hasLongWindowRateLimitEvidence now also matches the Chinese usage/quota cap phrasing;
CJK text has no word boundaries, so it checks separately from the ASCII pattern, the
same way the short-window owner already handles its Chinese phrases.

* fix(failover): share Chinese quota exhaustion evidence

Recognize the reported MiniMax Token Plan 2056 wording and exhausted Chinese quotas at the canonical message-pattern owner. Reuse that predicate for retry decisions and retain transient-throttle negative controls at the recovery boundary.

Fixes openclaw#172165. Original fix by @skyzhao1223; maintainer cleanup removes the duplicate matcher and replaces helper-only assertions with consumer coverage.

Co-authored-by: skyzhao1223 <[email protected]>

* test(failover): cover Chinese quota classification at its consumer

Replace incorrect verbatim reply expectations with the reported MiniMax wording in the classification corpus, including its representative ID. Existing reply sanitization remains unchanged; the fix concerns retry and classification behavior.

---------

Co-authored-by: skyzhao1223 <[email protected]>
Co-authored-by: Peter Steinberger <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling docs Improvements or additions to documentation P2 Normal backlog priority with limited blast radius. proof: telegram-e2e This PR needs Telegram Test Server proof with the repository E2E skill. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Chinese long-window rate-limit messages not recognized — exhausted quota retries 10x instead of failing over directly

2 participants