-
-
Notifications
You must be signed in to change notification settings - Fork 69.5k
Failover not triggered for zhipuai 1310 "Weekly/Monthly Limit Exhausted" error #33785
Description
Summary
When zhipuai (GLM-5) returns error code 1310: Weekly/Monthly Limit Exhausted, the failover system does not recognize the error and the configured fallback models are never attempted. The agent run ends with the primary model's error even though fallback candidates (e.g. moonshot/kimi-for-coding) are healthy and available.
Steps to reproduce
- Configure
zhipuai/glm-5as the primary model with fallback models inagents.defaults.model.fallbacks - Exhaust the zhipuai weekly/monthly API quota
- Send a message to the agent
Expected: Fallback to the next configured model (e.g. moonshot)
Actual: Run ends with isError=true, no fallback triggered, user sees unresolved error
Evidence from logs
2026-03-03T06:15:47Z [agent/embedded] embedded run agent end: runId=930be05d isError=true error=LLM error 1310: Weekly/Monthly Limit Exhausted. Your limit will reset at 2026-03-06 22:19:54 (request_id: 20260303141547610b7f574d1b44cb)
This error repeated 20+ times over ~17 hours (06:15 UTC through 23:15 UTC on 2026-03-03), with the fallback model never attempted once.
Root cause
Two layers prevent fallback from triggering:
1. Pattern not recognized (failover-matches.ts)
ERROR_PATTERNS.rateLimit does not match "Weekly/Monthly Limit Exhausted":
// src/agents/pi-embedded-helpers/failover-matches.ts:4-15
rateLimit: [
/rate[_ ]limit|too many requests|429/,
"exceeded your current quota",
"resource has been exhausted", // "limit exhausted" — no match
"quota exceeded",
"resource_exhausted",
"usage limit", // "Limit Exhausted" — no match
...
]classifyFailoverReason() returns null.
2. Inner runner skips rotation (run.ts)
Because the error is unclassified:
// src/agents/pi-embedded-runner/run.ts:1105
const failoverFailure = isFailoverAssistantError(lastAssistant); // → false
// run.ts:1142
const shouldRotate = (!aborted && failoverFailure) || (timedOut && !timedOutDuringCompaction);
// → falseshouldRotate is false, so the inner runner returns normally with the error payload. The outer runWithModelFallback loop treats this as a success and never tries the next candidate.
Suggested fix
Add patterns to ERROR_PATTERNS.rateLimit in src/agents/pi-embedded-helpers/failover-matches.ts:
rateLimit: [
...existing,
"limit exhausted",
/weekly\/monthly limit/i,
]Related issues
- Fallback not triggered for ZenMux 'subscription quota limit' errors #11164 — same class of bug (ZenMux
"subscription quota limit"unrecognized). Closed as stale without a fix. - [Bug]: Failover does not trigger on Anthropic "insufficient_quota" due to HTTP 400 masking and missing keywords #23440 — Anthropic
insufficient_quotaunrecognized due to HTTP 400 masking + missing keywords (open) - [Bug]: Failover not triggered when model returns stop_reason: "error" (GLM-5 via OpenRouter) #26970 — GLM-5
stop_reason: "error"unrecognized (fixed)
Environment
- OpenClaw version: 2026.3.2
- Provider: zhipuai (api.z.ai proxy)
- Primary model: zhipuai/glm-5
- Fallback models: moonshot/kimi-for-coding, minimax/MiniMax-M2.5, google/gemini-3.1-pro-preview, anthropic/claude-opus-4-6
- OS: macOS