Skip to content

Commit 6e962d8

Browse files
altaywtfaltaywtf
authored
fix(agents): handle overloaded failover separately (#38301)
* fix(agents): skip auth-profile failure on overload * fix(agents): note overload auth-profile fallback fix * fix(agents): classify overloaded failures separately * fix(agents): back off before overload failover * fix(agents): tighten overload probe and backoff state * fix(agents): persist overloaded cooldown across runs * fix(agents): tighten overloaded status handling * test(agents): add overload regression coverage * fix(agents): restore runner imports after rebase * test(agents): add overload fallback integration coverage * fix(agents): harden overloaded failover abort handling * test(agents): tighten overload classifier coverage * test(agents): cover all-overloaded fallback exhaustion * fix(cron): retry overloaded fallback summaries * fix(cron): treat HTTP 529 as overloaded retry
1 parent 110ca23 commit 6e962d8

36 files changed

Lines changed: 1036 additions & 84 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,7 @@ Docs: https://docs.openclaw.ai
121121
- Security/auth labels: remove token and API-key snippets from user-facing auth status labels so `/status` and `/models` do not expose credential fragments. (#33262) thanks @cu1ch3n.
122122
- Auth/credential semantics: align profile eligibility + probe diagnostics with SecretRef/expiry rules and harden browser download atomic writes. (#33733) thanks @joshavant.
123123
- Security/audit denyCommands guidance: suggest likely exact node command IDs for unknown `gateway.nodes.denyCommands` entries so ineffective denylist entries are easier to correct. (#29713) thanks @liquidhorizon88-bot.
124+
- Agents/overload failover handling: classify overloaded provider failures separately from rate limits/status timeouts, add short overload backoff before retry/failover, record overloaded prompt/assistant failures as transient auth-profile cooldowns (with probeable same-provider fallback) instead of treating them like persistent auth/billing failures, and keep one-shot cron retry classification aligned so overloaded fallback summaries still count as transient retries.
124125
- Docs/security hardening guidance: document Docker `DOCKER-USER` + UFW policy and add cross-linking from Docker install docs for VPS/public-host setups. (#27613) thanks @dorukardahan.
125126
- Docs/security threat-model links: replace relative `.md` links with Mintlify-compatible root-relative routes in security docs to prevent broken internal navigation. (#27698) thanks @clawdoo.
126127
- Plugins/Update integrity drift: avoid false integrity drift prompts when updating npm-installed plugins from unpinned specs, while keeping drift checks for exact pinned versions. (#37179) Thanks @vincentkoc.

‎docs/automation/cron-jobs.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -370,6 +370,7 @@ When a job fails, OpenClaw classifies errors as **transient** (retryable) or **p
370370
### Transient errors (retried)
371371

372372
- Rate limit (429, too many requests, resource exhausted)
373+
- Provider overload (for example Anthropic `529 overloaded_error`, overload fallback summaries)
373374
- Network errors (timeout, ECONNRESET, fetch failed, socket)
374375
- Server errors (5xx)
375376
- Cloudflare-related errors
@@ -407,7 +408,7 @@ Configure `cron.retry` to override these defaults (see [Configuration](/automati
407408
retry: {
408409
maxAttempts: 3,
409410
backoffMs: [60000, 120000, 300000],
410-
retryOn: ["rate_limit", "network", "server_error"],
411+
retryOn: ["rate_limit", "overloaded", "network", "server_error"],
411412
},
412413
webhook: "https://example.invalid/legacy", // deprecated fallback for stored notify:true jobs
413414
webhookToken: "replace-with-dedicated-webhook-token", // optional bearer token for webhook mode
@@ -665,7 +666,7 @@ openclaw system event --mode now --text "Next heartbeat: check battery."
665666
- OpenClaw applies exponential retry backoff for recurring jobs after consecutive errors:
666667
30s, 1m, 5m, 15m, then 60m between retries.
667668
- Backoff resets automatically after the next successful run.
668-
- One-shot (`at`) jobs retry transient errors (rate limit, network, server_error) up to 3 times with backoff; permanent errors disable immediately. See [Retry policy](/automation/cron-jobs#retry-policy).
669+
- One-shot (`at`) jobs retry transient errors (rate limit, overloaded, network, server_error) up to 3 times with backoff; permanent errors disable immediately. See [Retry policy](/automation/cron-jobs#retry-policy).
669670

670671
### Telegram delivers to the wrong place
671672

‎src/agents/auth-profiles.markauthprofilefailure.test.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,22 @@ describe("markAuthProfileFailure", () => {
114114
expect(reloaded.usageStats?.["anthropic:default"]?.cooldownUntil).toBe(firstCooldownUntil);
115115
});
116116
});
117+
it("records overloaded failures in the cooldown bucket", async () => {
118+
await withAuthProfileStore(async ({ agentDir, store }) => {
119+
await markAuthProfileFailure({
120+
store,
121+
profileId: "anthropic:default",
122+
reason: "overloaded",
123+
agentDir,
124+
});
125+
126+
const stats = store.usageStats?.["anthropic:default"];
127+
expect(typeof stats?.cooldownUntil).toBe("number");
128+
expect(stats?.disabledUntil).toBeUndefined();
129+
expect(stats?.disabledReason).toBeUndefined();
130+
expect(stats?.failureCounts?.overloaded).toBe(1);
131+
});
132+
});
117133
it("disables auth_permanent failures via disabledUntil (like billing)", async () => {
118134
await withAuthProfileStore(async ({ agentDir, store }) => {
119135
await markAuthProfileFailure({

‎src/agents/auth-profiles/types.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ export type AuthProfileFailureReason =
3939
| "auth"
4040
| "auth_permanent"
4141
| "format"
42+
| "overloaded"
4243
| "rate_limit"
4344
| "billing"
4445
| "timeout"

‎src/agents/auth-profiles/usage.test.ts‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,24 @@ describe("resolveProfilesUnavailableReason", () => {
177177
).toBe("auth");
178178
});
179179

180+
it("returns overloaded for active overloaded cooldown windows", () => {
181+
const now = Date.now();
182+
const store = makeStore({
183+
"anthropic:default": {
184+
cooldownUntil: now + 60_000,
185+
failureCounts: { overloaded: 2, rate_limit: 1 },
186+
},
187+
});
188+
189+
expect(
190+
resolveProfilesUnavailableReason({
191+
store,
192+
profileIds: ["anthropic:default"],
193+
now,
194+
}),
195+
).toBe("overloaded");
196+
});
197+
180198
it("falls back to rate_limit when active cooldown has no reason history", () => {
181199
const now = Date.now();
182200
const store = makeStore({

‎src/agents/auth-profiles/usage.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ const FAILURE_REASON_PRIORITY: AuthProfileFailureReason[] = [
99
"billing",
1010
"format",
1111
"model_not_found",
12+
"overloaded",
1213
"timeout",
1314
"rate_limit",
1415
"unknown",
@@ -35,7 +36,7 @@ export function resolveProfileUnusableUntil(
3536
}
3637

3738
/**
38-
* Check if a profile is currently in cooldown (due to rate limiting or errors).
39+
* Check if a profile is currently in cooldown (due to rate limits, overload, or other transient failures).
3940
*/
4041
export function isProfileInCooldown(
4142
store: AuthProfileStore,
@@ -508,7 +509,7 @@ export async function markAuthProfileFailure(params: {
508509
}
509510

510511
/**
511-
* Mark a profile as failed/rate-limited. Applies exponential backoff cooldown.
512+
* Mark a profile as transiently failed. Applies exponential backoff cooldown.
512513
* Cooldown times: 1min, 5min, 25min, max 1 hour.
513514
* Uses store lock to avoid overwriting concurrent usage updates.
514515
*/

‎src/agents/failover-error.test.ts‎

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ describe("failover-error", () => {
7575
expect(resolveFailoverReasonFromError({ status: 522 })).toBeNull();
7676
expect(resolveFailoverReasonFromError({ status: 523 })).toBeNull();
7777
expect(resolveFailoverReasonFromError({ status: 524 })).toBeNull();
78-
expect(resolveFailoverReasonFromError({ status: 529 })).toBe("rate_limit");
78+
expect(resolveFailoverReasonFromError({ status: 529 })).toBe("overloaded");
7979
});
8080

8181
it("classifies documented provider error shapes at the error boundary", () => {
@@ -90,7 +90,7 @@ describe("failover-error", () => {
9090
status: 529,
9191
message: ANTHROPIC_OVERLOADED_PAYLOAD,
9292
}),
93-
).toBe("rate_limit");
93+
).toBe("overloaded");
9494
expect(
9595
resolveFailoverReasonFromError({
9696
status: 429,
@@ -126,7 +126,22 @@ describe("failover-error", () => {
126126
status: 503,
127127
message: GROQ_SERVICE_UNAVAILABLE_MESSAGE,
128128
}),
129+
).toBe("overloaded");
130+
});
131+
132+
it("keeps status-only 503s conservative unless the payload is clearly overloaded", () => {
133+
expect(
134+
resolveFailoverReasonFromError({
135+
status: 503,
136+
message: "Internal database error",
137+
}),
129138
).toBe("timeout");
139+
expect(
140+
resolveFailoverReasonFromError({
141+
status: 503,
142+
message: '{"error":{"message":"The model is overloaded. Please try later"}}',
143+
}),
144+
).toBe("overloaded");
130145
});
131146

132147
it("treats 400 insufficient_quota payloads as billing instead of format", () => {
@@ -151,6 +166,14 @@ describe("failover-error", () => {
151166
).toBe("rate_limit");
152167
});
153168

169+
it("treats overloaded provider payloads as overloaded", () => {
170+
expect(
171+
resolveFailoverReasonFromError({
172+
message: ANTHROPIC_OVERLOADED_PAYLOAD,
173+
}),
174+
).toBe("overloaded");
175+
});
176+
154177
it("keeps raw-text 402 weekly/monthly limit errors in billing", () => {
155178
expect(
156179
resolveFailoverReasonFromError({
@@ -221,6 +244,10 @@ describe("failover-error", () => {
221244
expect(err?.model).toBe("claude-opus-4-5");
222245
});
223246

247+
it("maps overloaded to a 503 fallback status", () => {
248+
expect(resolveFailoverStatus("overloaded")).toBe(503);
249+
});
250+
224251
it("coerces format errors with a 400 status", () => {
225252
const err = coerceToFailoverError("invalid request format", {
226253
provider: "google",

‎src/agents/failover-error.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,8 @@ export function resolveFailoverStatus(reason: FailoverReason): number | undefine
4949
return 402;
5050
case "rate_limit":
5151
return 429;
52+
case "overloaded":
53+
return 503;
5254
case "auth":
5355
return 401;
5456
case "auth_permanent":

‎src/agents/model-fallback.probe.test.ts‎

Lines changed: 43 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ function expectPrimaryProbeSuccess(
5353
expect(result.result).toBe(expectedResult);
5454
expect(run).toHaveBeenCalledTimes(1);
5555
expect(run).toHaveBeenCalledWith("openai", "gpt-4.1-mini", {
56-
allowRateLimitCooldownProbe: true,
56+
allowTransientCooldownProbe: true,
5757
});
5858
}
5959

@@ -200,10 +200,48 @@ describe("runWithModelFallback – probe logic", () => {
200200
expect(result.result).toBe("fallback-ok");
201201
expect(run).toHaveBeenCalledTimes(2);
202202
expect(run).toHaveBeenNthCalledWith(1, "openai", "gpt-4.1-mini", {
203-
allowRateLimitCooldownProbe: true,
203+
allowTransientCooldownProbe: true,
204204
});
205205
expect(run).toHaveBeenNthCalledWith(2, "anthropic", "claude-haiku-3-5", {
206-
allowRateLimitCooldownProbe: true,
206+
allowTransientCooldownProbe: true,
207+
});
208+
});
209+
210+
it("attempts non-primary fallbacks during overloaded cooldown after primary probe failure", async () => {
211+
const cfg = makeCfg({
212+
agents: {
213+
defaults: {
214+
model: {
215+
primary: "openai/gpt-4.1-mini",
216+
fallbacks: ["anthropic/claude-haiku-3-5", "google/gemini-2-flash"],
217+
},
218+
},
219+
},
220+
} as Partial<OpenClawConfig>);
221+
222+
mockedIsProfileInCooldown.mockReturnValue(true);
223+
mockedGetSoonestCooldownExpiry.mockReturnValue(NOW + 30 * 1000);
224+
mockedResolveProfilesUnavailableReason.mockReturnValue("overloaded");
225+
226+
const run = vi
227+
.fn()
228+
.mockRejectedValueOnce(Object.assign(new Error("service overloaded"), { status: 503 }))
229+
.mockResolvedValue("fallback-ok");
230+
231+
const result = await runWithModelFallback({
232+
cfg,
233+
provider: "openai",
234+
model: "gpt-4.1-mini",
235+
run,
236+
});
237+
238+
expect(result.result).toBe("fallback-ok");
239+
expect(run).toHaveBeenCalledTimes(2);
240+
expect(run).toHaveBeenNthCalledWith(1, "openai", "gpt-4.1-mini", {
241+
allowTransientCooldownProbe: true,
242+
});
243+
expect(run).toHaveBeenNthCalledWith(2, "anthropic", "claude-haiku-3-5", {
244+
allowTransientCooldownProbe: true,
207245
});
208246
});
209247

@@ -326,10 +364,10 @@ describe("runWithModelFallback – probe logic", () => {
326364
});
327365

328366
expect(run).toHaveBeenNthCalledWith(1, "openai", "gpt-4.1-mini", {
329-
allowRateLimitCooldownProbe: true,
367+
allowTransientCooldownProbe: true,
330368
});
331369
expect(run).toHaveBeenNthCalledWith(2, "openai", "gpt-4.1-mini", {
332-
allowRateLimitCooldownProbe: true,
370+
allowTransientCooldownProbe: true,
333371
});
334372
});
335373
});

0 commit comments

Comments
 (0)