fix(auth): flat 30s cooldown replaces exponential backoff#31962
fix(auth): flat 30s cooldown replaces exponential backoff#31962teodorangel wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7b8ecad0d2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| void errorCount; // preserve signature | ||
| return 30 * 1000; |
There was a problem hiding this comment.
Keep longer backoff for persistent non-rate-limit errors
calculateAuthProfileCooldownMs now returns a flat 30s for every non-billing failure, but computeNextProfileUsageStats uses this path for reasons like auth, model_not_found, format, and unknown (not just rate_limit). In persistent misconfiguration cases (e.g., invalid credentials classified as auth), this change makes the same broken profile re-enter rotation every 30s indefinitely, creating repeated failing calls/log churn where the previous backoff quickly reduced retry pressure; consider scoping the flat cooldown to true rate-limit/transient reasons only.
Useful? React with 👍 / 👎.
Greptile SummaryReplaces exponential auth profile cooldown (1min → 5min → 25min → 60min) with flat 30-second cooldown to match real provider rate-limit windows. Key Changes:
Issues Found:
Rationale: Anthropic's actual rate limits clear in ~60s, but 3-4 errors locked profiles for 25-60 minutes. Production testing (6 weeks) shows 30s cooldown prevents retry storms while enabling fast recovery. Confidence Score: 2/5
Last reviewed commit: 7b8ecad |
| export function calculateAuthProfileCooldownMs(errorCount: number): number { | ||
| const normalized = Math.max(1, errorCount); | ||
| return Math.min( | ||
| 60 * 60 * 1000, // 1 hour max | ||
| 60 * 1000 * 5 ** Math.min(normalized - 1, 3), | ||
| ); | ||
| // Kai fork: flat 30-second cooldown replaces exponential backoff. | ||
| // Original formula: 60s * 5^(errors-1), capped at 1 hour. | ||
| // At 3 errors = 25 min, 4 errors = 60 min — way beyond Anthropic's actual limits. | ||
| // Author: Teodor 🔧 (Feb 18, 2026) | ||
| void errorCount; // preserve signature | ||
| return 30 * 1000; | ||
| } |
There was a problem hiding this comment.
tests in src/agents/auth-profiles.markauthprofilefailure.test.ts:201-209 expect old exponential behavior and will fail
| export function calculateAuthProfileCooldownMs(errorCount: number): number { | |
| const normalized = Math.max(1, errorCount); | |
| return Math.min( | |
| 60 * 60 * 1000, // 1 hour max | |
| 60 * 1000 * 5 ** Math.min(normalized - 1, 3), | |
| ); | |
| // Kai fork: flat 30-second cooldown replaces exponential backoff. | |
| // Original formula: 60s * 5^(errors-1), capped at 1 hour. | |
| // At 3 errors = 25 min, 4 errors = 60 min — way beyond Anthropic's actual limits. | |
| // Author: Teodor 🔧 (Feb 18, 2026) | |
| void errorCount; // preserve signature | |
| return 30 * 1000; | |
| } | |
| export function calculateAuthProfileCooldownMs(errorCount: number): number { | |
| // Kai fork: flat 30-second cooldown replaces exponential backoff. | |
| // Original formula: 60s * 5^(errors-1), capped at 1 hour. | |
| // At 3 errors = 25 min, 4 errors = 60 min — way beyond Anthropic's actual limits. | |
| // Author: Teodor 🔧 (Feb 18, 2026) | |
| void errorCount; // preserve signature | |
| return 30 * 1000; | |
| } | |
| // Note: Update tests in markauthprofilefailure.test.ts to expect 30_000 for all error counts |
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/auth-profiles/usage.ts
Line: 278-285
Comment:
tests in `src/agents/auth-profiles.markauthprofilefailure.test.ts:201-209` expect old exponential behavior and will fail
```suggestion
export function calculateAuthProfileCooldownMs(errorCount: number): number {
// Kai fork: flat 30-second cooldown replaces exponential backoff.
// Original formula: 60s * 5^(errors-1), capped at 1 hour.
// At 3 errors = 25 min, 4 errors = 60 min — way beyond Anthropic's actual limits.
// Author: Teodor 🔧 (Feb 18, 2026)
void errorCount; // preserve signature
return 30 * 1000;
}
// Note: Update tests in markauthprofilefailure.test.ts to expect 30_000 for all error counts
```
How can I resolve this? If you propose a fix, please make it concise.
Additional Comments (1)
Prompt To Fix With AIThis is a comment left during a code review.
Path: src/agents/auth-profiles/usage.ts
Line: 497-500
Comment:
update comment to reflect flat 30s cooldown (no longer exponential)
```suggestion
/**
* Mark a profile as failed/rate-limited. Applies flat 30-second cooldown.
* Uses store lock to avoid overwriting concurrent usage updates.
*/
```
How can I resolve this? If you propose a fix, please make it concise. |
15817ae to
c59e44b
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c59e44bdaf
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // rate-limit windows (Anthropic ~60s). The exponential curve makes profiles | ||
| // unusable long after the provider has cleared the limit. | ||
| void errorCount; // preserve signature | ||
| return 30 * 1000; |
There was a problem hiding this comment.
Keep non-rate-limit failures on a longer cooldown
Returning a hard-coded 30s here applies the same retry cadence to persistent failures, not just throttling: classifyFailoverReason emits values like session_expired/model_not_found (src/agents/pi-embedded-helpers/errors.ts:916-921), and maybeMarkAuthProfileFailure records every non-timeout reason (src/agents/pi-embedded-runner/run.ts:663-670). With this change, those misconfigured or expired profiles re-enter rotation every 30 seconds indefinitely, causing repeated failed attempts and log churn in production where the previous backoff reduced retry pressure over time.
Useful? React with 👍 / 👎.
Original formula punished errors way beyond Anthropic's actual rate limits: 1 error = 1 min, 2 = 5 min, 3 = 25 min, 4 = 60 min (capped) Now: always 30 seconds regardless of error count. Author: Teodor 🔧 Original: Patch 5 (dist, Feb 18, 2026) Ported to source: Mar 2, 2026
c59e44b to
a7b64d1
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a7b64d1e64
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // rate-limit windows (Anthropic ~60s). The exponential curve makes profiles | ||
| // unusable long after the provider has cleared the limit. | ||
| void errorCount; // preserve signature | ||
| return 30 * 1000; |
There was a problem hiding this comment.
Align flat cooldown with documented 60s rate-limit window
Returning a hard-coded 30_000 here causes retries to happen before the rate-limit window you describe in the same function comments as "Anthropic ~60s" has elapsed, so single-profile setups will incur at least one avoidable extra failed request after each throttle event. Because markAuthProfileFailure reapplies this same cooldown on every rate_limit failure, sustained throttling can devolve into a 30-second fail loop with unnecessary API/log churn; this should use a window that is at least as long as the documented provider limit or be provider-specific.
Useful? React with 👍 / 👎.
- Replace void errorCount with _errorCount parameter convention - Use 30_000 literal instead of 30 * 1000 - Restore kilocode cooldown bypass (still a valid provider)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d26bcf1204
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // At 3 errors = 25 min, 4 errors = 60 min — far exceeds real provider | ||
| // rate-limit windows (Anthropic ~60s). The exponential curve makes profiles | ||
| // unusable long after the provider has cleared the limit. | ||
| return 30_000; |
There was a problem hiding this comment.
Restore stronger backoff for persistent profile failures
calculateAuthProfileCooldownMs now hard-codes 30_000 for every failure, but computeNextProfileUsageStats applies this path to all non-billing/non-auth_permanent reasons, not just throttling (src/agents/auth-profiles/usage.ts, computeNextProfileUsageStats). In the runner, markAuthProfileFailure is recorded for every non-timeout failover (src/agents/pi-embedded-runner/run.ts:656-670), so persistent misconfigurations like auth, model_not_found, or format will re-enter rotation every 30 seconds indefinitely, causing repeated upstream failures/log churn where the previous escalating backoff reduced retry pressure.
Useful? React with 👍 / 👎.
…mit message Combines ideas from PRs openclaw#45113, openclaw#31962, and openclaw#45763 to address three cooldown-related issues: 1. Stepped cooldown (30s → 1m → 5m cap) replaces the aggressive exponential formula (1m → 5m → 25m → 1h) that locked out providers for far longer than the actual API rate-limit window. 2. Per-model cooldown scoping: rate_limit cooldowns now record which model triggered them. When a different model on the same auth profile is requested, the cooldown is bypassed — so one model hitting a 429 no longer blocks all other models on the same provider. 3. FallbackSummaryError with soonest-expiry countdown: when all candidates are exhausted, the user sees a clear message like '⚠️ Rate-limited — ready in ~28s' instead of a generic failure. Files changed: - types.ts: add cooldownReason/cooldownModel to ProfileUsageStats - usage.ts: stepped formula, model-aware isProfileInCooldown, modelId threading through computeNextProfileUsageStats/markAuthProfileFailure - model-fallback.ts: FallbackSummaryError class, model-aware availability check, soonestCooldownExpiry computation - pi-embedded-runner/run.ts: thread modelId into failure recording - agent-runner-execution.ts: buildCopilotCooldownMessage helper, rate-limit detection branch in error handler - usage.test.ts: update expected cooldown value (60s → 30s)
|
Closing — this was addressed via a different approach. What shipped instead:
Why this PR is no longer needed:
Thank you for the PR, @teodorangel — the cooldown escalation problem you called out was real and helped inform the fix that landed. |
Problem
calculateAuthProfileCooldownMs()insrc/agents/auth-profiles/usage.tsuses exponential backoff:60s × 5^(errors-1), capped at 1 hour.At 3 consecutive errors = 25 minutes. At 4 errors = 60 minutes. These durations far exceed any provider's actual rate-limit windows (Anthropic's is ~60 seconds). The result: after a brief rate-limit burst, the profile becomes unusable for up to an hour while the provider has long since cleared the limit.
This is especially painful during cron job collisions, where 2-3 rapid failures can lock out the only configured profile for 25+ minutes.
Fix
Replace the exponential formula with a flat 30-second cooldown regardless of error count. This matches real-world provider rate-limit windows while still preventing immediate retry storms.
Impact
src/agents/auth-profiles/usage.tsTesting
Running in production for 6 weeks. Before: cron collisions at 5 AM would lock out the Anthropic profile for 25+ minutes, causing all subsequent cron jobs to fail with 300s timeouts. After: recovery happens within 30 seconds, subsequent jobs succeed normally.