Skip to content

fix(auth): flat 30s cooldown replaces exponential backoff#31962

Closed
teodorangel wants to merge 3 commits into
openclaw:mainfrom
teodorangel:fix/auth-cooldown-flat
Closed

fix(auth): flat 30s cooldown replaces exponential backoff#31962
teodorangel wants to merge 3 commits into
openclaw:mainfrom
teodorangel:fix/auth-cooldown-flat

Conversation

@teodorangel

Copy link
Copy Markdown

Problem

calculateAuthProfileCooldownMs() in src/agents/auth-profiles/usage.ts uses exponential backoff: 60s × 5^(errors-1), capped at 1 hour.

At 3 consecutive errors = 25 minutes. At 4 errors = 60 minutes. These durations far exceed any provider's actual rate-limit windows (Anthropic's is ~60 seconds). The result: after a brief rate-limit burst, the profile becomes unusable for up to an hour while the provider has long since cleared the limit.

This is especially painful during cron job collisions, where 2-3 rapid failures can lock out the only configured profile for 25+ minutes.

Fix

Replace the exponential formula with a flat 30-second cooldown regardless of error count. This matches real-world provider rate-limit windows while still preventing immediate retry storms.

Impact

  • Single function change in src/agents/auth-profiles/usage.ts
  • Profiles recover from rate-limit events in 30 seconds instead of 25-60 minutes
  • No change to the cooldown-clear-on-success behavior

Testing

Running in production for 6 weeks. Before: cron collisions at 5 AM would lock out the Anthropic profile for 25+ minutes, causing all subsequent cron jobs to fail with 300s timeouts. After: recovery happens within 30 seconds, subsequent jobs succeed normally.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: XS labels Mar 2, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7b8ecad0d2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/auth-profiles/usage.ts Outdated
Comment on lines +283 to +284
void errorCount; // preserve signature
return 30 * 1000;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep longer backoff for persistent non-rate-limit errors

calculateAuthProfileCooldownMs now returns a flat 30s for every non-billing failure, but computeNextProfileUsageStats uses this path for reasons like auth, model_not_found, format, and unknown (not just rate_limit). In persistent misconfiguration cases (e.g., invalid credentials classified as auth), this change makes the same broken profile re-enter rotation every 30s indefinitely, creating repeated failing calls/log churn where the previous backoff quickly reduced retry pressure; consider scoping the flat cooldown to true rate-limit/transient reasons only.

Useful? React with 👍 / 👎.

@greptile-apps

greptile-apps Bot commented Mar 2, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Replaces exponential auth profile cooldown (1min → 5min → 25min → 60min) with flat 30-second cooldown to match real provider rate-limit windows.

Key Changes:

  • calculateAuthProfileCooldownMs() now returns constant 30s instead of exponential backoff
  • Import statement reordering (non-functional)

Issues Found:

  • Critical: Unit tests in src/agents/auth-profiles.markauthprofilefailure.test.ts still expect old exponential values and will fail
  • Minor: JSDoc comment at line 497 still describes old exponential behavior

Rationale: Anthropic's actual rate limits clear in ~60s, but 3-4 errors locked profiles for 25-60 minutes. Production testing (6 weeks) shows 30s cooldown prevents retry storms while enabling fast recovery.

Confidence Score: 2/5

  • Cannot merge - tests will fail with current changes
  • Logic is sound and production-tested, but unit tests still expect exponential backoff behavior and will fail in CI. Must update tests before merging.
  • Check src/agents/auth-profiles.markauthprofilefailure.test.ts - needs test updates to match new behavior

Last reviewed commit: 7b8ecad

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 file reviewed, 2 comments

Edit Code Review Agent Settings | Greptile

Comment thread src/agents/auth-profiles/usage.ts Outdated
Comment on lines 278 to 285
export function calculateAuthProfileCooldownMs(errorCount: number): number {
const normalized = Math.max(1, errorCount);
return Math.min(
60 * 60 * 1000, // 1 hour max
60 * 1000 * 5 ** Math.min(normalized - 1, 3),
);
// Kai fork: flat 30-second cooldown replaces exponential backoff.
// Original formula: 60s * 5^(errors-1), capped at 1 hour.
// At 3 errors = 25 min, 4 errors = 60 min — way beyond Anthropic's actual limits.
// Author: Teodor 🔧 (Feb 18, 2026)
void errorCount; // preserve signature
return 30 * 1000;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests in src/agents/auth-profiles.markauthprofilefailure.test.ts:201-209 expect old exponential behavior and will fail

Suggested change
export function calculateAuthProfileCooldownMs(errorCount: number): number {
const normalized = Math.max(1, errorCount);
return Math.min(
60 * 60 * 1000, // 1 hour max
60 * 1000 * 5 ** Math.min(normalized - 1, 3),
);
// Kai fork: flat 30-second cooldown replaces exponential backoff.
// Original formula: 60s * 5^(errors-1), capped at 1 hour.
// At 3 errors = 25 min, 4 errors = 60 min — way beyond Anthropic's actual limits.
// Author: Teodor 🔧 (Feb 18, 2026)
void errorCount; // preserve signature
return 30 * 1000;
}
export function calculateAuthProfileCooldownMs(errorCount: number): number {
// Kai fork: flat 30-second cooldown replaces exponential backoff.
// Original formula: 60s * 5^(errors-1), capped at 1 hour.
// At 3 errors = 25 min, 4 errors = 60 min — way beyond Anthropic's actual limits.
// Author: Teodor 🔧 (Feb 18, 2026)
void errorCount; // preserve signature
return 30 * 1000;
}
// Note: Update tests in markauthprofilefailure.test.ts to expect 30_000 for all error counts
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/auth-profiles/usage.ts
Line: 278-285

Comment:
tests in `src/agents/auth-profiles.markauthprofilefailure.test.ts:201-209` expect old exponential behavior and will fail

```suggestion
export function calculateAuthProfileCooldownMs(errorCount: number): number {
  // Kai fork: flat 30-second cooldown replaces exponential backoff.
  // Original formula: 60s * 5^(errors-1), capped at 1 hour.
  // At 3 errors = 25 min, 4 errors = 60 min — way beyond Anthropic's actual limits.
  // Author: Teodor 🔧 (Feb 18, 2026)
  void errorCount; // preserve signature
  return 30 * 1000;
}

// Note: Update tests in markauthprofilefailure.test.ts to expect 30_000 for all error counts
```

How can I resolve this? If you propose a fix, please make it concise.

@greptile-apps

greptile-apps Bot commented Mar 2, 2026

Copy link
Copy Markdown
Contributor
Additional Comments (1)

src/agents/auth-profiles/usage.ts
update comment to reflect flat 30s cooldown (no longer exponential)

/**
 * Mark a profile as failed/rate-limited. Applies flat 30-second cooldown.
 * Uses store lock to avoid overwriting concurrent usage updates.
 */
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/auth-profiles/usage.ts
Line: 497-500

Comment:
update comment to reflect flat 30s cooldown (no longer exponential)

```suggestion
/**
 * Mark a profile as failed/rate-limited. Applies flat 30-second cooldown.
 * Uses store lock to avoid overwriting concurrent usage updates.
 */
```

How can I resolve this? If you propose a fix, please make it concise.

@teodorangel
teodorangel force-pushed the fix/auth-cooldown-flat branch 3 times, most recently from 15817ae to c59e44b Compare March 2, 2026 20:43

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c59e44bdaf

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/auth-profiles/usage.ts Outdated
// rate-limit windows (Anthropic ~60s). The exponential curve makes profiles
// unusable long after the provider has cleared the limit.
void errorCount; // preserve signature
return 30 * 1000;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep non-rate-limit failures on a longer cooldown

Returning a hard-coded 30s here applies the same retry cadence to persistent failures, not just throttling: classifyFailoverReason emits values like session_expired/model_not_found (src/agents/pi-embedded-helpers/errors.ts:916-921), and maybeMarkAuthProfileFailure records every non-timeout reason (src/agents/pi-embedded-runner/run.ts:663-670). With this change, those misconfigured or expired profiles re-enter rotation every 30 seconds indefinitely, causing repeated failed attempts and log churn in production where the previous backoff reduced retry pressure over time.

Useful? React with 👍 / 👎.

Original formula punished errors way beyond Anthropic's actual rate limits:
  1 error = 1 min, 2 = 5 min, 3 = 25 min, 4 = 60 min (capped)
Now: always 30 seconds regardless of error count.

Author: Teodor 🔧
Original: Patch 5 (dist, Feb 18, 2026)
Ported to source: Mar 2, 2026
@teodorangel
teodorangel force-pushed the fix/auth-cooldown-flat branch from c59e44b to a7b64d1 Compare March 3, 2026 02:15

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a7b64d1e64

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/auth-profiles/usage.ts Outdated
// rate-limit windows (Anthropic ~60s). The exponential curve makes profiles
// unusable long after the provider has cleared the limit.
void errorCount; // preserve signature
return 30 * 1000;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Align flat cooldown with documented 60s rate-limit window

Returning a hard-coded 30_000 here causes retries to happen before the rate-limit window you describe in the same function comments as "Anthropic ~60s" has elapsed, so single-profile setups will incur at least one avoidable extra failed request after each throttle event. Because markAuthProfileFailure reapplies this same cooldown on every rate_limit failure, sustained throttling can devolve into a 30-second fail loop with unnecessary API/log churn; this should use a window that is at least as long as the documented provider limit or be provider-specific.

Useful? React with 👍 / 👎.

Kai added 2 commits March 4, 2026 01:34
- Replace void errorCount with _errorCount parameter convention
- Use 30_000 literal instead of 30 * 1000
- Restore kilocode cooldown bypass (still a valid provider)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d26bcf1204

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

// At 3 errors = 25 min, 4 errors = 60 min — far exceeds real provider
// rate-limit windows (Anthropic ~60s). The exponential curve makes profiles
// unusable long after the provider has cleared the limit.
return 30_000;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restore stronger backoff for persistent profile failures

calculateAuthProfileCooldownMs now hard-codes 30_000 for every failure, but computeNextProfileUsageStats applies this path to all non-billing/non-auth_permanent reasons, not just throttling (src/agents/auth-profiles/usage.ts, computeNextProfileUsageStats). In the runner, markAuthProfileFailure is recorded for every non-timeout failover (src/agents/pi-embedded-runner/run.ts:656-670), so persistent misconfigurations like auth, model_not_found, or format will re-enter rotation every 30 seconds indefinitely, causing repeated upstream failures/log churn where the previous escalating backoff reduced retry pressure.

Useful? React with 👍 / 👎.

@Takhoffman
Takhoffman requested a review from a team as a code owner March 24, 2026 20:16
altaywtf pushed a commit to kiranvk-2011/openclaw that referenced this pull request Mar 25, 2026
…mit message

Combines ideas from PRs openclaw#45113, openclaw#31962, and openclaw#45763 to address three
cooldown-related issues:

1. Stepped cooldown (30s → 1m → 5m cap) replaces the aggressive
   exponential formula (1m → 5m → 25m → 1h) that locked out providers
   for far longer than the actual API rate-limit window.

2. Per-model cooldown scoping: rate_limit cooldowns now record which
   model triggered them. When a different model on the same auth profile
   is requested, the cooldown is bypassed — so one model hitting a 429
   no longer blocks all other models on the same provider.

3. FallbackSummaryError with soonest-expiry countdown: when all
   candidates are exhausted, the user sees a clear message like
   '⚠️ Rate-limited — ready in ~28s' instead of a generic failure.

Files changed:
- types.ts: add cooldownReason/cooldownModel to ProfileUsageStats
- usage.ts: stepped formula, model-aware isProfileInCooldown, modelId
  threading through computeNextProfileUsageStats/markAuthProfileFailure
- model-fallback.ts: FallbackSummaryError class, model-aware availability
  check, soonestCooldownExpiry computation
- pi-embedded-runner/run.ts: thread modelId into failure recording
- agent-runner-execution.ts: buildCopilotCooldownMessage helper, rate-limit
  detection branch in error handler
- usage.test.ts: update expected cooldown value (60s → 30s)
@altaywtf

Copy link
Copy Markdown
Member

Closing — this was addressed via a different approach.

What shipped instead:

Why this PR is no longer needed:

  • The landed fix addressed the same cooldown-escalation problem area, but chose a stepped ladder (30s -> 1m -> 5m) together with per-model cooldown scoping and user-facing fallback messaging.
  • That broader fix now covers the recovery path we want on main, so keeping this flat-30s variant open would just leave two competing solutions for the same logic.

Thank you for the PR, @teodorangel — the cooldown escalation problem you called out was real and helped inform the fix that landed.

@altaywtf altaywtf closed this Mar 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling size: XS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants