fix: per-model cooldown scope, stepped backoff, and user-facing rate-limit message by kiranvk-2011 · Pull Request #49834 · openclaw/openclaw

kiranvk-2011 · 2026-03-18T13:51:54Z

Summary

This PR addresses three related cooldown issues that cause disproportionate service disruption when a single model on a shared auth profile (e.g. GitHub Copilot) hits a rate limit:

Stepped cooldown formula — replaces the exponential 1m -> 5m -> 25m -> 1h escalation with a capped 30s -> 1m -> 5m ladder that better matches actual API rate-limit windows
Per-model cooldown scoping — rate-limit cooldowns now record which model triggered them; other models on the same auth profile are allowed through
User-facing rate-limit message — a structured FallbackSummaryError with countdown replaces the generic "Agent failed" text

Combines ideas from #45113 (per-model cooldown metadata), #31962 (flat/stepped backoff), and #45763 (structured fallback error with UX).

Problem

When a single GitHub Copilot model (e.g. gpt-4.1) returns HTTP 429, the current code:

Puts the entire auth profile into cooldown — blocking all other models (claude-sonnet-4.6, gpt-4.1-mini, etc.) on the same profile
Escalates aggressively: 3 consecutive errors -> 25-minute cooldown, 4+ -> 1 hour
Shows a generic "Agent failed before reply" message with no actionable information

In multi-model deployments with fallback chains, this creates cascading failures where one model's rate limit locks out the entire provider for up to an hour.

Changes

`src/agents/auth-profiles/types.ts`

Add cooldownReason and cooldownModel fields to ProfileUsageStats

`src/agents/auth-profiles/usage.ts`

calculateAuthProfileCooldownMs() — stepped formula: 30s -> 1m -> 5m (cap)
isProfileInCooldown() — new forModel parameter; bypasses cooldown when the recorded cooldownModel differs from the requested model (rate_limit only — billing/auth failures remain profile-wide)
computeNextProfileUsageStats() — records cooldownReason and cooldownModel metadata; preserves existing metadata during active cooldown windows
markAuthProfileFailure() — accepts and threads modelId parameter
clearExpiredCooldowns() / resetUsageStats() — clear the new metadata fields

`src/agents/model-fallback.ts`

FallbackSummaryError — structured error class with attempts[] and soonestCooldownExpiry timestamp
throwFallbackFailureSummary() — computes soonest expiry across all candidate profiles and throws FallbackSummaryError
runWithModelFallback() — passes candidate.model into isProfileInCooldown() for model-aware availability check

`src/agents/pi-embedded-runner/run.ts`

Thread modelId through maybeMarkAuthProfileFailure and its two call sites
Pass modelId to all isProfileInCooldown() calls in the inner profile loop so model-scoped cooldowns are respected consistently

`src/auto-reply/reply/agent-runner-execution.ts`

buildCopilotCooldownMessage() — produces messages like "Rate-limited — ready in ~28s"
Rate-limit detection branch added to the error handler (before the generic fallback)

`src/agents/auth-profiles/usage.test.ts`

Updated expected cooldown value from 60_000 to 30_000 to match the new stepped formula
8 new per-model cooldown tests covering model-scoped bypass, profile-wide billing fallback, scope widening, and expiry calculation

Cooldown Behavior After This PR

Error count	Cooldown	Scope
1st rate_limit	30 seconds	Model-scoped
2nd rate_limit	1 minute	Model-scoped
3rd+ rate_limit	5 minutes	Model-scoped
billing / auth_permanent	Existing behavior	Profile-wide

Other models on the same auth profile remain fully available during any model-scoped rate-limit cooldown.

Testing

All 45 usage.test.ts tests pass (1 expectation updated for new formula, 8 new per-model tests)
All 59 passing model-fallback.test.ts tests pass (1 pre-existing ANSI escape failure, unrelated)
All 149 pi-embedded-runner/run tests pass
All 26 agent-runner-execution tests pass
oxlint --type-aware passes with 0 errors
oxfmt formatting passes
TypeScript compilation clean (no errors in modified files)

CI Note

Several CI jobs (contracts, channels, extensions, extension-fast, install-smoke) fail on this PR — these are pre-existing failures that also occur on main. See #49848 for full analysis. All CI jobs that cover our changed files (format:check, build-smoke, check, node test 1/2, node test 2/2, changed-scope, protocol:check, secrets) pass.

Related Issues / PRs

fix(auth): preserve per-model cooldown windows #45113 — per-model cooldown metadata
fix(auth): flat 30s cooldown replaces exponential backoff #31962 — replace exponential backoff
feat(agents): structured FallbackSummaryError with human-friendly rate limit messages #45763 — structured FallbackSummaryError
Rate-limited profiles cause immediate failure instead of waiting for cooldown expiry #24158 — aggressive cooldown escalation
[Bug]: after updating to 2026.2.22-2 all models break #24839 — providers enter cooldown without real rate limits
bug: Multiple CI jobs fail on main branch — contracts, channels, extensions, extension-fast, install-smoke, Windows shards #49848 — CI failures affecting all PRs (pre-existing, unrelated to this change)

greptile-apps · 2026-03-18T13:55:49Z

Greptile Summary

This PR addresses three related cooldown pain points in the auth-profile system: a stepped backoff formula (30 s → 1 min → 5 min), per-model cooldown scoping so a single 429 no longer blocks every model on a shared profile, and a structured FallbackSummaryError with a user-facing countdown. The approach is well-motivated and the implementation integrates cleanly with the existing auth-profile machinery.

Key changes reviewed:

calculateAuthProfileCooldownMs now uses a simple three-tier ladder instead of the previous 5^n exponential — tests updated accordingly.
isProfileInCooldown gains a forModel parameter with a guard that prevents the model-bypass from short-circuiting an active disabledUntil (billing/auth) — this correctly addresses the previous reviewer concern.
computeNextProfileUsageStats records cooldownReason/cooldownModel and widens the scope to profile-wide when a second model fails or when the reason is not rate_limit.
FallbackSummaryError / isPureTransientRateLimitSummary ensure the countdown UX fires only when all attempts are concretely rate_limit or overloaded — mixed-cause exhaustion correctly falls through to the generic error message.
Minor items flagged: buildCopilotCooldownMessage calls Date.now() twice (could produce a "~0s" countdown at the boundary); resolveFallbackSoonestCooldownExpiry lacks a try/catch around its synchronous file I/O; and an edge case in computeNextProfileUsageStats where an absent modelId during an active rate-limit window would preserve the existing model scope rather than widening it.

Confidence Score: 4/5

Safe to merge — all three stated goals are correctly implemented and well-tested; the remaining comments are minor style/robustness observations that do not affect correctness under normal operating conditions.
Core logic is sound: the model-scoped bypass, disabledUntil guard, and mixed-cause FallbackSummaryError filtering all work correctly. Previous reviewer issues have been addressed. The three flagged items are cosmetic (double Date.now()) and low-probability reliability concerns (loadAuthProfileStoreForRuntime not guarded, implicit cooldownModel preservation when modelId is absent), none of which affect the primary use-case. Test coverage is comprehensive (45 + 8 new tests).
src/agents/model-fallback.ts (the resolveFallbackSoonestCooldownExpiry I/O path) and src/auto-reply/reply/agent-runner-execution.ts (double Date.now() in buildCopilotCooldownMessage) are worth a second look before merge.

Comments Outside Diff (1)

src/agents/model-fallback.ts, line 546-575 (link)

resolveFallbackSoonestCooldownExpiry can throw, silently bypassing FallbackSummaryError

loadAuthProfileStoreForRuntime does synchronous file I/O. If the store file is absent, malformed, or temporarily locked by another process, it can throw. Because resolveFallbackSoonestCooldownExpiry is called inline as a parameter to throwFallbackFailureSummary, any such exception propagates out of runWithModelFallback before FallbackSummaryError is thrown. The downstream handler in agent-runner-execution.ts would then receive an unexpected I/O error: isFallbackSummaryError returns false, isRateLimitErrorMessage also returns false, and the user sees a raw filesystem error message in the "Agent failed before reply" text.

Wrapping the call in a try/catch (returning null on failure) makes the fallback behaviour gracefully degrade to the generic rate-limit message without a countdown rather than surfacing an unrelated I/O error:

function resolveFallbackSoonestCooldownExpiry(params: { ... }): number | null {
  if (!params.authStore) {
    return null;
  }
  try {
    const refreshedStore = loadAuthProfileStoreForRuntime(params.agentDir, {
      readOnly: true,
      allowKeychainPrompt: false,
    });
    // ... existing logic
    return getSoonestCooldownExpiry(refreshedStore, [...allProfileIds]);
  } catch {
    return null;
  }
}

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/agents/model-fallback.ts
Line: 546-575

Comment:
**`resolveFallbackSoonestCooldownExpiry` can throw, silently bypassing `FallbackSummaryError`**

`loadAuthProfileStoreForRuntime` does synchronous file I/O. If the store file is absent, malformed, or temporarily locked by another process, it can throw. Because `resolveFallbackSoonestCooldownExpiry` is called inline as a parameter to `throwFallbackFailureSummary`, any such exception propagates out of `runWithModelFallback` *before* `FallbackSummaryError` is thrown. The downstream handler in `agent-runner-execution.ts` would then receive an unexpected I/O error: `isFallbackSummaryError` returns `false`, `isRateLimitErrorMessage` also returns `false`, and the user sees a raw filesystem error message in the "Agent failed before reply" text.

Wrapping the call in a try/catch (returning `null` on failure) makes the fallback behaviour gracefully degrade to the generic rate-limit message without a countdown rather than surfacing an unrelated I/O error:

```ts
function resolveFallbackSoonestCooldownExpiry(params: { ... }): number | null {
  if (!params.authStore) {
    return null;
  }
  try {
    const refreshedStore = loadAuthProfileStoreForRuntime(params.agentDir, {
      readOnly: true,
      allowKeychainPrompt: false,
    });
    // ... existing logic
    return getSoonestCooldownExpiry(refreshedStore, [...allProfileIds]);
  } catch {
    return null;
  }
}
```

How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

This is a comment left during a code review.
Path: src/auto-reply/reply/agent-runner-execution.ts
Line: 87-93

Comment:
**Two `Date.now()` calls leave a window for `secsLeft` to be zero or negative**

`expiry > Date.now()` and `expiry - Date.now()` are evaluated at separate instants. Under normal conditions the gap is negligible, but under a very loaded event loop the second call can return a value past `expiry`. `Math.ceil` rounds small negative fractions to `0`, so the user would see "ready in ~0s" or, in extreme cases, a negative countdown like "ready in ~-1s".

Capture a single timestamp at the top of the function and reuse it:

```suggestion
  const expiry = err.soonestCooldownExpiry;
  const now = Date.now();
  if (typeof expiry === "number" && expiry > now) {
    const secsLeft = Math.ceil((expiry - now) / 1000);
    if (secsLeft <= 60) {
      return `⚠️ Rate-limited — ready in ~${secsLeft}s. Please wait a moment.`;
    }
    const minsLeft = Math.ceil(secsLeft / 60);
    return `⚠️ Rate-limited — ready in ~${minsLeft} min. Please try again shortly.`;
  }
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/agents/model-fallback.ts
Line: 546-575

Comment:
**`resolveFallbackSoonestCooldownExpiry` can throw, silently bypassing `FallbackSummaryError`**

`loadAuthProfileStoreForRuntime` does synchronous file I/O. If the store file is absent, malformed, or temporarily locked by another process, it can throw. Because `resolveFallbackSoonestCooldownExpiry` is called inline as a parameter to `throwFallbackFailureSummary`, any such exception propagates out of `runWithModelFallback` *before* `FallbackSummaryError` is thrown. The downstream handler in `agent-runner-execution.ts` would then receive an unexpected I/O error: `isFallbackSummaryError` returns `false`, `isRateLimitErrorMessage` also returns `false`, and the user sees a raw filesystem error message in the "Agent failed before reply" text.

Wrapping the call in a try/catch (returning `null` on failure) makes the fallback behaviour gracefully degrade to the generic rate-limit message without a countdown rather than surfacing an unrelated I/O error:

```ts
function resolveFallbackSoonestCooldownExpiry(params: { ... }): number | null {
  if (!params.authStore) {
    return null;
  }
  try {
    const refreshedStore = loadAuthProfileStoreForRuntime(params.agentDir, {
      readOnly: true,
      allowKeychainPrompt: false,
    });
    // ... existing logic
    return getSoonestCooldownExpiry(refreshedStore, [...allProfileIds]);
  } catch {
    return null;
  }
}
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/agents/auth-profiles/usage.ts
Line: 508-519

Comment:
**`cooldownModel` preserved when `modelId` is absent on a `rate_limit` during an active window**

In the `existingCooldownActive` branch the three conditions are:
1. Different model → widen to `undefined`
2. Non-rate-limit reason → widen to `undefined`
3. Else → keep `params.existing.cooldownModel`

Condition 3 is reached when `params.reason === "rate_limit"` **and** either `params.modelId` or `params.existing.cooldownModel` is falsy. When `params.modelId` is `undefined` (i.e. the caller didn't pass a model) and `params.existing.cooldownModel` is already set to `"model-A"`, the existing model-scoped value is silently preserved — even though the new failure came from an unknown model and should arguably widen the scope.

All current call sites in this PR correctly thread `modelId`, so this is not a live bug today. But adding an explicit guard prevents a silent regression if a future call-site omits `modelId`:

```ts
} else if (params.reason === "rate_limit" && !params.modelId && params.existing.cooldownModel) {
  // Unknown originating model — conservatively widen scope so no model bypasses.
  updatedStats.cooldownModel = undefined;
} else {
  updatedStats.cooldownModel = params.existing.cooldownModel;
}
```

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (3): Last reviewed commit: "fix(agents): scope cooldowns per model" | Re-trigger Greptile}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2b5c264330

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 106d513919

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9451a27678

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 36ac8951f0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-03-18T15:11:35Z

+          allProfileIds.add(id);
+        }
+      }
+      return getSoonestCooldownExpiry(authStore, [...allProfileIds]);


Recompute cooldown expiry from fresh store before summary

soonestCooldownExpiry is computed from the authStore snapshot captured before the fallback attempts run. In the embedded runner path, attempts can persist new cooldowns via markAuthProfileFailure, but this snapshot is never refreshed, so the summary countdown can be null/stale (for example after the first in-run 429), producing incorrect retry timing in the user-facing rate-limit message.

Useful? React with 👍 / 👎.

Acknowledged — this is a valid observation about the snapshot staleness.

However, this is existing architecture, not introduced by this PR. The authStore in model-fallback.ts is a clone created via ensureAuthProfileStore() → structuredClone() at the start of runWithModelFallback (line 555). The embedded runner in run.ts creates its own separate clone (line 423). Mutations from markAuthProfileFailure during attempts update the runner's clone and persist to disk via updateAuthProfileStoreWithLock, but the model-fallback.ts clone is never refreshed.

Impact is cosmetic only: The soonestCooldownExpiry computed at line 818 feeds into the user-facing error message countdown text (e.g. "retry in ~4m 30s"). Cooldown enforcement works correctly because each embedded runner call reads fresh state from disk via the lock-guarded store update. A stale countdown in the error message is a minor UX imperfection, not a correctness issue.

Fixing this properly would require refactoring the structuredClone snapshot pattern to either re-read the store after the attempt loop or use a shared mutable reference — both are broader architectural changes beyond this PR's scope. Happy to tackle that in a follow-up if maintainers prefer.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 971693935a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

altaywtf · 2026-03-18T18:18:28Z

I found one concrete merge blocker and one follow-up nit.

The blocker is that this branch changes calculateAuthProfileCooldownMs() to the stepped 30s -> 1m -> 5m ladder, but src/agents/auth-profiles.markauthprofilefailure.test.ts still asserts the old exponential contract. I verified it on the PR head with:

pnpm test -- src/agents/auth-profiles.markauthprofilefailure.test.ts

That fails at src/agents/auth-profiles.markauthprofilefailure.test.ts:271 with expected 30000 to be 60000, so the production change and targeted auth-profile test suite are currently out of sync.

kiranvk-2011 · 2026-03-18T18:45:01Z

Thanks for catching this, @altaywtf! You're right — the markauthprofilefailure test suite was still asserting the old exponential 1m → 5m → 25m → 1h contract.

Fixed in b2ccb3d9:

calculateAuthProfileCooldownMs assertions updated to the new stepped ladder: 30s → 1m → 5m (cap)
The "resets error count when previous cooldown has expired" test's upper bound tightened from 120_000 to 60_000 to match calculateAuthProfileCooldownMs(1) = 30_000
Comments updated throughout

Both test suites now pass:

pnpm test -- src/agents/auth-profiles.markauthprofilefailure.test.ts  # 9/9 passed
pnpm test -- src/agents/auth-profiles/usage.test.ts                  # 47/47 passed

You also mentioned a follow-up nit — happy to address that if you could share the details!

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 487deab3a5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

kiranvk-2011 · 2026-03-18T20:14:20Z

CI Failure Analysis — All Failures Pre-existing / Unrelated

I've analyzed all 4 failing CI jobs on the latest run (23264393916). None involve files modified by this PR (auth-profiles/usage.ts, auth-profiles/types.ts, model-fallback.ts, pi-embedded-runner/run.ts, agent-runner-execution.ts, or their test files).

Job	Failure	Related to PR?
`checks (node, test, 1, 2)`	Tlon submodule TypeScript errors (TS1360/TS2339/TS2307 in `channelContentConfig.ts`, `postContent.ts`, `debug.ts`, `groupTemplates.ts`)	❌
`checks (node, extensions)`	`llm-task-tool.test.ts` — 2 schema validation test failures	❌
`checks-windows (node, test, 5, 6)`	`windows-acl.test.ts` — 48/48 Windows ACL tests failed	❌
`checks-windows (node, test, 3, 6)`	`bundle-mcp.test.ts` — 2 path shortname mismatches (`RUNNER~1` vs `runneradmin`)	❌

For reference, the latest upstream main CI runs (23264812040, 23264809135) are all concluding success. The failures on this branch may be due to submodule version drift between our rebase point and latest main, or transient CI runner environment issues.

All jobs that exercise code paths touched by this PR are passing: check (lint), checks (node, test, 2, 2), checks (node, contracts), checks (bun, test), build-smoke, install-smoke, and checks-windows (node, test, 4, 6).

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bd401f7876

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

kiranvk-2011 · 2026-03-19T11:27:20Z

CI Failure Analysis — All Pre-Existing on `main`

Rebased onto latest green main (c4a4050ce4, main CI run #23293514090). All CI failures on this PR are identical to failures on main itself — verified by comparing against main CI run #23287263443 on 009a10bce2 (the most recent main commit that ran the full test suite).

Failure Breakdown

Job	Root Cause	Our PR touches this?
`check`	TS errors in `extensions/matrix/src/onboarding.ts`, `acp-spawn.test.ts`, `commands/channels/remove.ts`, `matrix-plugin-helper.test.ts`	No — all in Matrix extension / channel commands
`contracts`	`registry.contract.test.ts` expects `['discord','feishu','matrix','telegram']` but gets `['discord','feishu','telegram']` (matrix missing)	No — channel contract registry
`channels`	Discord: `createAccountListHelpers is not a function`, `resolveThreadBindingIdleTimeoutMs is not a function`; WhatsApp: `updateLastRoute` setter mock failure	No — Discord/WhatsApp extension tests
`install-smoke`	`Cannot find module '@vector-im/matrix-bot-sdk/package.json'`	No — Matrix extension install (also fails on main `c4a4050ce4`)
`windows shards`	Same contracts + channel test failures as above	No

Our PR's failures are a strict subset of main's

Main (009a10bce2): 11 failing jobs (check, bun-test, channels, contracts, compat-node22, release-check, windows shards 1/3/4/5/6)
Our PR (6d58d2f381): 7 failing jobs (check, channels, contracts, windows shards 1/4/5/6) — all present in main's failure list
Our PR passes jobs that main fails: bun test, compat-node22, release-check

Timeline

main was green at 0443ee82be (run #23284150176)
f3097b4c09 (refactor: install optional channels for remove) introduced these test failures
c4a4050ce4 (fix(macos): align exec command parity) fixed CI infrastructure but did not touch the failing test files — it passed CI only because changed-scope skipped all test shards (no src/ files changed)
The Install Smoke workflow fails on all recent main commits including c4a4050ce4

This PR's scope

Our changes are limited to:

src/agents/auth-profiles/ (types, usage logic, tests)
src/agents/model-fallback.ts
src/agents/pi-embedded-runner/run.ts
src/auto-reply/reply/agent-runner-execution.ts
changelog/fragments/cooldown-per-model-stepped-backoff.md

None of the failing tests are in or related to these files. The test shard 2/2, extensions, protocol, build-smoke, bun test, and all boundary checks pass.

kiranvk-2011 · 2026-03-19T11:46:53Z

Production Validation — 4 Days of A/B Comparison

I've been running this PR's code in a Docker deployment alongside production (stock main) since Mar 16. Both instances share the same GitHub Copilot token (same rate-limit pool) and serve real Telegram users. Here are the results:

Copilot Failure Rates

Date	Production (main)	PR #49834
2026-03-16	0% (6 calls)	92.8% (69 calls)¹
2026-03-17	100% (100 calls)	27.9% (43 calls)
2026-03-18	100% (16 calls)	42.1% (19 calls)
2026-03-19	100% (4 calls)	9.5% (42 calls)

¹ Mar 16 was the PR instance's first day — high failure rate due to initial cooldown cascade before the stepped backoff had a chance to stabilize.

What the Numbers Show

Production (main) enters a provider-wide cooldown cascade on the first 429 and stays locked out for 25-60 minutes (escalation formula: min(1h, 1m × 5^(n-1))). Once in this state, every subsequent request fails, each failure compounds the error counter, and the provider stays in cooldown until external intervention (watchdog restart). Result: 100% Copilot failure rate for 3 consecutive days.

PR instance (this code) uses per-model cooldown scoping + stepped backoff (30s → 1m → 5m cap). Gateway logs show the error counter consistently resetting between rate-limit windows:

2026-03-18T20:44:53 | errors: 0→1 | cooldown: 30s | reason: rate_limit
2026-03-18T20:45:41 | errors: 0→1 | cooldown: 30s | reason: rate_limit
2026-03-18T21:09:08 | errors: 0→1 | cooldown: 30s | reason: rate_limit
2026-03-19T06:52:56 | errors: 0→1 | cooldown: 30s | reason: rate_limit

Every rate-limit event starts fresh at errorCount: 0→1 with a 30-second cooldown — the cascade never compounds. After 30 seconds the model is retried and usually succeeds.

Model Isolation Working

A 429 on claude-haiku-4.5 (cron job) does not block claude-opus-4.6 (user request). The cooldownModel field ensures cooldown is scoped to the specific model that was rate-limited, not the entire provider. This is the core fix for issue #24158.

External Watchdog Interventions

Instance	Cascade alerts (4 days)	Proactive cooldown cleanups
Production	0 full cascades	4 today (stale cooldowns on disk)
PR #49834	2 cascade alerts	Normal — cooldowns created and expired naturally

Production's "0 cascade alerts" is misleading — the watchdog doesn't trigger because fallback providers absorb the failures. But Copilot is locked out 100% of the time, routing all traffic to fallback providers unnecessarily. The PR instance's 2 early cascade alerts (Mar 17-18) were during initial tuning; by Mar 19 the system is self-correcting with zero external intervention needed.

CI Note

All 6 failing CI jobs on this PR are pre-existing failures on main itself — verified by comparing against main CI run #23287263443 at the same commit (009a10bce2). The test shards containing our changes (test 1/2, test 2/2, bun) all pass ✅. See detailed CI analysis comment above.

altaywtf · 2026-03-24T18:11:53Z

@greptileai review
@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fd0e16c08a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@altaywtf

…limit message (#49834) Merged via squash. Prepared head SHA: 7c488c0 Co-authored-by: kiranvk-2011 <[email protected]> Co-authored-by: altaywtf <[email protected]> Reviewed-by: @altaywtf

@altaywtf

…limit message (openclaw#49834) Merged via squash. Prepared head SHA: 7c488c0 Co-authored-by: kiranvk-2011 <[email protected]> Co-authored-by: altaywtf <[email protected]> Reviewed-by: @altaywtf

* test: collapse msteams helper suites * test: collapse msteams graph suites * test: collapse msteams state and monitor suites * fix: preserve Telegram forum topic last-route delivery (#53052) (thanks @VACInc) * fix(telegram): preserve forum topic thread in last-route delivery * style(telegram): format last-route update * test(telegram): cover General topic last-route thread * test(telegram): align topic route helper * fix(telegram): skip bound-topic last-route writes --------- Co-authored-by: VACInc <[email protected]> Co-authored-by: Ayaan Zaidi <[email protected]> * fix(feishu): default requireMention to false for groupPolicy open Groups configured with groupPolicy: open are expected to respond to all messages. Previously, requireMention defaulted to true regardless of groupPolicy, causing image (and other non-text) messages to be silently dropped because they cannot carry @-mentions. Fix: when groupPolicy is 'open' and requireMention is not explicitly configured, resolve it to false instead of true. Users who want mention-required behaviour in open groups can still set requireMention: true explicitly. Adds three regression tests covering the new default, explicit override, and the unchanged allowlist-policy behaviour. Closes #52553 * test(feishu): update config-schema test for removed requireMention default * fix: finish feishu open-group docs and baselines (#54058) (thanks @byungsker) * test: refresh pairing reply assertions for fenced codes (#54058) (thanks @byungsker) * test: accept fenced discord pairing codes (#54058) (thanks @byungsker) * test: allow daemon start hints to grow on linux (#54058) (thanks @byungsker) * fix: preflight invalid telegram photos (#52545) (thanks @hnshah) * fix(telegram): validate photo dimensions before sendPhoto Prevents PHOTO_INVALID_DIMENSIONS errors by checking image dimensions against Telegram Bot API requirements before calling sendPhoto. If dimensions exceed limits (width + height > 10,000px), automatically falls back to sending as document instead of crashing with 400 error. Tested in production (openclaw 2026.3.13) where this error occurred: [telegram] tool reply failed: GrammyError: Call to 'sendPhoto' failed! (400: Bad Request: PHOTO_INVALID_DIMENSIONS) Uses existing sharp dependency to read image metadata. Gracefully degrades if sharp fails (lets Telegram handle validation, backward compatible behavior). Closes: #XXXXX (will reference OpenClaw issue if one exists) * fix(telegram): validate photo aspect ratio * refactor: use shared telegram image metadata * fix: fail closed on telegram image metadata * fix: preflight invalid telegram photos (#52545) (thanks @hnshah) --------- Co-authored-by: Bob Shah <[email protected]> Co-authored-by: Ayaan Zaidi <[email protected]> * fix: clarify cron best-effort partial delivery status (#42535) (thanks @MoerAI) * fix(cron): track and log bestEffort delivery failures, mark not delivered on partial failure * fix(cron): cache successful results on partial failure to preserve replay idempotency When a best-effort send partially fails, we now still cache the successful delivery results via rememberCompletedDirectCronDelivery. This prevents duplicate sends on same-process replay while still correctly marking the job as not fully delivered. * fix(cron): preserve partial-failure state on replay (#27069) * fix(cron): restore test infrastructure and fix formatting * fix: clarify cron best-effort partial delivery status (#42535) (thanks @MoerAI) --------- Co-authored-by: Ayaan Zaidi <[email protected]> * fix(cli): route telegram thread create to topic-create * fix: make telegram thread create use topic payload (#54336) (thanks @andyliu) * cron: queue isolated delivery awareness * fix(browser): add Edge LaunchServices bundle IDs for macOS default browser detection macOS registers Edge as 'com.microsoft.edgemac' in LaunchServices, which differs from the CFBundleIdentifier 'com.microsoft.Edge' in the app's own Info.plist. Without recognising the LaunchServices IDs, Edge users who set Edge as their default browser are not detected as having a Chromium browser. Add the four com.microsoft.edgemac* variants to CHROMIUM_BUNDLE_IDS and a corresponding test that mocks the LaunchServices → osascript resolution path for Edge. * fix: add changelog for macOS Edge default browser detection (#48561) (thanks @zoherghadyali) * fix: cover macOS Edge osascript fallback path (#48561) (thanks @zoherghadyali) * fix: fail loud when PTY cursor mode is unknown (#51490) (thanks @liuy) * fix(process): auto-detect PTY cursor key mode for send-keys When a PTY session sends smkx (\x1b[?1h) or rmkx (\x1b[?1l) to switch cursor key mode, send-keys now detects this and encodes cursor keys accordingly. - smkx/rmkx detection in handleStdout before sanitizeBinaryOutput - cursorKeyMode stored in ProcessSession - encodeKeySequence accepts cursorKeyMode parameter - DECCKM_SS3_KEYS for application mode (arrows + home/end) - CSI sequences for normal mode - Modified keys (including alt) always use xterm modifier scheme - Extract detectCursorKeyMode for unit testing - Use lastIndexOf to find last toggle in chunk (later one wins) Fixes #51488 * fix: fail loud when PTY cursor mode is unknown (#51490) (thanks @liuy) * style: format process send-keys guard (#51490) (thanks @liuy) --------- Co-authored-by: Ayaan Zaidi <[email protected]> * fix: isolate session:patch hook payload (#53880) (thanks @graciegould) * gateway: make session:patch hook typed and non-blocking * gateway(test): add session:patch hook coverage * docs(gateway): clarify session:patch security note * fix: address review feedback on session:patch hook Remove unused createInternalHookEvent import and fix doc example to use inline event.type check matching existing hook examples. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: isolate hook payload to prevent mutation leaking into response Shallow-copy sessionEntry and patch in the session:patch hook event so fire-and-forget handlers cannot mutate objects used by the response path. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: isolate session:patch hook payload (#53880) (thanks @graciegould) --------- Co-authored-by: “graciegould” <“[email protected]”> Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: Ayaan Zaidi <[email protected]> * fix: unify log timestamp offsets (#38904) (thanks @sahilsatralkar) * fix: skip session:patch hook clone without listeners * fix: preserve before_dispatch delivery semantics (#50444) (thanks @gfzhx) * Plugins: add before_dispatch hook * Tests: fix before_dispatch hook mock typing * Rebase: adapt before_dispatch hook to routeReplyRuntime refactor * fix: preserve before_dispatch delivery semantics (#50444) (thanks @gfzhx) --------- Co-authored-by: Ayaan Zaidi <[email protected]> * fix(whatsapp): avoid eager login tool runtime access * fix: normalize before_dispatch conversation id * test: harden parallels npm update runner * docs: sort changelog by user impact * test: collapse telegram helper suites * test: collapse telegram context and transport suites * docs: add missing changelog items * refactor: align pairing replies, daemon hints, and feishu mention policy * test: collapse telegram button and access suites * test: collapse telegram transport and status suites * fix(whatsapp): unwrap FutureProofMessage (botInvokeMessage) to restore reply-to-bot detection * fix(whatsapp): compare selfLid for reply-to-bot implicit mention in groups * fix(whatsapp): read selfLid from creds.json for reply-to-bot detection * fix(whatsapp): use async fs.promises.readFile for selfLid creds read * refactor(auth): separate profile ids from email metadata * refactor(openai): extract codex auth identity helper * test: isolate voice-call temp stores * build: prepare 2026.3.24-beta.1 * fix: copy openclaw bin before docker install * refactor: unify whatsapp identity handling * test: fix clobbered config snapshot expectation * fix(config): ignore same-base correction publish warnings * test: add Open WebUI docker smoke * fix(media): align outbound media access with fs policy * fix(update): preflight npm target node engine * fix(ci): restore e2e docker cache boundary * test(release): sync llama peer fixture * Discord: log rejected native command deploy failures (#54118) Merged via squash. Prepared head SHA: be250f96204ed6dc755c10d2b9640f7dd49bc70c Co-authored-by: huntharo <[email protected]> Co-authored-by: huntharo <[email protected]> Reviewed-by: @huntharo * fix(runtime): support Node 22.14 installs * test(media): make local roots fixture windows-safe * Tests: isolate security audit home skill resolution (#54473) Merged via squash. Prepared head SHA: 82181e15fbfa47244cf166f57f40646bc35629c6 Co-authored-by: huntharo <[email protected]> Co-authored-by: huntharo <[email protected]> Reviewed-by: @huntharo * docs: prepare 2026.3.24-beta.2 release * fix(sandbox): honor sandbox alsoAllow and explicit re-allows (#54492) * fix(sandbox): honor effective sandbox alsoAllow policy * fix(sandbox): prefer resolved sandbox context policy * fix: honor sandbox alsoAllow policy (#54492) (thanks @ngutman) * fix(feishu): close WebSocket connections on monitor stop (#52844) * fix(feishu): close WebSocket connections on monitor stop/abort Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * test(feishu): add WebSocket cleanup tests Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix(feishu): close WebSocket connections on monitor stop (#52844) (thanks @schumilin) --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> Co-authored-by: George Zhang <[email protected]> * fix(feishu): use message create_time for inbound timestamps (#52809) * fix(feishu): use message create_time instead of Date.now() for Timestamp field When a message is sent offline and later retried by the Feishu client upon reconnection, Date.now() captures the *delivery* time rather than the *authoring* time. This causes downstream consumers to see a timestamp that can be minutes or hours after the user actually composed the message, leading to incorrect temporal semantics — for example, a "delete this" command may target the wrong resource because the agent believes the instruction was issued much later than it actually was. Replace every Date.now() used for message timestamps with the original create_time from the Feishu event payload (millisecond-epoch string), falling back to Date.now() only when the field is absent. The definition is also hoisted to the top of handleFeishuMessage so that both the pending-history path and the main inbound-payload path share the same authoritative value. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * test(feishu): verify Timestamp uses message create_time Add two test cases: 1. When create_time is present, Timestamp must equal the parsed value 2. When create_time is absent, Timestamp falls back to Date.now() Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * chore: revert unrelated formatting change to lifecycle.test.ts This file was inadvertently formatted in a prior commit. Reverting to match main and keep the PR scoped to the Feishu timestamp fix only. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix(feishu): use message create_time for inbound timestamps (#52809) (thanks @schumilin) --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> Co-authored-by: George Zhang <[email protected]> * Fix local copied package installs honoring staged project .npmrc (#54543) * fix(release): add plugin-sdk:check-exports to release:check (#54283) * fix(plugins): resolve sdk alias from import.meta.url for external plugins When a plugin is installed outside the openclaw package (e.g. ~/.openclaw/extensions/), resolveLoaderPluginSdkPackageRoot() fails to locate the openclaw root via cwd or argv1 hints, resulting in an empty alias map. Jiti then cannot resolve openclaw/plugin-sdk/* imports and the plugin fails to load with "Cannot find module". Since sdk-alias.ts is always compiled into the openclaw package itself, import.meta.url reliably points inside the installation directory. Add it as an unconditional fallback in resolveLoaderPluginSdkPackageRoot() so external plugins can always resolve the plugin SDK. Fixes: Error: Cannot find module 'openclaw/plugin-sdk/plugin-entry' * fix(plugins): pass loader moduleUrl to resolve sdk alias for external plugins The previous approach of adding import.meta.url as an unconditional fallback inside resolveLoaderPluginSdkPackageRoot() broke test isolation: tests that expected null from untrusted fixtures started finding the real openclaw root. Revert that and instead thread an optional moduleUrl through buildPluginLoaderAliasMap → resolvePluginSdkScopedAliasMap → listPluginSdkExportedSubpaths → resolveLoaderPluginSdkPackageRoot. loader.ts passes its own import.meta.url as the hint, which is always inside the openclaw installation. This guarantees the sdk alias map is built correctly even when argv1 does not resolve to the openclaw root (e.g. single-binary distributions, custom launchers, or Docker images where the binary wrapper is not a standard npm symlink). Tests that call sdk-alias helpers directly without moduleUrl are unaffected and continue to enforce the existing isolation semantics. A new test covers the moduleUrl resolution path explicitly. * fix(plugins): use existing fixture file for moduleUrl hint in test The previous test pointed loaderModuleUrl to dist/plugins/loader.js which is not created by createPluginSdkAliasFixture, causing resolution to fall back to the real openclaw root instead of the fixture root. Use fixture.root/openclaw.mjs (created by the bin+marker fixture) so the moduleUrl hint reliably resolves to the fixture package root. * fix(test): use fixture.root as cwd in external plugin alias test When process.cwd() is mocked to the external plugin dir, the findNearestPluginSdkPackageRoot(process.cwd()) fallback resolves to the real openclaw repo root in the CI test runner, making the test resolve the wrong aliases. Using fixture.root as cwd ensures all resolution paths consistently point to the fixture. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(release): add plugin-sdk:check-exports to release:check plugin-sdk subpath exports (e.g. openclaw/plugin-sdk/plugin-entry, openclaw/plugin-sdk/provider-auth) were missing from the published package.json, causing external plugins to fail at load time with 'Cannot find module openclaw/plugin-sdk/plugin-entry'. Root cause: sync-plugin-sdk-exports.mjs syncs plugin-sdk-entrypoints.json into package.json exports, but this sync was never validated in the release:check pipeline. As a result, any drift between plugin-sdk-entrypoints.json and the published package.json goes undetected until users hit the runtime error. Fix: add plugin-sdk:check-exports to release:check so the CI gate fails loudly if the exports are out of sync before publishing. * fix(test): isolate moduleUrl hint test from process.cwd() fallback Use externalPluginRoot as cwd instead of fixture.root, so only the moduleUrl hint can resolve the openclaw package root. Previously, withCwd(fixture.root) allowed the process.cwd() fallback to also resolve the fixture root, making the moduleUrl path untested. Spotted by greptile-apps review on #54283. * fix(test): use empty string to disable argv1 in moduleUrl hint test Passing undefined for argv1 in buildPluginLoaderAliasMap triggers the STARTUP_ARGV1 default (process.argv[1], the vitest runner binary inside the openclaw repo). resolveTrustedOpenClawRootFromArgvHint then resolves to the real openclaw root before the moduleUrl hint is checked, making the test resolve wrong aliases. Pass "" instead: falsy so the hint is skipped, but does not trigger the default parameter value. Only the moduleUrl can bridge the gap. Made-with: Cursor * fix(plugins): thread moduleUrl through SDK alias resolution for external plugins (#54283) Thanks @xieyongliang --------- Co-authored-by: bojsun <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Jerry <[email protected]> Co-authored-by: yongliang.xie <[email protected]> Co-authored-by: George Zhang <[email protected]> * feat(minimax): add image generation provider and trim model catalog to M2.7 (#54487) * feat(minimax): add image generation and TTS providers, trim TUI model list Register MiniMax image-01 and speech-2.8 models as plugin providers for the image_generate and TTS tools. Both resolve CN/global base URLs from the configured model endpoint origin. - Image generation: base64 response, aspect-ratio support, image-to-image via subject_reference, registered for minimax and minimax-portal - TTS: speech-2.8-turbo (default) and speech-2.8-hd, hex-encoded audio, voice listing via get_voice API, telephony PCM support - Add MiniMax to TTS auto-detection cascade (after ElevenLabs, before Microsoft) and TTS config section - Remove MiniMax-VL-01, M2, M2.1, M2.5 and variants from TUI picker; keep M2.7 and M2.7-highspeed only (backend routing unchanged) * feat(minimax): trim legacy model catalog to M2.7 only Cherry-picked from temp/feat/minimax-trim-legacy-models (949ed28). Removes MiniMax-VL-01, M2, M2.1, M2.5 and variants from the model catalog, model order, modern model matchers, OAuth config, docs, and tests. Keeps only M2.7 and M2.7-highspeed. Conflicts resolved: - provider-catalog.ts: removed MINIMAX_TUI_MODELS filter (no longer needed since source array is now M2.7-only) - index.ts: kept image generation + speech provider registrations (added by this branch), moved media understanding registrations earlier (as intended by the cherry-picked commit) * fix(minimax): update discovery contract test to reflect M2.7-only catalog Cherry-picked from temp/feat/minimax-trim-legacy-models (2c750cb). * feat(minimax): add web search provider and register in plugin entry * fix(minimax): resolve OAuth credentials for TTS speech provider * MiniMax: remove web search and TTS providers * fix(minimax): throw on empty images array after generation failure * feat(minimax): add image generation provider and trim catalog to M2.7 (#54487) (thanks @liyuan97) --------- Co-authored-by: tars90percent <[email protected]> Co-authored-by: George Zhang <[email protected]> * build: prepare 2026.3.24 release * docs: format changelog for release * fix: reconcile session compaction count after late compaction success (#45493) Merged via squash. Prepared head SHA: d0715a5555791dd44a406d4732843454a3e9619e Co-authored-by: jackal092927 <[email protected]> Co-authored-by: jalehman <[email protected]> Reviewed-by: @jalehman * build: update appcast for 2026.3.24 * Telegram: ignore self-authored DM message updates (#54530) Merged via squash. Prepared head SHA: c1c8a851682944c49259afe1741d6d976016b08b Co-authored-by: huntharo <[email protected]> Co-authored-by: huntharo <[email protected]> Reviewed-by: @huntharo * test: keep vitest on forks only * fix: stop leaking reply tags in iMessage outbound text (#39512) (thanks @mvanhorn) * fix: stop leaking reply tags in iMessage outbound text (#39512) (thanks @mvanhorn) * fix: preserve iMessage outbound whitespace without directive tags (#39512) (thanks @mvanhorn) --------- Co-authored-by: Ayaan Zaidi <[email protected]> * fix: mid-turn 429 rate limit silent no-reply and context engine registration failure (#50930) Merged via squash. Prepared head SHA: eea7800df31f30caabfec2aefe8f20008365a8e8 Co-authored-by: infichen <[email protected]> Co-authored-by: jalehman <[email protected]> Reviewed-by: @jalehman * plugin-runtime: expose runHeartbeatOnce in system API (#40299) * plugin-runtime: expose runHeartbeatOnce in system API Plugins that enqueue system events and need the agent to deliver responses to the originating channel currently have no way to override the default `heartbeat.target: "none"` behaviour. Expose `runHeartbeatOnce` in the plugin runtime `system` namespace so plugins can trigger a single heartbeat cycle with an explicit `heartbeat: { target: "last" }` override — the same pattern the cron service already uses (see #28508). Changes: - Add `RunHeartbeatOnceOptions` type and `runHeartbeatOnce` to `PluginRuntimeCore.system` (types-core.ts) - Wire the function through a thin wrapper in runtime-system.ts - Update the test-utils plugin-runtime mock Made-with: Cursor * feat(plugins): expose runHeartbeatOnce in system API (#40299) (thanks @loveyana) --------- Co-authored-by: George Zhang <[email protected]> * fix(compaction): surface safeguard cancel reasons and clarify /compact skips (#51072) Merged via squash. Prepared head SHA: f1dbef044384fbd79ca5ef3616ad37fb03b05fae Co-authored-by: afurm <[email protected]> Co-authored-by: jalehman <[email protected]> Reviewed-by: @jalehman * Fix/telegram writeback admin scope gate (#54561) * fix(telegram): require operator.admin for legacy target writeback persistence * Address claude feedback * Update extensions/telegram/src/target-writeback.ts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Remove stray brace * Add updated docs * Add missing test file, address codex concerns * Fix test formatting error * Address comments, fix tests --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * test: fix windows tmp root assertions * fix: per-model cooldown scope, stepped backoff, and user-facing rate-limit message (#49834) Merged via squash. Prepared head SHA: 7c488c070c0cafb5a4b53c598d8ccd38c418b67c Co-authored-by: kiranvk-2011 <[email protected]> Co-authored-by: altaywtf <[email protected]> Reviewed-by: @altaywtf * fix: reject path traversal and home-dir patterns in media parse layer (#54642) * fix: reject path traversal and home-dir patterns in media parse layer * Update parse tests * Block reset-profile on lower-privilege browser request surfaces (#54618) * Block reset-profile on lower-privilege browser request surfaces * add missing tests * Fix tests * Test fix * fix: trigger compaction on LLM timeout with high context usage (#46417) Merged via squash. Prepared head SHA: 619bc4c1fa1db3829ff3aa78ffcab8e4201379b5 Co-authored-by: joeykrug <[email protected]> Co-authored-by: jalehman <[email protected]> Reviewed-by: @jalehman * fix(talk-voice): enforce operator.admin scope on /voice set config writes (#54461) * fix(talk-voice): enforce operator.admin scope on /voice set config writes * fix(talk-voice): align scope guard with phone-control pattern Use optional chaining (?.) instead of Array.isArray so webchat callers with undefined scopes are rejected, matching the established pattern in phone-control. Add test for webchat-with-no-scopes case. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> * fix: apply host-env blocklist to auth-profile env refs in daemon install (#54627) * fix: apply host-env blocklist to auth-profile env refs in daemon install Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * ci: retrigger checks --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> * OpenShell: exclude hooks/ from mirror sync (#54657) * OpenShell: exclude hooks/ from mirror sync * OpenShell: make excludeDirs case-insensitive for cross-platform safety * Trigger preflight compaction from transcript estimates when usage is stale (#49479) Merged via squash. Prepared head SHA: 8d214b708b433702aba21b86ad19b6e2721c3ee0 Co-authored-by: jared596 <[email protected]> Co-authored-by: jalehman <[email protected]> Reviewed-by: @jalehman * refactor(sandbox): remove tool policy facade (#54684) * refactor(sandbox): remove tool policy facade * fix(sandbox): harden blocked-tool guidance * fix(sandbox): avoid control-char guidance leaks * fix: harden sandbox blocked-tool guidance (#54684) (thanks @ngutman) * fix(ci): restore main green * Filter untrusted CWD .env entries before OpenClaw startup (#54631) * Filter untrusted CWD .env entries before OpenClaw startup * Add missing test file * Fix missing and updated files * Address feedback * Feedback updates * Feedback update * Add test coverage * Unit test fix * fix: enforce localRoots sandbox on Feishu docx upload file reads (#54693) * fix: enforce localRoots sandbox on Feishu docx upload file reads * Formatting fixes * Update tests * Feedback updates * test: introduce planner-backed test runner, stabilize local builds (#54650) * test: stabilize ci and local vitest workers * test: introduce planner-backed test runner * test: address planner review follow-ups * test: derive planner budgets from host capabilities * test: restore planner filter helper import * test: align planner explain output with execution * test: keep low profile as serial alias * test: restrict explicit planner file targets * test: clean planner exits and pnpm launch * test: tighten wrapper flag validation * ci: gate heavy fanout on check * test: key shard assignments by unit identity * ci(bun): shard vitest lanes further * test: restore ci overlap and stabilize planner tests * test: relax planner output worker assertions * test: reset plugin runtime state in optional tools suite * ci: split macos node and swift jobs * test: honor no-isolate top-level concurrency budgets * ci: fix macos swift format lint * test: cap max-profile top-level concurrency * ci: shard macos node checks * ci: use four macos node shards * test: normalize explain targets before classification * feat(cli): add json schema to cli tool (#54523) Merged via squash. Prepared head SHA: 39c15ee70d04a1721ebafde327b3b95355653fac Co-authored-by: kvokka <[email protected]> Co-authored-by: altaywtf <[email protected]> Reviewed-by: @altaywtf * fix(schema): tools.web.fetch.maxResponseBytes #53397 (#53401) Merged via squash. Prepared head SHA: 5d10a98bdb1f7fc3065adf438c4d572c3d61f8ce Co-authored-by: erhhung <[email protected]> Co-authored-by: altaywtf <[email protected]> Reviewed-by: @altaywtf * fix: make buttons schema optional in message tool (#54418) Merged via squash. Prepared head SHA: 0805c095e930d669ba1b3aedc09baec80882ce45 Co-authored-by: adzendo <[email protected]> Co-authored-by: altaywtf <[email protected]> Reviewed-by: @altaywtf * fix(gateway): block silent reconnect scope-upgrade escalation (#54694) * fix(gateway): block silent reconnect scope-upgrade escalation * formatting updateas * Resolve feedback * formatting fixes * Update src/gateway/server.silent-scope-upgrade-reconnect.poc.test.ts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Feedback updates * fix unit test * Feedback update * Review feedback update * More Greptile nit fixes --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: allow msteams feedback and welcome config keys (#54679) Merged via squash. Prepared head SHA: f56a15ddeaeb5369fd8feac09bb89d3d600d6ff4 Co-authored-by: gumclaw <[email protected]> Co-authored-by: altaywtf <[email protected]> Reviewed-by: @altaywtf * fix(mattermost): thread resolved cfg through reply delivery send calls (#48347) Merged via squash. Prepared head SHA: 7ca468e365087c802be32675696080a8dd11fae2 Co-authored-by: mathiasnagler <[email protected]> Co-authored-by: mukhtharcm <[email protected]> Reviewed-by: @mukhtharcm * feat: add video generation core infrastructure and extend image generation parameters (#53681) * feat: add video generation core infrastructure and extend image generation parameters Add full video generation capability to OpenClaw core: - New `video_generate` agent tool with support for prompt, duration, aspect ratio, resolution, seed, watermark, I2V (first/last frame), camerafixed, and draft mode - New `VideoGenerationProvider` plugin SDK type and `registerVideoGenerationProvider` API - New `src/video-generation/` module (types, runtime with fallback, provider registry) - New `openclaw/plugin-sdk/video-generation` export for external plugins - 200MB max file size for generated videos (vs default 5MB for images) Extend image generation with additional parameters: - `seed`, `watermark`, `guidanceScale`, `optimizePrompt`, `providerOptions` - New `readBooleanParam()` helper in tool common utilities Update plugin registry, contracts, and all test mocks to include `videoGenerationProviders` and `videoGenerationProviderIds`. Made-with: Cursor * fix: validate aspect ratio against target provider when model override is set * cleanup: remove redundant ?? undefined from video/image generate tools * chore: regenerate plugin SDK API baseline after video generation additions --------- Co-authored-by: yongliang.xie <[email protected]> * fix: deliver verbose tool summaries in Telegram forum topics (#43236) (thanks @frankbuild) * fix(auto-reply): deliver verbose tool summaries in Telegram forum topics Forum topics have ChatType 'group' but are threaded conversations where verbose tool output should be delivered (same as DMs). The shouldSendToolSummaries gate now checks IsForum to allow tool summaries in forum topic sessions. Fixes #43206 * test: add sendToolResult count assertion per review feedback * fix: add changelog for forum topic verbose tool summaries (#43236) (thanks @frankbuild) --------- Co-authored-by: Ayaan Zaidi <[email protected]> * docs: add WeChat channel via official Tencent iLink Bot plugin (#52131) (thanks @MonkeyLeeT) * docs: add WeChat channel via official Tencent iLink Bot plugin Add WeChat to the README channel lists and setup section. Uses the official Tencent-published plugin @tencent-weixin/openclaw-weixin which connects via the iLink Bot API (QR code login, long-poll). Requires WeChat 8.0.70+ with the ClawBot plugin enabled; the plugin is being rolled out gradually by Tencent. Covers: setup steps, capabilities (DM-only, media up to 100 MB, multi-account, pairing authorization, typing indicators, config path), and the context token restart caveat. * docs: update WeChat plugin install for v2.0 compatibility - Add version compatibility note (v2.x requires OpenClaw >= 2026.3.22, @legacy tag for older hosts) - Add plugins.allow step (required since plugins.allow was introduced) * docs: drop manual plugins.allow/enable steps (handled by plugins install) * docs: fix multi-account instruction to require explicit --account id * docs: trim WeChat section to match neighboring channels, fix pairing link * docs: sync WeChat channel docs --------- Co-authored-by: Ayaan Zaidi <[email protected]> * ci: collapse preflight manifest routing (#54773) * ci: collapse preflight manifest routing * ci: fix preflight workflow outputs * ci: restore compat workflow tasks * ci: match macos shards to windows * ci: collapse macos swift jobs * ci: skip empty submodule setup * ci: drop submodule setup from node env * fix(whatsapp): clarify allowFrom policy error (#54850) * fix: resolve telegram token fallback for binding-created accounts (#54362) (thanks @openperf) * fix(telegram): resolve channel-level token fallthrough for binding-created accountIds Fixes #53876 * fix(telegram): align isConfigured with resolveTelegramToken multi-bot guard * fix(telegram): use normalized account lookup and require available token * fix: tighten systemd duplicate gateway detection (#45328) (thanks @gregretkowski) * daemon: tighten systemd duplicate gateway detection (#15849) * fix three issues from PR review * fix windows unit tests due to posix/windows path differences * ensure line continuations are handled in systemd units * fix misleading test name * attempt fix windows test due to fs path separator * fix system_dir separator, fix platform side-effect * change approach for mocking systemd filesystem test * normalize systemd paths to linux style * revert to vers that didnt impact win32 tests * back out all systemd inspect tests * change test approach to avoid other tests issues * fix: tighten systemd duplicate gateway detection (#45328) (thanks @gregretkowski) --------- Co-authored-by: Ayaan Zaidi <[email protected]> * fix: use provider-aware context window lookup (#54796) (thanks @neeravmakwana) * fix(status): use provider-aware context window lookup * test(status): cover provider-aware context lookup * fix: use provider-aware context window lookup (#54796) (thanks @neeravmakwana) --------- Co-authored-by: Ayaan Zaidi <[email protected]> * fix: restore CLI message transcript mirroring (#54187) (thanks @KevInTheCloud5617) * fix: pass agentId in CLI message command to enable session transcript writes The CLI `openclaw message send` command was not passing `agentId` to `runMessageAction()`, causing the outbound session route resolution to be skipped (it's gated on `agentId && !dryRun`). Without a route, the `mirror` object is never constructed, and `appendAssistantMessageToSessionTranscript()` is never called. This fix resolves the agent ID from the config (defaulting to "main") and passes it through, enabling transcript mirroring for all channels when using the CLI. Closes #54186 * fix: format message.ts with oxfmt * fix: use resolveDefaultAgentId instead of cfg.agent * fix: restore CLI message transcript mirroring (#54187) (thanks @KevInTheCloud5617) --------- Co-authored-by: Ayaan Zaidi <[email protected]> * fix: restore Kimi Code under Moonshot setup (#54619) (thanks @sparkyrider) * Onboarding: restore Kimi Code under Moonshot setup * Update extensions/kimi-coding/index.ts Fix naming convention in metadata Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: auto-enable configured channel plugins in routed CLI commands (#54809) (thanks @neeravmakwana) * CLI: auto-enable configured channel plugins in routed commands * fix: auto-enable configured channel plugins in routed CLI commands (#54809) (thanks @neeravmakwana) --------- Co-authored-by: Ayaan Zaidi <[email protected]> * fix: restore image-tool generic provider fallback (#54858) (thanks @MonkeyLeeT) * Image tool: restore generic provider fallback * Image tool: cover multi-image generic fallback * test: tighten minimax-portal image fallback coverage * fix: restore image-tool generic provider fallback (#54858) (thanks @MonkeyLeeT) --------- Co-authored-by: Ayaan Zaidi <[email protected]> * fix: support OpenAI Codex media understanding (#54829) (thanks @neeravmakwana) * OpenAI: register Codex media understanding provider * fix: route codex image prompts through system instructions * fix: add changelog for codex image tool fix (#54829) (thanks @neeravmakwana) * fix: remove any from provider registration tests (#54829) (thanks @neeravmakwana) --------- Co-authored-by: Ayaan Zaidi <[email protected]> * fix(whatsapp): drop fromMe echoes in self-chat DMs using outbound ID tracking (#54570) Merged via squash. Prepared head SHA: dad53caf3974e2e90a358d0561a0b0ee9bd924f2 Co-authored-by: joelnishanth <[email protected]> Co-authored-by: mcaxtr <[email protected]> Reviewed-by: @mcaxtr * Revert "feat: add video generation core infrastructure and extend image generation parameters (#53681)" (#54943) This reverts commit 4cb8dde894e3a85ce81b770869d7bd86e6b6e7a1. * msteams: fetch thread history via Graph API for channel replies (#51643) * msteams: fetch thread history via Graph API for channel replies * msteams: address PR #51643 review feedback - Wrap resolveTeamGroupId Graph call in try/catch, fall back to raw conversationTeamId when Team.ReadBasic.All permission is missing - Remove dead fetchChatMessages function (exported but never called) - Add JSDoc documenting oldest-50-replies Graph API limitation Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * msteams: address thread history PR review comments * msteams: only cache team group IDs on successful Graph lookup Avoid caching raw conversationTeamId as a Graph team GUID when the /teams/{id} lookup fails — the raw ID may be a Bot Framework conversation key, not a valid GUID, causing silent thread-history failures for the entire cache TTL. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> * msteams: add search message action (#54832) * msteams: add pin/unpin, list-pins, and read message actions Wire up Graph API endpoints for message read, pin, unpin, and list-pins in the MS Teams extension, following the same patterns as edit/delete. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * msteams: address PR review comments for pin/unpin/read actions - Handle 204 No Content in postGraphJson (Graph mutations may return empty body) - Strip conversation:/user: prefixes in resolveConversationPath to avoid Graph 404s - Remove dead variable in channel pin branch - Rename unpin param from messageId to pinnedMessageId for semantic clarity - Accept both pinnedMessageId and messageId in unpin action handler for compat Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * msteams: resolve user targets + add User-Agent to Graph helpers - Resolve user:<aadId> targets to actual conversation IDs via conversation store before Graph API calls (fixes 404 for DM-context actions) - Add User-Agent header to postGraphJson/deleteGraphRequest for consistency with fetchGraphJson after rebase onto main Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * msteams: resolve DM targets to Graph chat IDs + expose pin IDs - Prefer cached graphChatId over Bot Framework conversation IDs for user targets; throw descriptive error when no Graph-compatible ID is available - Add `id` field to list-pins rows so default formatters surface the pinned resource ID needed for the unpin flow Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * msteams: add react and reactions (list) message actions * msteams: add search message action via Graph API * msteams: fix search query injection, add ConsistencyLevel header, use manual query string --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> * fix(plugins): skip allowlist warning for config paths * feat: Add Microsoft Foundry provider with Entra ID authentication (#51973) * Microsoft Foundry: add native provider * Microsoft Foundry: tighten review fixes * Microsoft Foundry: enable by default * Microsoft Foundry: stabilize API routing * fix: restore inbound image embedding for CLI routed BlueBubbles turns (#51373) * fix(cli): hydrate prompt image refs for inbound media * Agents: harden CLI prompt image hydration (#51373) * test: fix CLI prompt image hydration helper mocks * fix: route codex responses over websocket and preserve tool warnings (#53702) (thanks @Nanako0129) * fix: route codex responses over websocket and suppress gated core tool warnings * fix: rebase codex websocket patch onto main * fix: preserve explicit alsoAllow warnings (#53702) (thanks @Nanako0129) --------- Co-authored-by: Ayaan Zaidi <[email protected]> * Docs: rename modelstudio.md to qwen_modelstudio.md, add Standard API endpoints (#54407) * Docs: rename modelstudio.md to qwen_modelstudio.md, add Standard API endpoints * refine docs * Docs: fix broken link in providers/index.md after modelstudio rename * Docs: add redirect from /providers/modelstudio to /providers/qwen_modelstudio * Docs: adjust the order in index.md * docs: rename modelstudio to qwen_modelstudio, add Standard API endpoints (#54407) (thanks @wenmengzhou) --------- Co-authored-by: George Zhang <[email protected]> * fix: wire microsoft foundry into contract registry * docs: refresh config baseline for microsoft foundry * fix: add slack upload-file action (#54987) (thanks @kevinlin-openai) * feat(slack): add upload-file action Co-authored-by: Codex <[email protected]> * fix(slack): guard upload-file routing Co-authored-by: Codex <[email protected]> * fix(slack): tighten upload-file validation --------- Co-authored-by: kevinlin-openai <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: Ayaan Zaidi <[email protected]> * build: update plugin sdk api baseline * Remove Qwen OAuth integration (qwen-portal-auth) (#52709) * Remove Qwen OAuth integration (qwen-portal-auth) Qwen OAuth via portal.qwen.ai is being deprecated by the Qwen team due to traffic impact on their primary Qwen Code user base. Users should migrate to the officially supported Model Studio (Alibaba Cloud Coding Plan) provider instead. Ref: https://github.com/openclaw/openclaw/issues/49557 - Delete extensions/qwen-portal-auth/ plugin entirely - Remove qwen-portal from onboarding auth choices, provider aliases, auto-enable list, bundled plugin defaults, and pricing cache - Remove Qwen CLI credential sync (external-cli-sync, cli-credentials) - Remove QWEN_OAUTH_MARKER from model auth markers - Update docs/providers/qwen.md to redirect to Model Studio - Update model-providers docs (EN + zh-CN) to remove Qwen OAuth section - Regenerate config and plugin-sdk baselines - Update all affected tests Co-authored-by: Qwen-Coder <[email protected]> * Clean up residual qwen-portal references after OAuth removal * Add migration hint for deprecated qwen-portal OAuth provider * fix: finish qwen oauth removal follow-up --------- Co-authored-by: Qwen-Coder <[email protected]> Co-authored-by: Frank Yang <[email protected]> * fix(extensions): route fetch calls through fetchWithSsrFGuard (#53929) * fix(extensions): route fetch calls through fetchWithSsrFGuard Replace raw fetch() with fetchWithSsrFGuard in BlueBubbles, Mattermost, Nextcloud Talk, and Thread Ownership extensions so outbound requests go through the shared DNS-pinning and network-policy layer. BlueBubbles: thread allowPrivateNetwork from account config through all fetch call sites (send, chat, reactions, history, probe, attachments, multipart). Add _setFetchGuardForTesting hook for test overrides. Mattermost: add guardedFetchImpl wrapper in createMattermostClient that buffers the response body before releasing the dispatcher. Handle null-body status codes (204/304). Nextcloud Talk: wrap both sendMessage and sendReaction with fetchWithSsrFGuard and try/finally release. Thread Ownership: add fetchWithSsrFGuard and ssrfPolicyFromAllowPrivateNetwork to the plugin SDK surface; use allowPrivateNetwork:true for the Docker-internal forwarder. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix(extensions): improve null-body handling and test harness cleanup Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix(bluebubbles): default to strict SSRF policy when allowPrivateNetwork is unset Callers that omit allowPrivateNetwork previously got undefined policy, which caused blueBubblesFetchWithTimeout to fall through to raw fetch and bypass the SSRF guard entirely. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix(bluebubbles): thread allowPrivateNetwork through action and monitor call sites Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix(mattermost,nextcloud-talk): add allowPrivateNetwork config for self-hosted/LAN deployments * fix: regenerate config docs baseline for new allowPrivateNetwork fields --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> * BlueBubbles: enrich group participants with local Contacts names (#54984) * BlueBubbles: enrich group participants with Contacts names * BlueBubbles: gate contact enrichment behind opt in config * chore: add lockfile entry for `extensions/microsoft-foundry` * fix(discord): force fresh gateway reconnects (#54697) * fix(discord): force fresh gateway reconnects * fix(discord): harden forced reconnect teardown * fix(discord): retry after socket drain timeouts * fix(discord): guard forced socket teardown * fix(discord): stop cleanly during reconnect drain * Feishu: validate webhook signatures before parsing (#55083) * Feishu: validate webhook signatures before parsing * Scripts: allow Feishu raw body guard callsite * fix(agents): enforce session_status guard after sessionId resolution (#55105) * fix(agents): enforce visibility guard after sessionId resolution in session_status When a sessionId (rather than an explicit agent key) is passed to the session_status tool, the sessionId resolution block rewrites requestedKeyRaw to an explicit "agent:..." key. The subsequent visibility guard check at line 375 tested `!requestedKeyRaw.startsWith("agent:")`, which was now always false after resolution — skipping the visibility check entirely. This meant a sandboxed agent could bypass visibility restrictions by providing a sessionId instead of an explicit session key. Fix: use the original `isExplicitAgentKey` flag (captured before resolution) instead of re-checking the dynamic requestedKeyRaw. This ensures the visibility guard runs for sessionId inputs while still skipping the redundant check for inputs that were already validated at the earlier explicit-key check (lines 281-286). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * test: cover session status sessionId guard * test: align parent sessionId guard coverage --------- Co-authored-by: Kevin Sheng <[email protected]> Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> * Telegram: enforce DM auth for callbacks (#55112) * fix(msteams): align feedback invoke authorization (#55108) * msteams: align feedback invoke authorization * msteams: fix feedback allowlist regressions * msteams: tighten feedback group authorization * fix(bluebubbles): auto-allow private network for local serverUrl and add allowPrivateNetwork to channel schema * Matrix: gate verification notices on DM access (#55122) * fix: surface provider-specific rate limit error message (#54433) (#54512) Merged via squash. Prepared head SHA: 755cff833c1f6aca06ab2c3e1f802f35e7c4d553 Co-authored-by: bugkill3r <[email protected]> Co-authored-by: altaywtf <[email protected]> Reviewed-by: @altaywtf * test: fix bluebubbles attachment ssrf expectations * docs: add beta release testing guidance * fix(bluebubbles): enable group participant enrichment by default, add fallback fetch and handle field aliases * fix: prefer freshest duplicate row promotion * fix: replace stale canonical duplicate rows * docs: sync config baseline * build: prepare 2026.3.25 unreleased * ACP: sanitize terminal tool titles (#55137) * ACP: sanitize terminal tool titles Co-authored-by: nexrin <[email protected]> * Config: refresh config baseline and stabilize restart pid test --------- Co-authored-by: nexrin <[email protected]> * Google Chat: require stable group ids (#55131) * Google Chat: require stable group ids * Google Chat: fail closed on deprecated room keys * Feishu: reject legacy raw card command payloads (#55130) * Feishu: reject legacy raw card callbacks * Feishu: cover legacy text card payloads * Docs: refresh config baseline * CI: refresh PR checks * Feishu: limit legacy card guard scope * fix(bluebubbles): refactor sendMessageBlueBubbles to use resolveBlueBubblesServerAccount and enhance private network handling in tests * test(memory): initialize providers in lazy manager tests * test: fix portable stderr capture and env leakage (#55184) * docs: add beta blocker contributor guidance (#55199) * docs: add beta blocker contributor guidance * fix: tighten beta blocker labeling and flaky config test * docs: update PR template review guidance * refactor: dedupe msteams graph actions * fix(bluebubbles): throttle webhook auth guesses (#55133) * fix(bluebubbles): throttle webhook auth guesses * test(bluebubbles): isolate attachment ssrf config * test(bluebubbles): hoist attachment mocks * docs: refresh bluebubbles config baseline * fix(bluebubbles): trust proxied webhook client IPs * fix(bluebubbles): honor trusted proxy webhook IPs * fix(bluebubbles): honor real-ip fallback for webhooks * refactor: share web media loader * fix: avoid duplicate ACP Telegram finals (#55173) * fix: avoid duplicate final ACP text on telegram * fix: keep ACP final fallback for non-telegram blocks * fix: count telegram ACP block replies as success * fix: recover ACP final fallback after block failures * fix: settle telegram ACP block delivery before fallback * test: isolate ACP dispatch mocks under shared workers * fix: prefer telegram provider for ACP visibility * refactor: share matrix and telegram dedupe helpers * fix: prefer freshest duplicate store matches * feat: pluginize cli inference backends * test: dedupe foundry auth fixtures * fix(whatsapp): unwrap quoted wrapper messages * test: share pi compaction fixtures * test: share redact and approval fixtures * fix: preserve metadata on voice session touches * test: dedupe web search provider fixtures * synology-chat: throttle webhook token guesses (#55141) * synology-chat: throttle webhook token guesses * synology-chat: keep valid webhook traffic within configured limits * docs: refresh generated config baseline * synology-chat: enforce lockout after repeated token failures * fix: preserve reset ownership metadata * test: improve test runner help text (#55227) * test: improve test runner help text * test: print extension help to stdout * test: leave extension help passthrough alone * test: parse timing update flags in one pass * perf: speed up channel test runs * test: share msteams monitor and pi runner fixtures * fix: preserve reset spawn depth * test: dedupe config compatibility fixtures * ci: make docker release tag-driven * fix: preserve reset elevated level * fix: preserve reset spawn context * test: share subagent and policy test fixtures * perf: enable local channel planner parallelism on node 25 * test: dedupe remaining agent test seams * telegram: throttle repeated webhook auth guesses (#55142) * telegram: throttle repeated webhook auth guesses * telegram: use per-listener webhook rate limits * config: stabilize doc baseline ordering * test: share matrix migration fixtures * test: share gateway authz and watchdog fixtures * test: share ui reconnect and storage helpers * perf: overlap isolated channel runs with shared lane * fix: preserve reset session behavior config * fix: preserve reset channel identity * fix: preserve reset acp session metadata * fix: preserve reset cli session linkage * refactor: share auto-reply reply helpers * test: add docker cli-backend smoke * docs: note guest openclaw shim in parallels skill * test: share plugin auth and ui storage fixtures * test: dedupe matrix setup seams * test: dedupe ui chat seams * style: normalize ui slash executor formatting * fix(ci): refresh plugin sdk api baseline * fix: keep plugin HTTP runtime scopes least-privileged (#55284) * Gateway: require caller scope for subagent session deletion (#55281) * gateway: require pairing for backend scope upgrades (#55286) * test: dedupe secrets and guardrail fixtures * bluebubbles: honor reaction mention gating (#55283) * Gateway: align HTTP session history scopes (#55285) * Gateway: require scopes for HTTP session history * Gateway: cover missing HTTP history scope header * refactor: dedupe gateway and binding helpers * test: dedupe extension channel fixtures * Gateway: require requester ownership for HTTP session kills (#55308) * test: share cli and channel setup fixtures * test: share auto-reply typing helpers * refactor: share plugin setup helpers * ci: optimize windows test shard fanout (#55261) * ci: reduce windows test shard fanout * ci: tighten windows shard target * ci: back off windows shard target * ci: restore windows shard cap * fix: include dashboard children in owner filters * fix: expose parent session keys in sessions list * fix: expose spawned session owners in sessions list * fix: keep spawned session owners in live events * refactor: share browser and sandbox helpers * refactor: share discord outbound session routing * refactor: dedupe gateway session resolve visibility * refactor: share slack and telegram action helpers * fix: auto-load bundled plugin capabilities from config refs * test: share cli and doctor test helpers * fix(ci): repair discord regression tests * test(gateway): strip MiniMax live scaffolding * test: share cli command and discord test helpers * test: dedupe extension channel fixtures * fix(ci): repair discord message handler tests * test: reduce remaining clone seams * perf: speed up test parallelism * telegram: rebuild transport after stalled polling cycles * test: share discord monitor fixtures * fix: preserve cli sessions across model changes * test: split cli agent command coverage * test: eliminate remaining clone seams * refactor: dedupe cli runner session reuse * fix(ci): repair discord harness regressions * fix(ci): clean up discord harness types * fix(agents): classify "Failed to extract accountId from token" as auth error for failover (#27055) (#55206) Co-authored-by: Lyle Hopkins <[email protected]> * test: dedupe discord provider proxy overrides * fix(ci): restore discord provider test seams * fix(ci): format discord provider follow-up * refactor: split telegram polling and sdk surfaces * fix: export shared channel action enum helpers * fix: backfill claude cli chat history * fix: skip cli backends in models auth warnings * test: dedupe telegram polling session harness * fix(ci): repair discord and telegram follow-ups * refactor: clean plugin capability boundaries * refactor: share speech normalization helpers * perf: speed up shared extension test batches * refactor: simplify bundled plugin contracts * refactor: move memory flush ownership into memory plugin * refactor: move memory tooling into memory-core extension * refactor: add memory-core extension sources * fix: unify claude cli imported tool messages * test: share planner and sandbox test helpers * refactor: split memory-core plugin helpers * fix: regenerate pnpm-lock.yaml after upstream merge (workspace dep added) * fix: resolve post-absorb CI failures After absorbing 140 upstream commits, several botster-specific files needed updating to match upstream SDK changes: 1. extensions/zulip/index.ts — import order reordered by oxfmt (types import after value import, as required by formatter) 2. scripts/lib/plugin-sdk-entrypoints.json + package.json — './plugin-sdk/zulip' subpath export missing from upstream's entrypoints list (our addition). Added 'zulip' to the canonical entrypoints JSON so sync-exports keeps it. 3. src/seks/spine-exec-intercept.test.ts — - TextContent|ImageContent union: .text access now requires type narrowing (cast to { type: 'text'; text: string }) - AgentTool now requires execute: test case for missing-execute path casts to any (intentionally testing fallback behavior) 4. docs/.generated/* — regenerated config-baseline and plugin-sdk-api-baseline after upstream config schema + plugin SDK surface changes. * fix: lint and docs issues from post-absorb CI - src/seks/spine-exec-intercept.test.ts: replace 'as any' with 'as Parameters<typeof createSpineExecTool>[0]' to satisfy oxlint no-explicit-any rule while still testing the no-execute fallback path - docs/providers/anthropic.md: rename duplicate '### Config snippet' heading (added by upstream in ab4de18) to '### Config snippet (claude-cli)' to satisfy markdownlint MD024 --------- Co-authored-by: Peter Steinberger <[email protected]> Co-authored-by: VACInc <[email protected]> Co-authored-by: VACInc <[email protected]> Co-authored-by: Ayaan Zaidi <[email protected]> Co-authored-by: lbo728 <[email protected]> Co-authored-by: hnshah <[email protected]> Co-authored-by: Bob Shah <[email protected]> Co-authored-by: ToToKr <[email protected]> Co-authored-by: Andy <[email protected]> Co-authored-by: Sparkyrider <[email protected]> Co-authored-by: Zoher Ghadyali <[email protected]> Co-authored-by: Liu Yuan <[email protected]> Co-authored-by: Gracie Gould <[email protected]> Co-authored-by: “graciegould” <“[email protected]”> Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: ZhangXuan <[email protected]> Co-authored-by: khhjoe <[email protected]> Co-authored-by: Harold Hunt <[email protected]> Co-authored-by: huntharo <[email protected]> Co-authored-by: Nimrod Gutman <[email protected]> Co-authored-by: Lin Z <[email protected]> Co-authored-by: George Zhang <[email protected]> Co-authored-by: Devin Robison <[email protected]> Co-authored-by: xieyongliang <[email protected]> Co-authored-by: bojsun <[email protected]> Co-authored-by: Jerry <[email protected]> Co-authored-by: yongliang.xie <[email protected]> Co-authored-by: liyuan97 <[email protected]> Co-authored-by: tars90percent <[email protected]> Co-authored-by: Jackal Xin <[email protected]> Co-authored-by: jackal092927 <[email protected]> Co-authored-by: jalehman <[email protected]> Co-authored-by: Tak Hoffman <[email protected]> Co-authored-by: Matt Van Horn <[email protected]> Co-authored-by: chenxingzhen <[email protected]> Co-authored-by: infichen <[email protected]> Co-authored-by: M1a0 <[email protected]> Co-authored-by: Andrii Furmanets <[email protected]> Co-authored-by: afurm <[email protected]> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: kiranvk2011 <[email protected]> Co-authored-by: altaywtf <[email protected]> Co-authored-by: Joseph Krug <[email protected]> Co-authored-by: joeykrug <[email protected]> Co-authored-by: Jacob Tomlinson <[email protected]> Co-authored-by: Jared <[email protected]> Co-authored-by: Mikhail Beliakov <[email protected]> Co-authored-by: kvokka <[email protected]> Co-authored-by: Erhhung Yuan <[email protected]> Co-authored-by: erhhung <[email protected]> Co-authored-by: adzendo <[email protected]> Co-authored-by: adzendo <[email protected]> Co-authored-by: gumclaw <[email protected]> Co-authored-by: gumclaw <[email protected]> Co-authored-by: Mathias Nagler <[email protected]> Co-authored-by: mathiasnagler <[email protected]> Co-authored-by: mukhtharcm <[email protected]> Co-authored-by: Frank the Builder <[email protected]> Co-authored-by: Ted Li <[email protected]> Co-authored-by: Marcus Castro <[email protected]> Co-authored-by: wangchunyue <[email protected]> Co-authored-by: Greg Retkowski <[email protected]> Co-authored-by: Neerav Makwana <[email protected]> Co-authored-by: Kevin Boyle <[email protected]> Co-authored-by: OfflynAI <[email protected]> Co-authored-by: sudie-codes <[email protected]> Co-authored-by: MetaX e|acc <[email protected]> Co-authored-by: Tyler Yust <[email protected]> Co-authored-by: Nyanako <[email protected]> Co-authored-by: wenmeng zhou <[email protected]> Co-authored-by: kevinlin-openai <[email protected]> Co-authored-by: kevinlin-openai <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: pomelo <[email protected]> Co-authored-by: Qwen-Coder <[email protected]> Co-authored-by: Frank Yang <[email protected]> Co-authored-by: Altay <[email protected]> Co-authored-by: Kevin Sheng <[email protected]> Co-authored-by: Tyler Yust <[email protected]> Co-authored-by: Saurabh Mishra <[email protected]> Co-authored-by: bugkill3r <[email protected]> Co-authored-by: Shakker <[email protected]> Co-authored-by: nexrin <[email protected]> Co-authored-by: pkuGeo <[email protected]> Co-authored-by: Lyle Hopkins <[email protected]> Co-authored-by: Lyle Hopkins <[email protected]>

@altaywtf

…limit message (openclaw#49834) Merged via squash. Prepared head SHA: 7c488c0 Co-authored-by: kiranvk-2011 <[email protected]> Co-authored-by: altaywtf <[email protected]> Reviewed-by: @altaywtf

@altaywtf

…limit message (openclaw#49834) Merged via squash. Prepared head SHA: 7c488c0 Co-authored-by: kiranvk-2011 <[email protected]> Co-authored-by: altaywtf <[email protected]> Reviewed-by: @altaywtf

@altaywtf

…limit message (openclaw#49834) Merged via squash. Prepared head SHA: 7c488c0 Co-authored-by: kiranvk-2011 <[email protected]> Co-authored-by: altaywtf <[email protected]> Reviewed-by: @altaywtf

@altaywtf

…limit message (openclaw#49834) Merged via squash. Prepared head SHA: 7c488c0 Co-authored-by: kiranvk-2011 <[email protected]> Co-authored-by: altaywtf <[email protected]> Reviewed-by: @altaywtf

@altaywtf

…limit message (openclaw#49834) Merged via squash. Prepared head SHA: 7c488c0 Co-authored-by: kiranvk-2011 <[email protected]> Co-authored-by: altaywtf <[email protected]> Reviewed-by: @altaywtf

@altaywtf

…limit message (openclaw#49834) Merged via squash. Prepared head SHA: 7c488c0 Co-authored-by: kiranvk-2011 <[email protected]> Co-authored-by: altaywtf <[email protected]> Reviewed-by: @altaywtf

kiranvk-2011 requested a review from a team as a code owner March 18, 2026 13:51

openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels Mar 18, 2026

greptile-apps Bot reviewed Mar 18, 2026

View reviewed changes

Comment thread src/agents/auth-profiles/usage.ts Outdated

Comment thread src/auto-reply/reply/agent-runner-execution.ts Outdated

Comment thread src/auto-reply/reply/agent-runner-execution.ts Outdated

chatgpt-codex-connector Bot reviewed Mar 18, 2026

View reviewed changes

Comment thread src/agents/auth-profiles/usage.ts Outdated

Comment thread src/auto-reply/reply/agent-runner-execution.ts Outdated

kiranvk-2011 force-pushed the fix/per-model-cooldown-stepped-backoff branch from 2b5c264 to 106d513 Compare March 18, 2026 14:08

chatgpt-codex-connector Bot reviewed Mar 18, 2026

View reviewed changes

Comment thread src/agents/auth-profiles/usage.ts Outdated

Comment thread src/auto-reply/reply/agent-runner-execution.ts Outdated

kiranvk-2011 mentioned this pull request Mar 18, 2026

bug: Multiple CI jobs fail on main branch — contracts, channels, extensions, extension-fast, install-smoke, Windows shards #49848

Closed

openclaw-barnacle Bot added size: M and removed size: S labels Mar 18, 2026

chatgpt-codex-connector Bot reviewed Mar 18, 2026

View reviewed changes

Comment thread src/agents/model-fallback.ts

chatgpt-codex-connector Bot reviewed Mar 18, 2026

View reviewed changes

Comment thread src/agents/model-fallback.ts Outdated

altaywtf self-assigned this Mar 18, 2026

chatgpt-codex-connector Bot reviewed Mar 18, 2026

View reviewed changes

Comment thread src/agents/model-fallback.ts

kiranvk-2011 force-pushed the fix/per-model-cooldown-stepped-backoff branch from 487deab to 0016af2 Compare March 18, 2026 20:00

kiranvk-2011 force-pushed the fix/per-model-cooldown-stepped-backoff branch from 0016af2 to bd401f7 Compare March 19, 2026 11:15

chatgpt-codex-connector Bot reviewed Mar 19, 2026

View reviewed changes

Comment thread src/agents/model-fallback.ts Outdated

kiranvk-2011 force-pushed the fix/per-model-cooldown-stepped-backoff branch from 16c4fa3 to 6d58d2f Compare March 19, 2026 11:55

altaywtf force-pushed the fix/per-model-cooldown-stepped-backoff branch 2 times, most recently from d1764f1 to fd0e16c Compare March 24, 2026 18:10

chatgpt-codex-connector Bot reviewed Mar 24, 2026

View reviewed changes

Comment thread src/auto-reply/reply/agent-runner-execution.ts Outdated

Comment thread src/agents/model-fallback.ts Outdated

altaywtf force-pushed the fix/per-model-cooldown-stepped-backoff branch from fd0e16c to 27b7573 Compare March 24, 2026 19:11

github-actions Bot mentioned this pull request Mar 25, 2026

📡 Upstream Digest — 2026-03-25 20:31 UTC curtismercier/openclaw-mods#363

Open

kiranvk-2011 mentioned this pull request Apr 3, 2026

fix: distinguish user-initiated model switches from system fallbacks in LiveSessionModelSwitchError #60266

Merged

4 tasks

kai-the-man Bot mentioned this pull request Apr 29, 2026

[Feature]: MCP bundle materializer needs exponential backoff for rate-limited server init #74183

Closed

yanbinwa mentioned this pull request Apr 29, 2026

[Migrated #74183] [Feature]: MCP bundle materializer needs exponential backoff for rate-limited server init yanbinwa/public-repo-openclaw#54

Open

This was referenced May 3, 2026

[Feature]: Make fallback cooldowns RPM-aware to reduce unnecessary errors and improve availability #24107

Closed

Dynamic Model Switching Based on API Rate Limits #6966

Closed

clawsweeper Bot mentioned this pull request May 20, 2026

Model-scoped cooldowns are not persisted for incomplete-turn exhaustion paths #67661

Closed

fanyangCS mentioned this pull request Jun 2, 2026

GitHub Copilot timeout marks auth profile in cooldown and blocks same-provider fallback models #89382

Closed

This was referenced Jun 17, 2026

fix: Reset subagent session idle timeout on successful tool calls to prevent premature termination #94124

Closed

fix: refresh subagent idle timeout on successful tool calls #94133

Closed

This was referenced Jun 21, 2026

Persist model exhaustion cooldowns #81046

Closed

[Feature] Queue prompts when API limits prevent an immediate response #49251

Open

Uh oh!

Uh oh!

Conversation

kiranvk-2011 commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Changes

src/agents/auth-profiles/types.ts

src/agents/auth-profiles/usage.ts

src/agents/model-fallback.ts

src/agents/pi-embedded-runner/run.ts

src/auto-reply/reply/agent-runner-execution.ts

src/agents/auth-profiles/usage.test.ts

Cooldown Behavior After This PR

Testing

CI Note

Related Issues / PRs

Uh oh!

greptile-apps Bot commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Comments Outside Diff (1)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

kiranvk-2011 Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

altaywtf commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kiranvk-2011 commented Mar 18, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

kiranvk-2011 commented Mar 18, 2026

CI Failure Analysis — All Failures Pre-existing / Unrelated

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

kiranvk-2011 commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

kiranvk-2011 commented Mar 18, 2026 •

edited

Loading

`src/agents/auth-profiles/types.ts`

`src/agents/auth-profiles/usage.ts`

`src/agents/model-fallback.ts`

`src/agents/pi-embedded-runner/run.ts`

`src/auto-reply/reply/agent-runner-execution.ts`

`src/agents/auth-profiles/usage.test.ts`

greptile-apps Bot commented Mar 18, 2026 •

edited

Loading

altaywtf commented Mar 18, 2026 •

edited

Loading

kiranvk-2011 commented Mar 19, 2026 •

edited

Loading

CI Failure Analysis — All Pre-Existing on `main`