Skip to content

feat(active-memory): add timeout circuit breaker to skip recall after consecutive failures#74158

Merged
fabianwilliams merged 1 commit into
openclaw:mainfrom
yelog:feat/active-memory-circuit-breaker-74054
Apr 29, 2026
Merged

feat(active-memory): add timeout circuit breaker to skip recall after consecutive failures#74158
fabianwilliams merged 1 commit into
openclaw:mainfrom
yelog:feat/active-memory-circuit-breaker-74054

Conversation

@yelog

@yelog yelog commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add a per-agent/model circuit breaker to the Active Memory plugin that skips recall after consecutive timeouts, preventing token waste and stuck sessions.
  • Add two new configurable options: circuitBreakerMaxTimeouts (default: 3) and circuitBreakerCooldownMs (default: 60s).
  • Add 4 focused tests covering circuit breaker trip, reset, and config normalization.

Closes #74054

Problem

When using a slow-responding model (e.g., xiaomi/mimo-v2-flash with 15s+ response times) with the Active Memory plugin, API calls frequently timeout but the plugin continues to attempt recall on every eligible prompt. Since timeout results are deliberately not cached, each new message triggers another full recall attempt against the same slow model — consuming tokens, causing context overflow, and leaving sessions stuck in processing state for 10+ minutes.

The root cause: Active Memory has no plugin-owned mechanism to stop trying after repeated failures against the same provider/model.

Design

Circuit Breaker State

A module-level Map<string, CircuitBreakerEntry> keyed by agentId:provider/model tracks:

  • consecutiveTimeouts: count of consecutive timeout/timeout_partial results
  • lastTimeoutAt: timestamp of the last timeout

Behavior

  1. Before each recall attempt: Check if the circuit breaker is open (consecutive timeouts >= threshold AND within cooldown window). If open, return { status: "timeout", elapsedMs: 0 } immediately without calling the subagent.
  2. After timeout/timeout_partial: Increment the counter via recordCircuitBreakerTimeout().
  3. After ok/empty (success): Reset the counter via resetCircuitBreaker().
  4. After cooldown expires: The next attempt goes through (one retry allowed), and if it succeeds the breaker resets; if it times out again the breaker re-trips.

Config Options

Option Type Default Range
circuitBreakerMaxTimeouts integer 3 1–20
circuitBreakerCooldownMs integer 60000 5000–600000

Both are added to the plugin manifest schema with additionalProperties: false preserved, and include UI hints.

Files Changed

File Change
extensions/active-memory/index.ts Circuit breaker state, helpers, integration into maybeResolveActiveRecall, config normalization, __testing exports
extensions/active-memory/index.test.ts 4 new tests: breaker trips after consecutive timeouts, breaker resets on success after cooldown, config defaults, config clamping
extensions/active-memory/openclaw.plugin.json New config schema properties and UI hints

Test Plan

  • pnpm test extensions/active-memory — 99 tests (95 existing + 4 new), all pass
  • New test cases:
    • Circuit breaker trips after maxTimeouts consecutive timeouts (subagent not called again)
    • Circuit breaker resets after cooldown + successful recall
    • Config normalization produces correct defaults (3 / 60000)
    • Config clamping enforces min bounds (1 / 5000)

Notes

  • Circuit breaker is extension-owned, not a generic runner feature — this follows the architecture principle that extension-specific behavior stays in the extension.
  • The breaker keys are scoped by agentId:provider/model to avoid cross-agent or cross-model interference.
  • The existing behavior of not caching timeout results is preserved; the circuit breaker is an additional layer above the cache.
  • Backward compatible: new config options are optional with sensible defaults.

@aisle-research-bot

aisle-research-bot Bot commented Apr 29, 2026

Copy link
Copy Markdown

🔒 Aisle Security Analysis

We found 1 potential security issue(s) in this PR:

# Severity Title
1 🔵 Low Global circuit breaker state allows remote chat participants to disable Active Memory recall across sessions (availability/DoS)
1. 🔵 Global circuit breaker state allows remote chat participants to disable Active Memory recall across sessions (availability/DoS)
Property Value
Severity Low
CWE CWE-400
Location extensions/active-memory/index.ts:292-2402

Description

The Active Memory plugin introduces a module-level circuit breaker (timeoutCircuitBreaker) that is shared across all sessions and keyed only by agentId:provider/model.

  • Inputs: untrusted remote chat messages (e.g., webchat/group chat) influence the recall query in before_prompt_build and can repeatedly trigger slow/timeout subagent runs.
  • State: consecutive timeouts are recorded globally via recordCircuitBreakerTimeout(cbKey).
  • Impact: once the breaker opens, future recalls for any session using the same agent/model are skipped for circuitBreakerCooldownMs, returning a synthetic {status:"timeout", elapsedMs:0, summary:null} without attempting recall.

This enables a single remote participant to degrade the operator’s instance behavior (loss of memory recall) across unrelated sessions that share the same agent/model, by intentionally causing consecutive timeouts.

Vulnerable code:

const timeoutCircuitBreaker = new Map<string, CircuitBreakerEntry>();
function buildCircuitBreakerKey(agentId: string, provider?: string, model?: string): string {
  return `${agentId}:${provider ?? "unknown"}/${model ?? "unknown"}`;
}
...
if (isCircuitBreakerOpen(cbKey, ...)) {
  const result: ActiveRecallResult = { status: "timeout", elapsedMs: 0, summary: null };
  return result;
}
...
recordCircuitBreakerTimeout(cbKey);

Recommendation

Scope the circuit breaker to the session (or other per-conversation boundary) so one conversation cannot trip recall suppression for others.

Options:

  1. Include sessionKey (and/or channelId + messageProvider) in the circuit breaker key.
  2. Maintain separate breakers per session and additionally a global breaker only for clearly global failures (e.g., provider outage), not for content-driven timeouts.

Example (per-session key):

function buildCircuitBreakerKey(params: {
  agentId: string;
  sessionKey?: string;
  messageProvider?: string;
  channelId?: string;
  provider?: string;
  model?: string;
}) {
  const sessionPart = params.sessionKey ?? `${params.messageProvider ?? "unknown"}:${params.channelId ?? "unknown"}`;
  return `${params.agentId}:${sessionPart}:${params.provider ?? "unknown"}/${params.model ?? "unknown"}`;
}

Also consider returning a distinct status like "skipped" instead of "timeout" to avoid downstream logic treating the result as an actual timeout event.


Analyzed PR: #74158 at commit e061b01

Last updated on: 2026-04-29T06:42:32Z

@greptile-apps

greptile-apps Bot commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a per-agent/model circuit breaker to the Active Memory plugin that skips recall after circuitBreakerMaxTimeouts consecutive timeouts, preventing repeated token waste against slow-responding models. The implementation is well-scoped, backward-compatible, and covered by four focused new tests.

Confidence Score: 4/5

Safe to merge; only minor style/design P2 observations, no functional defects.

All findings are P2. The logic is correct, tests cover the key paths, and the feature is additive with sensible defaults.

No files require special attention.

Comments Outside Diff (1)

  1. extensions/active-memory/index.ts, line 2404-2419 (link)

    P2 unavailable errors leave circuit breaker counter unchanged

    Non-timeout errors (network failures, parse errors, etc.) reach the status: "unavailable" branch without calling either recordCircuitBreakerTimeout or resetCircuitBreaker. The counter is silently frozen at its previous value. This is defensible — a crash isn't the same as a slow model — but a run of intermixed timeouts and errors will never reset the counter either. A comment explaining the intentional non-interaction would prevent future confusion.

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: extensions/active-memory/index.ts
    Line: 2404-2419
    
    Comment:
    **`unavailable` errors leave circuit breaker counter unchanged**
    
    Non-timeout errors (network failures, parse errors, etc.) reach the `status: "unavailable"` branch without calling either `recordCircuitBreakerTimeout` or `resetCircuitBreaker`. The counter is silently frozen at its previous value. This is defensible — a crash isn't the same as a slow model — but a run of intermixed timeouts and errors will never reset the counter either. A comment explaining the intentional non-interaction would prevent future confusion.
    
    How can I resolve this? If you propose a fix, please make it concise.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: extensions/active-memory/index.ts
Line: 297-306

Comment:
**Side-effecting predicate mutates state on cooldown expiry**

`isCircuitBreakerOpen` deletes the map entry as a side-effect of checking. In the presence of concurrent async calls, both callers can pass the check: Call A deletes the entry and returns `false`; while Call A's `await` is pending, Call B finds the entry gone and also returns `false`, letting two "trial" requests through instead of one. For a best-effort plugin circuit breaker this is tolerable, but extracting the deletion into the call-site (or returning an enum like `open | half-open | closed`) would make the half-open transition explicit and easier to reason about.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: extensions/active-memory/index.ts
Line: 2404-2419

Comment:
**`unavailable` errors leave circuit breaker counter unchanged**

Non-timeout errors (network failures, parse errors, etc.) reach the `status: "unavailable"` branch without calling either `recordCircuitBreakerTimeout` or `resetCircuitBreaker`. The counter is silently frozen at its previous value. This is defensible — a crash isn't the same as a slow model — but a run of intermixed timeouts and errors will never reset the counter either. A comment explaining the intentional non-interaction would prevent future confusion.

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "feat(active-memory): add timeout circuit..." | Re-trigger Greptile

Comment on lines +297 to +306

function isCircuitBreakerOpen(key: string, maxTimeouts: number, cooldownMs: number): boolean {
const entry = timeoutCircuitBreaker.get(key);
if (!entry || entry.consecutiveTimeouts < maxTimeouts) {
return false;
}
if (Date.now() - entry.lastTimeoutAt >= cooldownMs) {
// Cooldown expired — reset and allow one attempt through.
timeoutCircuitBreaker.delete(key);
return false;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Side-effecting predicate mutates state on cooldown expiry

isCircuitBreakerOpen deletes the map entry as a side-effect of checking. In the presence of concurrent async calls, both callers can pass the check: Call A deletes the entry and returns false; while Call A's await is pending, Call B finds the entry gone and also returns false, letting two "trial" requests through instead of one. For a best-effort plugin circuit breaker this is tolerable, but extracting the deletion into the call-site (or returning an enum like open | half-open | closed) would make the half-open transition explicit and easier to reason about.

Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/active-memory/index.ts
Line: 297-306

Comment:
**Side-effecting predicate mutates state on cooldown expiry**

`isCircuitBreakerOpen` deletes the map entry as a side-effect of checking. In the presence of concurrent async calls, both callers can pass the check: Call A deletes the entry and returns `false`; while Call A's `await` is pending, Call B finds the entry gone and also returns `false`, letting two "trial" requests through instead of one. For a best-effort plugin circuit breaker this is tolerable, but extracting the deletion into the call-site (or returning an enum like `open | half-open | closed`) would make the half-open transition explicit and easier to reason about.

How can I resolve this? If you propose a fix, please make it concise.

@clawsweeper

clawsweeper Bot commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge.

Keep this PR open. Current main still lacks the proposed Active Memory consecutive-timeout circuit breaker and config knobs, while the linked #74054 report remains plausible because timeout results are intentionally not cached and repeated eligible prompts can launch new recall attempts. The PR is a relevant implementation candidate, but it needs maintainer review for breaker scope/security plus docs/changelog coverage before merge.

Maintainer follow-up before merge:

Keep this PR open for maintainer review. The best path is to finish an Active Memory-owned circuit breaker in the plugin, decide whether breaker state must include session/channel scope, document and changelog the new config knobs, then validate with targeted Active Memory tests and the changed gate before landing.

Best possible solution:

Keep this PR open for maintainer review. The best path is to finish an Active Memory-owned circuit breaker in the plugin, decide whether breaker state must include session/channel scope, document and changelog the new config knobs, then validate with targeted Active Memory tests and the changed gate before landing.

Acceptance criteria:

  • pnpm test extensions/active-memory
  • pnpm check:changed in Testbox before handoff/merge

What I checked:

  • Current config has no breaker knobs: ActiveRecallPluginConfig and ResolvedActiveRecallPluginConfig include timeoutMs and cacheTtlMs but no circuitBreakerMaxTimeouts or circuitBreakerCooldownMs on current main. (extensions/active-memory/index.ts:91, dc810437e767)
  • Runtime normalization has no breaker defaults: normalizePluginConfig clamps timeoutMs and cacheTtlMs only; there is no consecutive-timeout state or cooldown configuration on current main. (extensions/active-memory/index.ts:699, dc810437e767)
  • Recall path is per-attempt bounded, not circuit-broken: maybeResolveActiveRecall checks the cache, starts a watchdog/Promise.race recall, returns timeout or timeout_partial on timeout, and caches only cacheable success/empty results; there is no cross-prompt consecutive-timeout state before starting recall. (extensions/active-memory/index.ts:2133, dc810437e767)
  • Timeouts are deliberately not cached: The existing regression test calls the same timeout-triggering prompt twice, expects two session-store updates, and asserts no cached log line, supporting the linked repeated-timeout behavior. (extensions/active-memory/index.test.ts:2031, dc810437e767)
  • Manifest rejects unknown fields and lacks proposed knobs: The Active Memory manifest has additionalProperties: false and lists timeoutMs/cacheTtlMs but not the proposed circuit-breaker fields on current main. (extensions/active-memory/openclaw.plugin.json:8, dc810437e767)
  • Docs lack circuit-breaker configuration: The public Active Memory docs list timeoutMs and cacheTtlMs and suggest lowering timeoutMs for slow recall; they do not document circuitBreakerMaxTimeouts or circuitBreakerCooldownMs. Public docs: docs/concepts/active-memory.md. (docs/concepts/active-memory.md:561, dc810437e767)

Likely related people:

  • vincentkoc: Remote commit metadata shows df9d26e changed the Active Memory prompt hook timeout and tests shortly before this PR, and b85edb3 backfilled the changelog entry for that Active Memory timeout behavior. (role: recent timeout-surface maintainer; confidence: high; commits: df9d26eb4335, b85edb3f0cf6; files: extensions/active-memory/index.ts, extensions/active-memory/index.test.ts, CHANGELOG.md)
  • Takhoffman: Remote commit metadata identifies b83726d as the original Active Memory plugin feature PR, adding the plugin runtime, tests, manifest, docs, and changelog surface that this PR extends. (role: original feature introducer; confidence: medium; commits: b83726d13e33; files: extensions/active-memory/index.ts, extensions/active-memory/index.test.ts, extensions/active-memory/openclaw.plugin.json)
  • luyao618: Remote commit metadata shows fbefbf0 added the Active Memory Promise.race hard-deadline behavior and tests, which is adjacent to the repeated-timeout circuit-breaker request. (role: adjacent hard-timeout contributor; confidence: medium; commits: fbefbf05bd74; files: extensions/active-memory/index.ts, extensions/active-memory/index.test.ts)

Remaining risk / open question:

  • The PR's agent/model-scoped module-level breaker may let one session suppress recall for other sessions using the same agent/model during the cooldown window.
  • The new user-facing config knobs need docs and changelog coverage because the manifest is strict and operators need to understand the degradation behavior.
  • Returning synthetic status=timeout for a skipped recall may blur actual provider timeout telemetry unless maintainers decide that status is intentional.

Codex review notes: model gpt-5.5, reasoning high; reviewed against dc810437e767.

@fabianwilliams fabianwilliams left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Circuit breaker for slow-recall providers is the right shape — counts consecutive timeouts per agent/model, trips at circuitBreakerMaxTimeouts (default 3), self-heals after circuitBreakerCooldownMs (default 60s). Sane defaults that won't trip on transient blips but will protect against a stuck provider taking down a session loop. 4 tests cover trip/reset/config-normalization. Particularly liked the use of runEmbeddedPiAgent.mockImplementation(async () => await new Promise<never>(() => {})) to deterministically force the timeout path.

@fabianwilliams
fabianwilliams merged commit 89cd2b6 into openclaw:main Apr 29, 2026
72 of 73 checks passed
lxe pushed a commit to lxe/openclaw that referenced this pull request May 6, 2026
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 9, 2026
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
jameslcowan pushed a commit to jameslcowan/openclaw that referenced this pull request Jun 2, 2026
sablehead pushed a commit to sablehead/openclaw that referenced this pull request Jun 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: active-memory plugin infinite retry on API timeout causes token waste

2 participants