feat(active-memory): add timeout circuit breaker to skip recall after consecutive failures#74158
Conversation
… consecutive failures (openclaw#74054)
🔒 Aisle Security AnalysisWe found 1 potential security issue(s) in this PR:
1. 🔵 Global circuit breaker state allows remote chat participants to disable Active Memory recall across sessions (availability/DoS)
DescriptionThe Active Memory plugin introduces a module-level circuit breaker (
This enables a single remote participant to degrade the operator’s instance behavior (loss of memory recall) across unrelated sessions that share the same agent/model, by intentionally causing consecutive timeouts. Vulnerable code: const timeoutCircuitBreaker = new Map<string, CircuitBreakerEntry>();
function buildCircuitBreakerKey(agentId: string, provider?: string, model?: string): string {
return `${agentId}:${provider ?? "unknown"}/${model ?? "unknown"}`;
}
...
if (isCircuitBreakerOpen(cbKey, ...)) {
const result: ActiveRecallResult = { status: "timeout", elapsedMs: 0, summary: null };
return result;
}
...
recordCircuitBreakerTimeout(cbKey);RecommendationScope the circuit breaker to the session (or other per-conversation boundary) so one conversation cannot trip recall suppression for others. Options:
Example (per-session key): function buildCircuitBreakerKey(params: {
agentId: string;
sessionKey?: string;
messageProvider?: string;
channelId?: string;
provider?: string;
model?: string;
}) {
const sessionPart = params.sessionKey ?? `${params.messageProvider ?? "unknown"}:${params.channelId ?? "unknown"}`;
return `${params.agentId}:${sessionPart}:${params.provider ?? "unknown"}/${params.model ?? "unknown"}`;
}Also consider returning a distinct status like Analyzed PR: #74158 at commit Last updated on: 2026-04-29T06:42:32Z |
Greptile SummaryThis PR adds a per-agent/model circuit breaker to the Active Memory plugin that skips recall after Confidence Score: 4/5Safe to merge; only minor style/design P2 observations, no functional defects. All findings are P2. The logic is correct, tests cover the key paths, and the feature is additive with sensible defaults. No files require special attention.
|
|
|
||
| function isCircuitBreakerOpen(key: string, maxTimeouts: number, cooldownMs: number): boolean { | ||
| const entry = timeoutCircuitBreaker.get(key); | ||
| if (!entry || entry.consecutiveTimeouts < maxTimeouts) { | ||
| return false; | ||
| } | ||
| if (Date.now() - entry.lastTimeoutAt >= cooldownMs) { | ||
| // Cooldown expired — reset and allow one attempt through. | ||
| timeoutCircuitBreaker.delete(key); | ||
| return false; |
There was a problem hiding this comment.
Side-effecting predicate mutates state on cooldown expiry
isCircuitBreakerOpen deletes the map entry as a side-effect of checking. In the presence of concurrent async calls, both callers can pass the check: Call A deletes the entry and returns false; while Call A's await is pending, Call B finds the entry gone and also returns false, letting two "trial" requests through instead of one. For a best-effort plugin circuit breaker this is tolerable, but extracting the deletion into the call-site (or returning an enum like open | half-open | closed) would make the half-open transition explicit and easier to reason about.
Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/active-memory/index.ts
Line: 297-306
Comment:
**Side-effecting predicate mutates state on cooldown expiry**
`isCircuitBreakerOpen` deletes the map entry as a side-effect of checking. In the presence of concurrent async calls, both callers can pass the check: Call A deletes the entry and returns `false`; while Call A's `await` is pending, Call B finds the entry gone and also returns `false`, letting two "trial" requests through instead of one. For a best-effort plugin circuit breaker this is tolerable, but extracting the deletion into the call-site (or returning an enum like `open | half-open | closed`) would make the half-open transition explicit and easier to reason about.
How can I resolve this? If you propose a fix, please make it concise.|
Codex review: needs maintainer review before merge. Keep this PR open. Current main still lacks the proposed Active Memory consecutive-timeout circuit breaker and config knobs, while the linked #74054 report remains plausible because timeout results are intentionally not cached and repeated eligible prompts can launch new recall attempts. The PR is a relevant implementation candidate, but it needs maintainer review for breaker scope/security plus docs/changelog coverage before merge. Maintainer follow-up before merge: Keep this PR open for maintainer review. The best path is to finish an Active Memory-owned circuit breaker in the plugin, decide whether breaker state must include session/channel scope, document and changelog the new config knobs, then validate with targeted Active Memory tests and the changed gate before landing. Best possible solution: Keep this PR open for maintainer review. The best path is to finish an Active Memory-owned circuit breaker in the plugin, decide whether breaker state must include session/channel scope, document and changelog the new config knobs, then validate with targeted Active Memory tests and the changed gate before landing. Acceptance criteria:
What I checked:
Likely related people:
Remaining risk / open question:
Codex review notes: model gpt-5.5, reasoning high; reviewed against dc810437e767. |
fabianwilliams
left a comment
There was a problem hiding this comment.
LGTM. Circuit breaker for slow-recall providers is the right shape — counts consecutive timeouts per agent/model, trips at circuitBreakerMaxTimeouts (default 3), self-heals after circuitBreakerCooldownMs (default 60s). Sane defaults that won't trip on transient blips but will protect against a stuck provider taking down a session loop. 4 tests cover trip/reset/config-normalization. Particularly liked the use of runEmbeddedPiAgent.mockImplementation(async () => await new Promise<never>(() => {})) to deterministically force the timeout path.
… consecutive failures (openclaw#74054) (openclaw#74158)
… consecutive failures (openclaw#74054) (openclaw#74158)
… consecutive failures (openclaw#74054) (openclaw#74158)
… consecutive failures (openclaw#74054) (openclaw#74158)
… consecutive failures (openclaw#74054) (openclaw#74158)
Summary
circuitBreakerMaxTimeouts(default: 3) andcircuitBreakerCooldownMs(default: 60s).Closes #74054
Problem
When using a slow-responding model (e.g.,
xiaomi/mimo-v2-flashwith 15s+ response times) with the Active Memory plugin, API calls frequently timeout but the plugin continues to attempt recall on every eligible prompt. Since timeout results are deliberately not cached, each new message triggers another full recall attempt against the same slow model — consuming tokens, causing context overflow, and leaving sessions stuck inprocessingstate for 10+ minutes.The root cause: Active Memory has no plugin-owned mechanism to stop trying after repeated failures against the same provider/model.
Design
Circuit Breaker State
A module-level
Map<string, CircuitBreakerEntry>keyed byagentId:provider/modeltracks:consecutiveTimeouts: count of consecutive timeout/timeout_partial resultslastTimeoutAt: timestamp of the last timeoutBehavior
{ status: "timeout", elapsedMs: 0 }immediately without calling the subagent.recordCircuitBreakerTimeout().resetCircuitBreaker().Config Options
circuitBreakerMaxTimeoutscircuitBreakerCooldownMsBoth are added to the plugin manifest schema with
additionalProperties: falsepreserved, and include UI hints.Files Changed
extensions/active-memory/index.tsmaybeResolveActiveRecall, config normalization,__testingexportsextensions/active-memory/index.test.tsextensions/active-memory/openclaw.plugin.jsonTest Plan
pnpm test extensions/active-memory— 99 tests (95 existing + 4 new), all passmaxTimeoutsconsecutive timeouts (subagent not called again)Notes
agentId:provider/modelto avoid cross-agent or cross-model interference.