Skip to content

fix(compaction): add circuit breaker for retry loops#58857

Closed
aaron-he-zhu wants to merge 1 commit into
openclaw:mainfrom
aaron-he-zhu:fix/compaction-circuit-breaker
Closed

fix(compaction): add circuit breaker for retry loops#58857
aaron-he-zhu wants to merge 1 commit into
openclaw:mainfrom
aaron-he-zhu:fix/compaction-circuit-breaker

Conversation

@aaron-he-zhu

Copy link
Copy Markdown
Contributor

Fixes #58838

Summary

When compaction repeatedly fails (model errors, timeouts, malformed output), retryAsync burns tokens without progress. Add CompactionCircuitBreaker that stops attempts after N consecutive failures and resumes after a cooldown period.

  • CompactionCircuitBreaker class with closed/open/half-open states
  • Default: opens after 3 consecutive failures, resets after 60s cooldown
  • Wire into summarizeChunks() as optional parameter (backward compatible)
  • When circuit is open, falls back to previousSummary instead of retrying

Test plan

  • State transitions: closed → open → half-open → closed/open
  • Counter reset on mid-sequence success
  • Default config values (3 failures, 60s cooldown)
  • reset() clears all state
  • Fake timers for deterministic time-based tests
  • Existing compaction tests pass (10 tests)
  • tsgo --noEmit clean, oxlint clean, oxfmt --check clean

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: M labels Apr 1, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3663b3bebb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/compaction.ts
Comment on lines +256 to +258
const { circuitBreaker } = params;
if (circuitBreaker && !circuitBreaker.canAttempt()) {
log.warn("Compaction skipped — circuit breaker is open");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Wire circuit breaker into the compaction call path

The new breaker logic is effectively unreachable in production because summarizeChunks only consults it when params.circuitBreaker is set, but no caller ever supplies one. In the same file, summarizeWithFallback does not accept a circuitBreaker field (src/agents/compaction.ts:341-353) and both invocations call summarizeChunks with params/spread params that therefore never contain a breaker (src/agents/compaction.ts:362, 389-392). This means canAttempt()/recordFailure()/recordSuccess() never run, so repeated compaction failures still retry indefinitely and continue burning tokens.

Useful? React with 👍 / 👎.

@greptile-apps

greptile-apps Bot commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR introduces a CompactionCircuitBreaker class to prevent token waste when compaction repeatedly fails, and wires it into summarizeChunks() as an optional backward-compatible parameter. The class itself is well-implemented with correct state transitions, deterministic time-based tests, and sensible defaults (3 failures, 60s cooldown).

However, there are two correctness defects that leave the feature non-functional:

  • Circuit breaker is unreachable (P1): summarizeChunks is a private unexported function. Neither public entry point (summarizeWithFallback, summarizeInStages) accepts or forwards a circuitBreaker parameter, so circuitBreaker is always undefined inside summarizeChunks. The entire circuit-breaking path (canAttempt, recordSuccess, recordFailure) is dead code at runtime. The CompactionCircuitBreaker type needs to be added to summarizeWithFallback's (and summarizeInStages's) params and threaded through, and compaction-safeguard.ts needs to own an instance and pass it in.
  • AbortError incorrectly trips the circuit (P1): The catch block in summarizeChunks calls recordFailure() for any exception, including user-initiated AbortError cancellations. This would allow normal user cancellations to advance the failure counter and potentially open the circuit, even though shouldRetry already correctly excludes AbortError from retry logic. The failure recording should mirror the same guard.

Confidence Score: 4/5

Not safe to merge as-is: the circuit breaker feature is entirely dead code and AbortError would incorrectly trip it once wired up.

Two P1 defects are present. The first (dead code / missing public API threading) means the PR's stated goal — preventing token waste from retry loops — is not actually achieved. The second (AbortError tripping the circuit) would cause incorrect behavior once the first is fixed. No production code paths are broken by the current state, but the feature does not work at all and the fix requires non-trivial threading changes across multiple functions and their callers.

src/agents/compaction.ts needs the circuitBreaker parameter threaded through summarizeWithFallback and summarizeInStages, and the catch block needs an AbortError guard. src/agents/pi-hooks/compaction-safeguard.ts (unchanged in this PR) will need to instantiate and own the CompactionCircuitBreaker.

Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/agents/compaction.ts
Line: 250

Comment:
**Circuit breaker is dead code — never reachable through the public API**

`summarizeChunks` is a private (unexported) function. Its `circuitBreaker` parameter can only be populated by callers within `compaction.ts`, but both callers — `summarizeWithFallback` (line 362, 389) and the spread in `summarizeInStages` (via `summarizeWithFallback`) — never pass a `circuitBreaker`. Neither public function accepts this parameter in their own `params` type.

The actual production call path is:

```
compaction-safeguard.ts
  → summarizeInStages()
  → summarizeWithFallback()
  → summarizeChunks(params)   ← params never contains circuitBreaker
```

As a result, `circuitBreaker` is always `undefined` inside `summarizeChunks`, making every line of circuit-breaker logic in that function (lines 256–259, 295, 297–298) unreachable. The import of `CompactionCircuitBreaker` in `compaction.ts` is also unused at runtime.

To actually activate this feature, `circuitBreaker?` needs to be added to the params type of `summarizeWithFallback` (and `summarizeInStages`), and then forwarded when `summarizeChunks` is called:

```ts
// In summarizeWithFallback's params type:
circuitBreaker?: CompactionCircuitBreaker;

// When calling summarizeChunks:
return await summarizeChunks(params);  // params now carries circuitBreaker
```

`compaction-safeguard.ts` would then need to create and own a `CompactionCircuitBreaker` instance and pass it through.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/agents/compaction.ts
Line: 296-298

Comment:
**AbortError incorrectly trips the circuit breaker**

The `catch` block unconditionally calls `circuitBreaker?.recordFailure()` for every thrown error — including `AbortError`. When the user cancels compaction via `AbortSignal`, `retryAsync` propagates the error without retrying (because `shouldRetry` returns false for `AbortError`), and this catch block would then count the cancellation as a compaction failure and advance toward opening the circuit.

The circuit breaker is intended to guard against model errors, timeouts, and malformed output — not against user-initiated cancellation. A cancellation should leave the failure counter unchanged:

```ts
} catch (err) {
    if (!(err instanceof Error && err.name === "AbortError")) {
        circuitBreaker?.recordFailure();
    }
    throw err;
}
```

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "fix(compaction): add circuit breaker for..." | Re-trigger Greptile

Comment thread src/agents/compaction.ts
customInstructions?: string;
summarizationInstructions?: CompactionSummarizationInstructions;
previousSummary?: string;
circuitBreaker?: CompactionCircuitBreaker;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Circuit breaker is dead code — never reachable through the public API

summarizeChunks is a private (unexported) function. Its circuitBreaker parameter can only be populated by callers within compaction.ts, but both callers — summarizeWithFallback (line 362, 389) and the spread in summarizeInStages (via summarizeWithFallback) — never pass a circuitBreaker. Neither public function accepts this parameter in their own params type.

The actual production call path is:

compaction-safeguard.ts
  → summarizeInStages()
  → summarizeWithFallback()
  → summarizeChunks(params)   ← params never contains circuitBreaker

As a result, circuitBreaker is always undefined inside summarizeChunks, making every line of circuit-breaker logic in that function (lines 256–259, 295, 297–298) unreachable. The import of CompactionCircuitBreaker in compaction.ts is also unused at runtime.

To actually activate this feature, circuitBreaker? needs to be added to the params type of summarizeWithFallback (and summarizeInStages), and then forwarded when summarizeChunks is called:

// In summarizeWithFallback's params type:
circuitBreaker?: CompactionCircuitBreaker;

// When calling summarizeChunks:
return await summarizeChunks(params);  // params now carries circuitBreaker

compaction-safeguard.ts would then need to create and own a CompactionCircuitBreaker instance and pass it through.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/compaction.ts
Line: 250

Comment:
**Circuit breaker is dead code — never reachable through the public API**

`summarizeChunks` is a private (unexported) function. Its `circuitBreaker` parameter can only be populated by callers within `compaction.ts`, but both callers — `summarizeWithFallback` (line 362, 389) and the spread in `summarizeInStages` (via `summarizeWithFallback`) — never pass a `circuitBreaker`. Neither public function accepts this parameter in their own `params` type.

The actual production call path is:

```
compaction-safeguard.ts
  → summarizeInStages()
  → summarizeWithFallback()
  → summarizeChunks(params)   ← params never contains circuitBreaker
```

As a result, `circuitBreaker` is always `undefined` inside `summarizeChunks`, making every line of circuit-breaker logic in that function (lines 256–259, 295, 297–298) unreachable. The import of `CompactionCircuitBreaker` in `compaction.ts` is also unused at runtime.

To actually activate this feature, `circuitBreaker?` needs to be added to the params type of `summarizeWithFallback` (and `summarizeInStages`), and then forwarded when `summarizeChunks` is called:

```ts
// In summarizeWithFallback's params type:
circuitBreaker?: CompactionCircuitBreaker;

// When calling summarizeChunks:
return await summarizeChunks(params);  // params now carries circuitBreaker
```

`compaction-safeguard.ts` would then need to create and own a `CompactionCircuitBreaker` instance and pass it through.

How can I resolve this? If you propose a fix, please make it concise.

Comment thread src/agents/compaction.ts
Comment on lines +296 to +298
} catch (err) {
circuitBreaker?.recordFailure();
throw err;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 AbortError incorrectly trips the circuit breaker

The catch block unconditionally calls circuitBreaker?.recordFailure() for every thrown error — including AbortError. When the user cancels compaction via AbortSignal, retryAsync propagates the error without retrying (because shouldRetry returns false for AbortError), and this catch block would then count the cancellation as a compaction failure and advance toward opening the circuit.

The circuit breaker is intended to guard against model errors, timeouts, and malformed output — not against user-initiated cancellation. A cancellation should leave the failure counter unchanged:

} catch (err) {
    if (!(err instanceof Error && err.name === "AbortError")) {
        circuitBreaker?.recordFailure();
    }
    throw err;
}
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/compaction.ts
Line: 296-298

Comment:
**AbortError incorrectly trips the circuit breaker**

The `catch` block unconditionally calls `circuitBreaker?.recordFailure()` for every thrown error — including `AbortError`. When the user cancels compaction via `AbortSignal`, `retryAsync` propagates the error without retrying (because `shouldRetry` returns false for `AbortError`), and this catch block would then count the cancellation as a compaction failure and advance toward opening the circuit.

The circuit breaker is intended to guard against model errors, timeouts, and malformed output — not against user-initiated cancellation. A cancellation should leave the failure counter unchanged:

```ts
} catch (err) {
    if (!(err instanceof Error && err.name === "AbortError")) {
        circuitBreaker?.recordFailure();
    }
    throw err;
}
```

How can I resolve this? If you propose a fix, please make it concise.

@SonicBotMan

This comment was marked as spam.

When compaction repeatedly fails (model errors, timeouts, malformed output),
retryAsync burns tokens without progress. Add CompactionCircuitBreaker with
closed/open/half-open states that stops attempts after N consecutive failures
and resumes after a cooldown period.

Wire into summarizeChunks() as an optional parameter — callers can pass a
shared circuit breaker instance to prevent cascading retry storms.
@aaron-he-zhu
aaron-he-zhu force-pushed the fix/compaction-circuit-breaker branch from 3663b3b to a64e7ad Compare April 1, 2026 08:36
@aaron-he-zhu

Copy link
Copy Markdown
Contributor Author

Closing — after reviewing how recent fixes have been structured on main, I think this approach (separate file + class) doesn't match the codebase style well. Will revisit with a more targeted approach if compaction retry issues come up again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(compaction): retry loop can burn tokens when summarizer model is unavailable

2 participants