Skip to content

fix(compaction): add circuit breaker for retry loops#58846

Closed
aaron-he-zhu wants to merge 1 commit into
openclaw:mainfrom
aaron-he-zhu:fix/compaction-circuit-breaker
Closed

fix(compaction): add circuit breaker for retry loops#58846
aaron-he-zhu wants to merge 1 commit into
openclaw:mainfrom
aaron-he-zhu:fix/compaction-circuit-breaker

Conversation

@aaron-he-zhu

Copy link
Copy Markdown
Contributor

Fixes #58838

Summary

When compaction repeatedly fails (model errors, timeouts, malformed output), retryAsync burns tokens without progress. Add CompactionCircuitBreaker that stops attempts after N consecutive failures and resumes after a cooldown period.

  • CompactionCircuitBreaker class with closed/open/half-open states
  • Default: opens after 3 consecutive failures, resets after 60s cooldown
  • Wire into summarizeChunks() as optional parameter (backward compatible)
  • When circuit is open, falls back to previousSummary instead of retrying

Test plan

  • State transitions: closed → open (after N failures)
  • State transitions: open → half-open (after cooldown)
  • State transitions: half-open → closed (on success) / open (on failure)
  • Counter reset on mid-sequence success
  • Default config values (3 failures, 60s cooldown)
  • reset() clears all state
  • Fake timers for deterministic time-based tests
  • Existing compaction tests pass (10 tests)

When compaction repeatedly fails (model errors, timeouts, malformed output),
retryAsync burns tokens without progress. Add CompactionCircuitBreaker with
closed/open/half-open states that stops attempts after N consecutive failures
and resumes after a cooldown period.

Wire into summarizeChunks() as an optional parameter — callers can pass a
shared circuit breaker instance to prevent cascading retry storms.
@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: M labels Apr 1, 2026
@greptile-apps

greptile-apps Bot commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR introduces a CompactionCircuitBreaker class to prevent runaway token consumption when compaction retries repeatedly fail. The class itself is well-designed — state machine, cooldown timer, and config defaults are all correct — and the unit tests are thorough. The wiring into summarizeChunks (try/catch around the chunk loop, fallback to previousSummary when the breaker is open) is also logically sound.

However, there is one blocking issue:

  • Circuit breaker is unreachable from the public API. summarizeChunks is a module-private function. The two public entry points, summarizeWithFallback and summarizeInStages, do not include circuitBreaker in their parameter types and pass params directly into summarizeChunks without augmenting it. As a result, circuitBreaker will always be undefined at runtime — the feature exists but cannot be activated by any external caller. Both public functions need circuitBreaker?: CompactionCircuitBreaker added to their parameter objects and forwarded through.

Additionally:

  • recordFailure() uses >= maxFailures to log "OPENED", so a half-open → open re-transition emits a second spurious warning (the count is already above the threshold). Using === maxFailures would restrict the log to the initial trip only.

Confidence Score: 3/5

Not safe to merge as-is — the circuit breaker feature is inaccessible via the public API and effectively dead code.

There is one P1 functional defect: the circuitBreaker parameter is wired only into the private summarizeChunks function and is always undefined because neither summarizeWithFallback nor summarizeInStages accepts or forwards it. The PR's described feature — stopping runaway retries — does not actually work. This needs to be fixed before the change has any effect in production.

src/agents/compaction.ts — the public API functions summarizeWithFallback and summarizeInStages need circuitBreaker? added to their parameter types and forwarded to the internal summarizeChunks calls.

Comments Outside Diff (1)

  1. src/agents/compaction.ts, line 341-353 (link)

    P1 Circuit breaker unreachable via public API

    summarizeChunks is a module-private (unexported) function, so the new circuitBreaker? parameter can never be populated by external callers. Both public entry points — summarizeWithFallback and summarizeInStages — do not include circuitBreaker in their parameter types, and both call summarizeChunks(params) where params comes from their own unaugmented type. At runtime circuitBreaker is always undefined inside summarizeChunks, making the entire feature inert.

    To make the circuit breaker usable, summarizeWithFallback (and summarizeInStages) need to accept and forward it:

    export async function summarizeWithFallback(params: {
      // ... existing fields ...
      circuitBreaker?: CompactionCircuitBreaker;
    }): Promise<string> {

    Then all three internal calls to summarizeChunks (params direct spread, { ...params, messages: smallMessages }, etc.) automatically pick it up. The same addition is needed for summarizeInStages so it passes through when it delegates to summarizeWithFallback.

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: src/agents/compaction.ts
    Line: 341-353
    
    Comment:
    **Circuit breaker unreachable via public API**
    
    `summarizeChunks` is a module-private (unexported) function, so the new `circuitBreaker?` parameter can never be populated by external callers. Both public entry points — `summarizeWithFallback` and `summarizeInStages` — do not include `circuitBreaker` in their parameter types, and both call `summarizeChunks(params)` where `params` comes from their own unaugmented type. At runtime `circuitBreaker` is always `undefined` inside `summarizeChunks`, making the entire feature inert.
    
    To make the circuit breaker usable, `summarizeWithFallback` (and `summarizeInStages`) need to accept and forward it:
    
    ```ts
    export async function summarizeWithFallback(params: {
      // ... existing fields ...
      circuitBreaker?: CompactionCircuitBreaker;
    }): Promise<string> {
    ```
    
    Then all three internal calls to `summarizeChunks` (`params` direct spread, `{ ...params, messages: smallMessages }`, etc.) automatically pick it up. The same addition is needed for `summarizeInStages` so it passes through when it delegates to `summarizeWithFallback`.
    
    How can I resolve this? If you propose a fix, please make it concise.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/agents/compaction.ts
Line: 341-353

Comment:
**Circuit breaker unreachable via public API**

`summarizeChunks` is a module-private (unexported) function, so the new `circuitBreaker?` parameter can never be populated by external callers. Both public entry points — `summarizeWithFallback` and `summarizeInStages` — do not include `circuitBreaker` in their parameter types, and both call `summarizeChunks(params)` where `params` comes from their own unaugmented type. At runtime `circuitBreaker` is always `undefined` inside `summarizeChunks`, making the entire feature inert.

To make the circuit breaker usable, `summarizeWithFallback` (and `summarizeInStages`) need to accept and forward it:

```ts
export async function summarizeWithFallback(params: {
  // ... existing fields ...
  circuitBreaker?: CompactionCircuitBreaker;
}): Promise<string> {
```

Then all three internal calls to `summarizeChunks` (`params` direct spread, `{ ...params, messages: smallMessages }`, etc.) automatically pick it up. The same addition is needed for `summarizeInStages` so it passes through when it delegates to `summarizeWithFallback`.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/agents/compaction-circuit-breaker.ts
Line: 82-92

Comment:
**Spurious "OPENED" log on half-open → open re-transition**

When the circuit is already in the half-open state (`consecutiveFailures == maxFailures`) and `recordFailure()` is called, `consecutiveFailures` is incremented to `maxFailures + 1`. The guard `consecutiveFailures >= maxFailures` fires again and emits another "OPENED after N consecutive failures" warning, even though the circuit was previously opened and is simply re-opening after a probe failure. This can be confusing in logs and slightly inflates the reported failure count.

```ts
  recordFailure(): void {
    this.consecutiveFailures += 1;
    this.lastFailureTime = Date.now();

    if (this.consecutiveFailures === this.maxFailures) {   // was >=
      log.warn(
        `Compaction circuit breaker OPENED after ${this.consecutiveFailures} consecutive failures — ` +
          `pausing compaction for ${Math.ceil(this.resetAfterMs / 1000)}s`,
      );
    }
  }
```

Changing `>=` to `===` emits the log only on the exact transition, not on repeated re-openings from half-open.

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "fix(compaction): add circuit breaker for..." | Re-trigger Greptile

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3663b3bebb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/compaction.ts
customInstructions?: string;
summarizationInstructions?: CompactionSummarizationInstructions;
previousSummary?: string;
circuitBreaker?: CompactionCircuitBreaker;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Wire circuit breaker through compaction entrypoints

circuitBreaker was added only to the private summarizeChunks params, but the exported entrypoints still cannot pass it (summarizeWithFallback/summarizeInStages do not accept this field, and existing call sites continue passing the old shape). In practice this leaves circuitBreaker undefined on every normal path, so the new open-circuit guard never triggers and compaction retries keep running as before under repeated failures.

Useful? React with 👍 / 👎.

Comment on lines +82 to +92
recordFailure(): void {
this.consecutiveFailures += 1;
this.lastFailureTime = Date.now();

if (this.consecutiveFailures >= this.maxFailures) {
log.warn(
`Compaction circuit breaker OPENED after ${this.consecutiveFailures} consecutive failures — ` +
`pausing compaction for ${Math.ceil(this.resetAfterMs / 1000)}s`,
);
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Spurious "OPENED" log on half-open → open re-transition

When the circuit is already in the half-open state (consecutiveFailures == maxFailures) and recordFailure() is called, consecutiveFailures is incremented to maxFailures + 1. The guard consecutiveFailures >= maxFailures fires again and emits another "OPENED after N consecutive failures" warning, even though the circuit was previously opened and is simply re-opening after a probe failure. This can be confusing in logs and slightly inflates the reported failure count.

  recordFailure(): void {
    this.consecutiveFailures += 1;
    this.lastFailureTime = Date.now();

    if (this.consecutiveFailures === this.maxFailures) {   // was >=
      log.warn(
        `Compaction circuit breaker OPENED after ${this.consecutiveFailures} consecutive failures — ` +
          `pausing compaction for ${Math.ceil(this.resetAfterMs / 1000)}s`,
      );
    }
  }

Changing >= to === emits the log only on the exact transition, not on repeated re-openings from half-open.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/compaction-circuit-breaker.ts
Line: 82-92

Comment:
**Spurious "OPENED" log on half-open → open re-transition**

When the circuit is already in the half-open state (`consecutiveFailures == maxFailures`) and `recordFailure()` is called, `consecutiveFailures` is incremented to `maxFailures + 1`. The guard `consecutiveFailures >= maxFailures` fires again and emits another "OPENED after N consecutive failures" warning, even though the circuit was previously opened and is simply re-opening after a probe failure. This can be confusing in logs and slightly inflates the reported failure count.

```ts
  recordFailure(): void {
    this.consecutiveFailures += 1;
    this.lastFailureTime = Date.now();

    if (this.consecutiveFailures === this.maxFailures) {   // was >=
      log.warn(
        `Compaction circuit breaker OPENED after ${this.consecutiveFailures} consecutive failures — ` +
          `pausing compaction for ${Math.ceil(this.resetAfterMs / 1000)}s`,
      );
    }
  }
```

Changing `>=` to `===` emits the log only on the exact transition, not on repeated re-openings from half-open.

How can I resolve this? If you propose a fix, please make it concise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(compaction): retry loop can burn tokens when summarizer model is unavailable

1 participant