fix(compaction): add circuit breaker for retry loops#58846
Conversation
When compaction repeatedly fails (model errors, timeouts, malformed output), retryAsync burns tokens without progress. Add CompactionCircuitBreaker with closed/open/half-open states that stops attempts after N consecutive failures and resumes after a cooldown period. Wire into summarizeChunks() as an optional parameter — callers can pass a shared circuit breaker instance to prevent cascading retry storms.
Greptile SummaryThis PR introduces a However, there is one blocking issue:
Additionally:
Confidence Score: 3/5Not safe to merge as-is — the circuit breaker feature is inaccessible via the public API and effectively dead code. There is one P1 functional defect: the circuitBreaker parameter is wired only into the private summarizeChunks function and is always undefined because neither summarizeWithFallback nor summarizeInStages accepts or forwards it. The PR's described feature — stopping runaway retries — does not actually work. This needs to be fixed before the change has any effect in production. src/agents/compaction.ts — the public API functions summarizeWithFallback and summarizeInStages need circuitBreaker? added to their parameter types and forwarded to the internal summarizeChunks calls.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3663b3bebb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| customInstructions?: string; | ||
| summarizationInstructions?: CompactionSummarizationInstructions; | ||
| previousSummary?: string; | ||
| circuitBreaker?: CompactionCircuitBreaker; |
There was a problem hiding this comment.
Wire circuit breaker through compaction entrypoints
circuitBreaker was added only to the private summarizeChunks params, but the exported entrypoints still cannot pass it (summarizeWithFallback/summarizeInStages do not accept this field, and existing call sites continue passing the old shape). In practice this leaves circuitBreaker undefined on every normal path, so the new open-circuit guard never triggers and compaction retries keep running as before under repeated failures.
Useful? React with 👍 / 👎.
| recordFailure(): void { | ||
| this.consecutiveFailures += 1; | ||
| this.lastFailureTime = Date.now(); | ||
|
|
||
| if (this.consecutiveFailures >= this.maxFailures) { | ||
| log.warn( | ||
| `Compaction circuit breaker OPENED after ${this.consecutiveFailures} consecutive failures — ` + | ||
| `pausing compaction for ${Math.ceil(this.resetAfterMs / 1000)}s`, | ||
| ); | ||
| } | ||
| } |
There was a problem hiding this comment.
Spurious "OPENED" log on half-open → open re-transition
When the circuit is already in the half-open state (consecutiveFailures == maxFailures) and recordFailure() is called, consecutiveFailures is incremented to maxFailures + 1. The guard consecutiveFailures >= maxFailures fires again and emits another "OPENED after N consecutive failures" warning, even though the circuit was previously opened and is simply re-opening after a probe failure. This can be confusing in logs and slightly inflates the reported failure count.
recordFailure(): void {
this.consecutiveFailures += 1;
this.lastFailureTime = Date.now();
if (this.consecutiveFailures === this.maxFailures) { // was >=
log.warn(
`Compaction circuit breaker OPENED after ${this.consecutiveFailures} consecutive failures — ` +
`pausing compaction for ${Math.ceil(this.resetAfterMs / 1000)}s`,
);
}
}Changing >= to === emits the log only on the exact transition, not on repeated re-openings from half-open.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/compaction-circuit-breaker.ts
Line: 82-92
Comment:
**Spurious "OPENED" log on half-open → open re-transition**
When the circuit is already in the half-open state (`consecutiveFailures == maxFailures`) and `recordFailure()` is called, `consecutiveFailures` is incremented to `maxFailures + 1`. The guard `consecutiveFailures >= maxFailures` fires again and emits another "OPENED after N consecutive failures" warning, even though the circuit was previously opened and is simply re-opening after a probe failure. This can be confusing in logs and slightly inflates the reported failure count.
```ts
recordFailure(): void {
this.consecutiveFailures += 1;
this.lastFailureTime = Date.now();
if (this.consecutiveFailures === this.maxFailures) { // was >=
log.warn(
`Compaction circuit breaker OPENED after ${this.consecutiveFailures} consecutive failures — ` +
`pausing compaction for ${Math.ceil(this.resetAfterMs / 1000)}s`,
);
}
}
```
Changing `>=` to `===` emits the log only on the exact transition, not on repeated re-openings from half-open.
How can I resolve this? If you propose a fix, please make it concise.
Fixes #58838
Summary
When compaction repeatedly fails (model errors, timeouts, malformed output),
retryAsyncburns tokens without progress. AddCompactionCircuitBreakerthat stops attempts after N consecutive failures and resumes after a cooldown period.CompactionCircuitBreakerclass with closed/open/half-open statessummarizeChunks()as optional parameter (backward compatible)previousSummaryinstead of retryingTest plan
reset()clears all state