Skip to content

Bug: 180s compaction timeout is a single wall clock over the whole chunk pipeline with no partial-progress reuse, so a legitimately-long compaction fails identically every turn  #92043

Description

@yetval

Summary

#91361 lowered the default embedded compaction timeout from 900s to 180s (EMBEDDED_COMPACTION_TIMEOUT_MS). For any install whose summarization legitimately needs more than 180s (long history, slow/local provider, fallback chain), this converts a slow-but-recoverable compaction into a deterministic, permanent failure loop: every turn re-runs the full multi-chunk summarization from scratch under the same 180s cap, times out at the same point, and discards all completed chunk summaries. There is no exit except a manual config edit or /new (history loss), and the failure message names neither the cause (timeout) nor the knob that fixes it.

Three structural facts make the 180s default unrecoverable rather than merely slow:

  • (a) One wall clock for the entire pipeline. The single 180s budget spans every sequential generateSummary call across all chunks (plus the merge call and quality-guard retries), not one call. Chunked safeguard mode targets exactly the long histories that exceed 3 minutes.

  • (b) No partial-progress reuse. Completed chunk summaries live only in a local variable inside one summarizeChunks call. On timeout the whole call throws; the next attempt restarts from chunk 1. Every retry is identical, so the loop can never converge.

  • (c) The error hides the cause and the knob. The user-facing failure text never mentions the timeout or agents.defaults.compaction.timeoutSeconds; the reason is suppressed unless verbose.

  • Impact: every turn (and manual /compact) returns the context-too-large warning forever, for the affected install, until the user hand-edits config or starts a fresh session and loses history. P0-like for those installs.

  • Trigger: session grows past the compaction threshold and safeguard summarization needs more than 180s wall-clock. A fallback chain compounds it: each fallback provider burns another full 180s.

  • Affected: any unconfigured install (the default) on a slow/local provider or with a large history; no agents.defaults.compaction.timeoutSeconds set.

  • Version audited: main @ 0923ee251e1

Root cause

1. The default cap is one budget for the whole pipelinesrc/agents/embedded-agent-runner/compaction-safety-timeout.ts:9,59-65,67-137:

// src/agents/embedded-agent-runner/compaction-safety-timeout.ts:9
export const EMBEDDED_COMPACTION_TIMEOUT_MS = 180_000; // was 900_000 (#91361)

// :59-65 — every unconfigured install resolves to the 180s default
export function resolveCompactionTimeoutMs(cfg?: OpenClawConfig): number {
  return (
    finiteSecondsToTimerSafeMilliseconds(cfg?.agents?.defaults?.compaction?.timeoutSeconds, {
      floorSeconds: true,
    }) ?? EMBEDDED_COMPACTION_TIMEOUT_MS
  );
}

compactWithSafetyTimeout(compact, timeoutMs) wraps the entire compaction in one withTimeout(..., timeoutMs, "Compaction"). On expiry it aborts the whole pipeline and rejects with Compaction timed out (src/node-host/with-timeout.ts:22). The same resolver feeds every surface that can trigger compaction, so they all share the one cap: preflight (src/agents/embedded-agent-runner/compact.ts:1142,1423), in-run overflow and timeout-recovery (src/agents/embedded-agent-runner/run.ts:2020,2217), and manual /compact (src/agents/command/cli-compaction.ts:302,388,439).

2. The summarize loop holds no progress across attemptssrc/agents/compaction.ts:174-225. summarizeChunks walks chunks sequentially, accumulating into a local summary. The single wall clock spans every iteration:

// src/agents/compaction.ts:174-225 (abridged)
let summary = params.previousSummary;
for (const chunk of chunks) {
  try {
    summary = await retryAsync(
      () => generateSummary(chunk, params.model, /* ... */ summary),
      { attempts: 3, minDelayMs: 500, maxDelayMs: 5000, /* ... */ },
    );
    hasGeneratedChunk = true;
  } catch (err) {
    if (isAbortError(err) || isTimeoutError(err)) {
      throw err; // timeout/abort propagates immediately — partial summary is dropped
    }
    // ... partial summary only attached to a thrown Error, consumed within this
    //     summarizeWithFallback call; never persisted for the next attempt.
    const partial = new Error("partial summarization failure");
    (partial as PartialSummaryError).partialSummary = `${summary!}\n\n[Partial summary: chunks 1-${completedChunks} ...]`;
    throw partial;
  }
}

On a timeout (isTimeoutError) the loop throws before attaching even the in-call partial, so the completed chunk summaries are discarded. The next turn re-enters compactWithSafetyTimeout and recomputes from chunk 1 under the same 180s budget. Nothing on disk or in the session store carries completed chunk work forward, so attempt N is bit-for-bit the same race as attempt 1.

3. The failure text names neither the timeout nor the knobsrc/auto-reply/reply/agent-runner-execution.ts:708-728:

// src/auto-reply/reply/agent-runner-execution.ts:708-728
export function buildPreflightCompactionFailureText(
  message: string,
  options?: { includeDetails?: boolean },
): string | null {
  // ...
  const reasonSuffix = options?.includeDetails && reason ? ` Reason: ${reason}.` : "";
  return (
    "⚠️ Context is too large and auto-compaction could not recover this turn." +
    `${reasonSuffix} Try again, use /compact, or use /new to start a fresh session.`
  );
}

In the default (non-verbose) path reasonSuffix is empty, so even Compaction timed out is hidden, and agents.defaults.compaction.timeoutSeconds is never surfaced. (Note the file already surfaces a different agents.defaults.timeoutSeconds knob for CLI-backend timeouts at line 744, so the omission here is specifically the compaction knob.)

Reproduction

Standalone vitest against pristine main (place at src/compaction-timeout-loop.test.ts). It drives the real shipped resolveCompactionTimeoutMs, compactWithSafetyTimeout, and buildPreflightCompactionFailureText. The per-chunk LLM call is modeled as a faithful stand-in for the sequential generateSummary loop at compaction.ts:174-225 (each chunk fast, four chunks exceed the cap), and the chunk-start log is shared across attempts so any cross-attempt reuse would show up.

import * as fs from "node:fs";
import { describe, expect, it, vi } from "vitest";
import {
  EMBEDDED_COMPACTION_TIMEOUT_MS,
  compactWithSafetyTimeout,
  resolveCompactionTimeoutMs,
} from "./agents/embedded-agent-runner/compaction-safety-timeout.js";
import { buildPreflightCompactionFailureText } from "./auto-reply/reply/agent-runner-execution.js";

const PER_CHUNK_MS = 60_000; // one chunk summary ~ 60s on a slow/local provider
const TOTAL_CHUNKS = 4; // 4 * 60s = 240s of real summarization work

function abortableDelay(ms: number, signal?: AbortSignal): Promise<void> {
  return new Promise((resolve, reject) => {
    if (signal?.aborted) return reject(signal.reason ?? new Error("aborted"));
    const timer = setTimeout(resolve, ms);
    signal?.addEventListener("abort", () => {
      clearTimeout(timer);
      reject(signal.reason ?? new Error("aborted"));
    }, { once: true });
  });
}

describe("180s compaction wall clock creates a deterministic failure loop", () => {
  it("default cap is 180s for any unconfigured install (no knob set)", () => {
    expect(EMBEDDED_COMPACTION_TIMEOUT_MS).toBe(180_000);
    expect(resolveCompactionTimeoutMs(undefined)).toBe(180_000);
  });

  it("one wall clock spans every chunk; partial summaries are discarded and re-run every attempt", async () => {
    vi.useFakeTimers();
    try {
      const chunkStartsByAttempt: number[][] = [];
      const runSafeguardPipeline = (attempt: number) => async (signal?: AbortSignal) => {
        const started: number[] = [];
        chunkStartsByAttempt[attempt] = started;
        let summary: string | undefined; // local-only; nothing persists it across calls
        for (let chunk = 1; chunk <= TOTAL_CHUNKS; chunk++) {
          started.push(chunk);
          await abortableDelay(PER_CHUNK_MS, signal); // one generateSummary call
          summary = `summary after chunk ${chunk}`;
        }
        return summary;
      };

      const outcomes: string[] = [];
      for (let attempt = 1; attempt <= 3; attempt++) {
        const cap = resolveCompactionTimeoutMs(undefined); // 180_000, unchanged each turn
        const run = compactWithSafetyTimeout(runSafeguardPipeline(attempt), cap);
        const settled = run.then(() => "compacted", (err: Error) => `threw: ${err.message}`);
        await vi.advanceTimersByTimeAsync(cap + 1_000);
        outcomes.push(await settled);
      }

      console.log("CHUNK_STARTS_PER_ATTEMPT", JSON.stringify(chunkStartsByAttempt.slice(1)));
      console.log("OUTCOMES", JSON.stringify(outcomes));
      expect(outcomes).toEqual([
        "threw: Compaction timed out", "threw: Compaction timed out", "threw: Compaction timed out",
      ]);
      expect(chunkStartsByAttempt[1]).toEqual([1, 2, 3]); // chunk 4 never ran
      expect(chunkStartsByAttempt[2]).toEqual([1, 2, 3]); // re-summarized chunk 1 from scratch
      expect(chunkStartsByAttempt[3]).toEqual([1, 2, 3]); // identical every turn -> never converges
    } finally {
      vi.useRealTimers();
    }
  });

  it("the user-facing failure text names no cause and no knob", () => {
    const text = buildPreflightCompactionFailureText(
      "Preflight compaction required but failed: Compaction timed out",
    );
    console.log("USER_FACING_TEXT", JSON.stringify(text));
    expect(text).not.toContain("timed out");
    expect(text).not.toContain("timeoutSeconds");
  });
});

Run:

export PATH=/opt/node/bin:$PATH
node scripts/run-vitest.mjs src/compaction-timeout-loop.test.ts

Observed (main @ 0923ee251e1, all assertions pass — i.e. the buggy behavior is exactly what ships)

resolveCompactionTimeoutMs(undefined) = 180000
EMBEDDED_COMPACTION_TIMEOUT_MS        = 180000

CHUNK_STARTS_PER_ATTEMPT = [[1,2,3],[1,2,3],[1,2,3]]
OUTCOMES                 = ["threw: Compaction timed out","threw: Compaction timed out","threw: Compaction timed out"]

USER_FACING_TEXT = "⚠️ Context is too large and auto-compaction could not recover this turn. Try again, use /compact, or use /new to start a fresh session."
  • The single 180s clock fires mid-pipeline every time: 3 of 4 chunks start, chunk 4 never runs, and the three completed chunk summaries are thrown away.
  • Attempts 2 and 3 restart at chunk 1 and reproduce the identical [1,2,3] progress: zero cross-attempt reuse, so the loop is deterministic and permanent.
  • The shipped failure text shows neither the timeout reason nor agents.defaults.compaction.timeoutSeconds.

Expected

A compaction that legitimately needs more than the default budget should make forward progress or fail gracefully, not loop forever:

  • Completed chunk summaries are reused across attempts (or the budget is applied per chunk/stage so a long pipeline is not killed wholesale).
  • When compaction does time out, the non-verbose failure text states the cause and points at agents.defaults.compaction.timeoutSeconds.

Related issues

This is the structural-default issue behind a cluster of compaction-timeout reports; it is distinct from each:

  • #91361 (merged, the cause) lowered 900s to 180s and its body notes that sessions needing more than 3 minutes "should set an explicit timeoutSeconds". That acknowledges the tradeoff but not (a) the single-wall-clock shape, (b) the zero partial-progress reuse, or (c) the silent error. This issue is the unacknowledged structural half.
  • #43661 (open, P1) "Session hangs indefinitely when compaction times out, causing repeated duplicate message sends" describes the loop under the old ~10-minute timeout and from the delivery-retry angle. After #91361 the same loop is a fast 180s deterministic failure; same family, different timeout and surface.
  • #64962 (open, P2) "Timeout-compaction doesn't escalate when compaction fails to reduce context" is the case where compaction completes but does not shrink context. Here it never completes at all.
  • #90526 (open, P1) "Auto-compaction silently fails after context overflow (attempt 1/3 with no follow-up)" and #81182 "Overflow recovery should truncate tool results before waiting full auto-compaction timeout" are adjacent symptoms/mitigations of the same no-progress path.

Happy to consolidate or cross-link if a maintainer prefers one umbrella.

Suggested direction

  • Budget per stage/chunk, not per pipeline. Apply the timeout to each generateSummary call (or scale the total by plan.chunks.length) so a long-but-healthy pipeline is not killed wholesale by one wall clock.
  • Persist completed chunk summaries across attempts. Carry forward the per-chunk summaries (the previousSummary seam already exists) so a retry resumes instead of restarting; this alone turns the permanent loop into eventual convergence.
  • Surface the cause and the knob. In the non-verbose failure text, when the reason is a timeout, say so and name agents.defaults.compaction.timeoutSeconds (mirroring the existing CLI-backend timeout hint at agent-runner-execution.ts:744).

Happy to send a PR with the fix plus the regression test above.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions