fix: prevent memory flush failure from aborting user reply (#85645)#88968
fix: prevent memory flush failure from aborting user reply (#85645)#88968Jerry-Xin wants to merge 10 commits into
Conversation
|
Thanks for the context here. I swept through the related work, and this is now duplicate or superseded. Close this contributor PR as superseded: the same memory-flush recovery work now lives in the narrower maintainer replacement at #100618, which is open, mergeable, proof-positive, and explicitly preserves contributor credit. Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Canonical path: Live GitHub state shows #100618 is open, same-repo, MERGEABLE/CLEAN, labeled maintainer and proof sufficient, closes #85645, and has a focused 16-file diff. So I’m closing this here and keeping the remaining discussion on #100618 and #85645. Review detailsBest possible solution: Use #100618 as the canonical landing path, preserving this contributor's proof and credit there while closing this drifted source branch. Do we have a high-confidence way to reproduce the issue? Yes. Current origin/main source shows visible memory-flush payloads still call Is this the best way to solve the issue? Yes for the recovery behavior, but not through this branch anymore. The best landing surface is the maintainer replacement PR because it carries the same fix narrowly, credits the contributor, and avoids this branch's unrelated drift. Security review: Security review needs attention: The source branch should not merge because it now includes unrelated dependency/vendor-like package paths and broad plugin/app drift; the replacement PR avoids that supply-chain surface.
AGENTS.md: found and applied where relevant. What I checked:
Likely related people:
Codex review notes: model internal, reasoning high; reviewed against 89f911f322c9. |
776738c to
4af5d16
Compare
|
@clawsweeper re-review PR body updated with comprehensive real behavior proof:
All 200 tests pass across 6 test files (0 regressions). |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review PR body updated with compiled binary verification proof:
|
|
🦞👀 Command router queued. I will update this comment with the next step. Re-review progress:
|
4af5d16 to
29cdf0e
Compare
29cdf0e to
c0791b1
Compare
c0791b1 to
953bfbd
Compare
|
Rebased on latest upstream/main (965e680, clean rebase, no conflicts). |
953bfbd to
788b9b0
Compare
|
Rebased on latest upstream/main (646df2d, clean rebase, no conflicts). |
|
Rebased onto the latest upstream/main (3630d50, clean rebase, no conflicts). No code changes; the fail-open memoryFlush recovery, non-destructive session rotation, and notifyUser degraded-notice documentation remain in place. New head: cbca559. @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Rebased onto the latest upstream/main (07b9349, clean rebase, no conflicts). No code changes; the fail-open memoryFlush recovery, non-destructive session rotation, and notifyUser degraded-notice documentation remain in place. New head: 0c5fa42. @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Rebased onto the latest upstream/main (8d168c8, clean rebase, no conflicts). No code changes; the fail-open memoryFlush recovery, non-destructive session rotation, and notifyUser degraded-notice documentation remain in place. New head: e68f79a. @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Rebased onto the latest upstream/main (08d15ec, clean rebase, no conflicts). No code changes; the fail-open memoryFlush recovery, non-destructive session rotation, and notifyUser degraded-notice documentation remain in place. New head: 4ce0639. @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Rebased onto the latest upstream/main (2064575, clean rebase, no conflicts). No code changes; the fail-open memoryFlush recovery, non-destructive session rotation, and notifyUser degraded-notice documentation remain in place. New head: ccc7c0b. @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Rebased onto the latest upstream/main (clean, no conflicts) and fixed the @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Rebased onto the latest upstream/main (clean, no conflicts). Code-class CI is green on the new head (check-test-types, check-lint, build-artifacts, check-guards, check-shrinkwrap all passing). @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Rebased onto the latest upstream/main (clean, no conflicts) and fixed the @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
…85645) When memoryFlush fails (provider timeout, rate limit, etc.), the visible error payload was being treated as the final reply, aborting the user's actual message via replyOperation.fail(). This created a death loop: every subsequent message triggered the same flush failure, producing an error instead of a real reply. Changes: - agent-runner.ts: Log flush error payloads via logVerbose() and continue to the main reply. On exhaustion (failureCount >= MAX_FLUSH_FAILURES), rotate the bloated session via resetReplyRunSession(). - session.ts: Clear memoryFlushFailureCount, memoryFlushLastFailedAt, and memoryFlushLastFailureError in initSessionState() on session reset. - agent-runner-session-reset.ts: Clear compactionCount and all memoryFlush* fields in resetReplyRunSession(). The resetSession/resetSessionAfterRoleOrderingConflict helpers are hoisted earlier in the function so they are available before the memory flush phase, with no behavioral change to their later callers. Refs openclaw#85645
…eset After memory flush exhaustion triggers resetReplyRunSession(), the replyOperation.sessionId was still pointing to the old session ID. All three sibling rotation paths in agent-runner-memory.ts correctly call replyOperation.updateSessionId() after rotation — this path was the only one missing it. Without the rebind, the active-run registry, wait/abort routing, and restart-recovery checks use the stale session ID, which can mis-associate behavior after exhaustion recovery. Add dedicated regression test verifying updateSessionId is called with the rotated session ID after exhaustion reset.
…degraded notice Address two compatibility concerns on the memory-flush death-loop fix: - Exhaustion-triggered session rotation now uses cleanupTranscripts: false. Rotation alone breaks the death loop (the new session no longer hits the bloated transcript); the old transcript JSONL is kept on disk for forensics/recovery instead of being unlinked. The role-ordering reset path is unchanged. - When compaction.notifyUser is enabled, emit a short non-terminal degraded notice on flush exhaustion via the existing onBlockReply bypass. It never calls replyOperation.fail() or aborts; the main reply still proceeds. Default (notifyUser=false) keeps the current silent behavior. No new config surface. Tests updated/added for both paths; 110 tests pass across the affected files.
Resolve oxlint unbound-method and unnecessary-type-assertion findings by capturing the updateSessionId spy in a local const and dropping the redundant non-null assertion. Fix TS2345 by giving the rotated session entry literals concrete updatedAt/memoryFlushFailureCount values and moving sessionFile out of the entry object so it matches the expected parameter type.
The ReplyOperation interface gained terminalRecovery and markTerminalRecovery; update the createReplyOperation test mock to match so check-test-types passes.
|
ClawSweeper applied the proposed close for this PR.
|
|
Thanks @Jerry-Xin — the repaired version landed in #100618 as We could not safely update this fork branch: GitHub's verified fork-editor path rewrote nine workspace symlinks as regular files, so the maintainer replacement branch avoided pushing a lossy tree. For future PRs, enabling Allow edits by maintainers can make direct fixups possible when GitHub can preserve the tree correctly. #85645 is now closed by the landed fix. Thank you for finding and fixing this failure mode. |
Summary
When
memoryFlushfails during the reply pipeline (provider timeout, rate limit, network error), the visible error payload was treated as the final reply viareplyOperation.fail("run_failed", ...). This aborted the user's actual message. Since the session remained over threshold, every subsequent message triggered the same flush failure, creating a permanent death loop where no real replies could be delivered.Changes
src/auto-reply/reply/agent-runner.tslogVerbose()and not delivered as the reply. The main reply run proceeds normally.memoryFlushFailureCountcrossesMAX_FLUSH_FAILURESduring the current turn), the bloated session is automatically rotated viaresetReplyRunSession(). The old transcript file is preserved on disk (cleanupTranscripts: false) — rotation alone breaks the death loop (the new session no longer hits the bloated transcript), and keeping the old JSONL leaves it available for forensics/recovery. The active session only replays recent turns regardless of this flag.agents.defaults.compaction.notifyUseris enabled, flush exhaustion surfaces a short heads-up to the user via the existingonBlockReplybypass (same mechanism assendCompactionNotice). It never callsreplyOperation.fail()and never aborts the reply. Default (notifyUser=false) keeps the silent log-and-continue behavior. No new config surface is added.resetSession/resetSessionAfterRoleOrderingConflicthelpers are hoisted earlier in the function so they are available before the memory flush phase. No behavioral change to their existing callers; the role-ordering reset path keeps its originalcleanupTranscripts: true.src/auto-reply/reply/session.tsinitSessionState()now clearsmemoryFlushFailureCount,memoryFlushLastFailedAt, andmemoryFlushLastFailureErrorwhen a session is reset (e.g., via/new), preventing stale failure state from carrying over.src/auto-reply/reply/agent-runner-session-reset.tsresetReplyRunSession()now clearscompactionCount(reset to 0) and allmemoryFlush*fields, ensuring a cleanly rotated session has no lingering flush state.What is NOT changed
runMemoryFlushIfNeededbehavior is unchanged — it still tracks failures, incrementsmemoryFlushFailureCount, and emits lifecycle events (memory_flush_failed,memory_flush_exhausted).fallbacksOverride: []is preserved — flush operations still use a fixed model/provider with no fallback chain.MAX_FLUSH_FAILURES = 3threshold is unchanged.compaction.notifyUsergate.Tests
agent-runner-direct-runtime-config.test.ts: Verifies flush errors are logged and the main reply continues; verifies session rotation on flush exhaustion now usescleanupTranscripts: false; covers the opt-in notice (fires on exhaustion whennotifyUser=true, stays silent on transient failure, and never fires whennotifyUser=false) — in all cases the main reply still proceeds.agent-runner-session-reset.test.ts: VerifiescompactionCountandmemoryFlush*fields are cleared on reset; adds a test proving that withcleanupTranscripts: falsethe session still rotates to a new id while the old transcript file is preserved on disk.session.test.ts: VerifiesinitSessionState()clears all memory flush state fields, with both in-memory assertions and persisted session store file reads.Real behavior proof
Behavior or issue addressed: When
memoryFlush/ auto-compaction fails permanently (provider timeout, rate limit, 5xx), currentmainroutes the visible flush error intoreplyOperation.fail("run_failed")before the main reply run, aborting the user's real reply. Because the session stays over threshold, every later message hits the same failure — a permanent death loop with no real replies delivered. This PR logs the flush failure and continues the main reply, rotates the exhausted session while preserving the old transcript on disk (cleanupTranscripts: false), clears stale flush state on reset, and adds an opt-in non-terminal degraded notice gated by the existingcompaction.notifyUser.Real environment tested: OpenClaw Gateway run directly from this PR's source at current head
5dc502d76f(branchfix/memory-flush-death-loop-85645) viatsx(node --import tsx src/index.ts gateway run), on macOS arm64, Node.js v24.10.0. Isolated state dir,auth: none, Gateway port14881. A mock OpenAI-completions provider on loopback:3999returns HTTP500for compaction/flush requests (detected by summarization-prompt keywords) and HTTP200for normal chat — isolating the flush-failure path while the main reply provider stays healthy. Config:softThresholdTokens: 800,contextWindow: 4096,maxTokens: 1024.Exact steps or command run after this patch:
node mock-provider/server.mjs(loopback:3999, returns 500 on flush requests, 200 on normal chat).5dc502d76f:node --import tsx src/index.ts gateway --force --port 14881 --auth none runwith an isolated state dir.agent.waitcompletion.Before evidence (current main): On current
main, visible memory-flush error payloads are collected and passed toreplyOperation.fail("run_failed")before the main reply run, so the user's real reply is aborted and subsequent messages repeat the failure (src/auto-reply/reply/agent-runner.tsflush-error path onmain).Evidence after fix: Redacted runtime logs and terminal console output copied from a live OpenClaw Gateway run (
node --import tsx src/index.ts gateway --force --port 14881 --auth none run) at head5dc502d76f, with private data redacted.Mock provider — every flush request rejected with 500 (36 flush 500s, 1 normal 200):
Gateway — flush failures logged, session mapping preserved, run continues (not aborted):
WebSocket client — all 12 messages received a reply (
status=ok), none aborted:Observed result after fix: With the flush provider permanently broken (36 consecutive 500s), all 12 user messages still completed with
status=ok— the death loop is gone. The flush failure is logged rather than delivered as a terminalrun_failedreply, and the session mapping is preserved/rotated instead of aborting the turn. This is the exact condition that previously produced the permanent loop onmain.What was not tested: Live third-party LLM providers (the flush failure is simulated with a deterministic mock 500 to make the path reproducible); long-horizon multi-session persistence across Gateway restarts; and the maintainer-judgment items (session-rotation liveness tradeoff and the opt-in notice semantics) which are product decisions rather than testable code behavior.