Skip to content

fix: replies fail when memory flush is exhausted#100618

Merged
steipete merged 2 commits into
mainfrom
codex/fix-memory-flush-death-loop-85645
Jul 6, 2026
Merged

fix: replies fail when memory flush is exhausted#100618
steipete merged 2 commits into
mainfrom
codex/fix-memory-flush-death-loop-85645

Conversation

@steipete

@steipete steipete commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Closes #85645

Replacement for #88968. The contributor branch was takeover-ready, but GitHub's verified fork editor could not preserve workspace symlink modes while replaying current main. This maintainer branch carries the reviewed tree as one commit and preserves contributor credit.

What Problem This Solves

Fixes an issue where users could stop receiving normal agent replies when pre-compaction memory maintenance repeatedly failed. The maintenance error replaced the real reply, and the oversized active session remained vulnerable to the same failure on later messages.

Why This Change Was Made

Memory-flush failures are now non-terminal to the user turn. The helper returns an explicit maintenance outcome; after three failures the runner rotates to a fresh active session while preserving the old transcript, clears stale compaction/flush state, and optionally emits a degraded maintenance notice through the existing agents.defaults.compaction.notifyUser path.

Resolved embedded runs that contain visible error payloads now enter the same failure counter instead of being stamped successful. If that run already completed a transcript rotation, the new session binding is preserved before the failure is recorded.

User Impact

Normal replies continue when memory maintenance is temporarily unavailable. Persistently failing, oversized sessions recover automatically after the existing retry budget; operators retain the previous transcript for recovery or forensics. Default behavior remains silent unless compaction notices are enabled.

Evidence

  • Reproduced source-level failure on current main: visible memory-flush errors terminate the turn before the normal agent run.
  • Contributor live Gateway proof: deterministic flush-provider HTTP 500 responses while all 12 user messages completed successfully after the fix.
  • Sanitized direct AWS Crabbox, provider aws, lease cbx_882f0c2cb4bb, run run_c7e91a20d30f: 7 focused files, 330 tests passed; hydration disabled, public networking, scrubbed launch environment, pinned checksum-verified Node 24 bootstrap. Lease auto-stopped.
  • Final resolved-error/rotation regression: node scripts/run-vitest.mjs src/auto-reply/reply/agent-runner-memory.test.ts — 47 tests passed.
  • git diff --check — clean.
  • Fresh full-branch Codex autoreview: no accepted/actionable findings.

Co-authored with @Jerry-Xin; the squashed commit retains both contributor commit identities.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation gateway Gateway runtime size: L maintainer Maintainer-authored PR labels Jul 6, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b62c414076

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

fallbackNoticeSelectedModel: undefined,
fallbackNoticeActiveModel: undefined,
fallbackNoticeReason: undefined,
compactionCount: 0,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Clear stale auth-profile compaction state on reset

When this reset mints a fresh session after memory-flush exhaustion or a role-ordering conflict, it resets compactionCount to 0 but the spread above still preserves authProfileOverrideCompactionCount. resolveSessionAuthProfileOverride rotates auto-selected profiles only when compactionCount > authProfileOverrideCompactionCount (src/agents/auth-profiles/session-override.ts:251-275), so a session reset from compaction 5 can keep the old auto profile pinned until the new session compacts six more times. Clear or rebase the auth-profile compaction marker along with the other reset-only state.

Useful? React with 👍 / 👎.

@clawsweeper

clawsweeper Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs changes before merge. Reviewed July 6, 2026, 1:38 AM ET / 05:38 UTC.

Summary
The PR changes reply-run memory-flush failures to return an explicit outcome, continue the user reply, rotate exhausted sessions while preserving transcripts, clear stale flush state, and document the expanded compaction notice behavior.

PR surface: Source +17, Tests +245, Docs 0. Total +262 across 16 files.

Reproducibility: yes. Current main source still returns visible memory-flush errors as a terminal reply before the normal run, and the PR body includes after-fix live Gateway output showing repeated flush 500s while all 12 messages complete.

Review metrics: 2 noteworthy metrics.

  • Config notice surface: 1 existing option expanded, 0 added, 0 removed. agents.defaults.compaction.notifyUser now also controls degraded memory-flush notices, so existing opt-in deployments get new user-visible behavior.
  • Reset state coupling: 1 compaction counter reset plus 6 memoryFlush fields cleared. The reset-state bundle is central to the fix and exposes the missing auth-profile compaction marker reset before merge.

Stored data model
Persistent data-model change detected: serialized state: src/auto-reply/reply/agent-runner-session-reset.test.ts, serialized state: src/auto-reply/reply/session.test.ts, unknown-data-model-change: src/auto-reply/reply/session.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #85645
Summary: This PR is the active maintainer replacement candidate for the canonical memoryFlush failure-loop issue; the earlier contributor PR is superseded by this branch.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦞 diamond lobster
Patch quality: 🦐 gold shrimp
Result: needs maintainer review before merge.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Clear or rebase authProfileOverrideCompactionCount during resetReplyRunSession and add focused regression coverage.
  • Get maintainer acceptance for the fail-open session rotation and expanded notifyUser degraded notice.

Risk before merge

  • [P1] The PR resets compactionCount to 0 during resetReplyRunSession but preserves authProfileOverrideCompactionCount, so an auto-selected auth profile can remain pinned longer than intended after a rotated session.
  • [P1] Existing deployments with agents.defaults.compaction.notifyUser=true would receive a new degraded memory-maintenance notice on memory-flush exhaustion without a new config key.
  • [P1] The recovery intentionally trades active-session continuity for reply liveness by rotating the exhausted session while preserving the previous transcript, which needs maintainer acceptance before merge.

Maintainer options:

  1. Fix reset state before merge (recommended)
    Clear or rebase authProfileOverrideCompactionCount when resetReplyRunSession resets compactionCount so rotated sessions do not stale-pin auto auth profiles.
  2. Accept the recovery tradeoff
    Maintainers can intentionally accept non-destructive session rotation after exhaustion once the narrow reset-state bug is repaired.
  3. Split the degraded notice
    If the expanded notifyUser behavior is too broad, keep the recovery path but move the degraded memoryFlush notice behind a narrower product decision.

Next step before merge

  • [P2] A narrow automated repair can fix the stale auth-profile compaction marker, though final merge still needs maintainer acceptance of the recovery behavior.

Maintainer decision needed

  • Question: Should OpenClaw adopt this fail-open memoryFlush exhaustion behavior after the reset auth-profile marker is fixed?
  • Rationale: The code path changes compatibility-sensitive session recovery and user-visible notice behavior; automation can repair the narrow reset bug but cannot decide the product tradeoff.
  • Likely owner: steipete — steipete has the strongest current ownership signal across this PR, reply/session recovery, and auth-profile selection history.
  • Options:
    • Fix marker and accept recovery (recommended): Clear or rebase the auth-profile compaction marker, then merge once maintainers confirm reply liveness should win over keeping the bloated active session.
    • Keep recovery but narrow notices: Land the fail-open rotation but remove or split the degraded notice if existing notifyUser deployments should only receive compaction start/end messages.
    • Pause for pipeline reconciliation: Hold this PR until maintainers choose a combined direction with the adjacent memoryFlush/preflight-compaction ordering work.

Security
Cleared: No concrete security or supply-chain concern found; the diff does not change dependencies, CI, lockfiles, permissions, secret handling, or external code execution.

Review findings

  • [P2] Clear the auth-profile compaction marker on reset — src/auto-reply/reply/agent-runner-session-reset.ts:84-90
Review details

Best possible solution:

Fix the reset auth-profile marker, then land this or an equivalent fail-open memoryFlush recovery only after maintainers accept the session-rotation and notice semantics.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main source still returns visible memory-flush errors as a terminal reply before the normal run, and the PR body includes after-fix live Gateway output showing repeated flush 500s while all 12 messages complete.

Is this the best way to solve the issue?

No as-is. The fail-open recovery is a plausible narrow fix for the bug, but the reset helper must clear or rebase auth-profile compaction state and maintainers still need to accept the recovery and notice semantics.

Full review comments:

  • [P2] Clear the auth-profile compaction marker on reset — src/auto-reply/reply/agent-runner-session-reset.ts:84-90
    This reset now forces compactionCount back to 0, but the spread above still carries any auto-selected authProfileOverrideCompactionCount from the exhausted session. resolveSessionAuthProfileOverride compares the new compaction count with that stored marker, so a reset from compaction 5 can keep the old auto auth profile pinned until the new session compacts six more times. Clear or rebase that marker with the other reset-only state.
    Confidence: 0.88

Overall correctness: patch is incorrect
Overall confidence: 0.86

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 89f911f322c9.

Label changes

Label changes:

  • add P1: The PR targets a real agent/channel liveness failure where repeated memoryFlush errors can prevent normal user replies.
  • add merge-risk: 🚨 compatibility: The PR expands an existing notifyUser option and changes exhaustion recovery semantics for current deployments.
  • add merge-risk: 🚨 session-state: The PR rotates active sessions and clears compaction/memoryFlush state after exhaustion, changing session continuity behavior.
  • add merge-risk: 🚨 auth-provider: The current patch can preserve a stale auto auth-profile compaction marker after resetting the session compaction counter.
  • add proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix live Gateway output with repeated mock flush-provider 500s and all 12 WebSocket messages completing status=ok, plus redacted runtime logs.
  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body includes after-fix live Gateway output with repeated mock flush-provider 500s and all 12 WebSocket messages completing status=ok, plus redacted runtime logs.

Label justifications:

  • P1: The PR targets a real agent/channel liveness failure where repeated memoryFlush errors can prevent normal user replies.
  • merge-risk: 🚨 compatibility: The PR expands an existing notifyUser option and changes exhaustion recovery semantics for current deployments.
  • merge-risk: 🚨 session-state: The PR rotates active sessions and clears compaction/memoryFlush state after exhaustion, changing session continuity behavior.
  • merge-risk: 🚨 auth-provider: The current patch can preserve a stale auto auth-profile compaction marker after resetting the session compaction counter.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body includes after-fix live Gateway output with repeated mock flush-provider 500s and all 12 WebSocket messages completing status=ok, plus redacted runtime logs.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix live Gateway output with repeated mock flush-provider 500s and all 12 WebSocket messages completing status=ok, plus redacted runtime logs.
Evidence reviewed

PR surface:

Source +17, Tests +245, Docs 0. Total +262 across 16 files.

View PR surface stats
Area Files Added Removed Net
Source 7 108 91 +17
Tests 7 310 65 +245
Docs 2 3 3 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 16 421 159 +262

Acceptance criteria:

  • [P1] pnpm test src/auto-reply/reply/agent-runner-session-reset.test.ts src/agents/auth-profiles/session-override.test.ts.
  • [P1] pnpm check:changed -- src/auto-reply/reply/agent-runner-session-reset.ts src/auto-reply/reply/agent-runner-session-reset.test.ts src/agents/auth-profiles/session-override.test.ts.

What I checked:

  • Repository policy applied: Root AGENTS.md and scoped docs/agents guides were read; the review applied the repo guidance that session state, config/default behavior, auth/provider routing, and message delivery are compatibility-sensitive surfaces. (AGENTS.md:1, 89f911f322c9)
  • Current main aborts on visible memory-flush errors: Current main collects visible memory-flush error payloads, calls replyOperation.fail("run_failed"), and returns before creating the normal follow-up runner. (src/auto-reply/reply/agent-runner.ts:1621, 89f911f322c9)
  • PR head makes memory flush non-terminal: The PR head logs visible memory-flush maintenance payloads, treats exhaustion as a reset trigger, updates the reply operation session id, and optionally sends the degraded notice before running the main reply. (src/auto-reply/reply/agent-runner.ts:1631, 964b684b00c3)
  • Blocking reset-state mismatch: The PR resets compactionCount and memoryFlush fields in resetReplyRunSession, but the spread still preserves authProfileOverrideCompactionCount from the previous session. (src/auto-reply/reply/agent-runner-session-reset.ts:84, 964b684b00c3)
  • Auth-profile rotation contract: resolveSessionAuthProfileOverride compares compactionCount with authProfileOverrideCompactionCount when deciding whether to rotate an auto-selected auth profile, so resetting one counter without the other can stale-pin the old profile. (src/agents/auth-profiles/session-override.ts:251, 89f911f322c9)
  • Real behavior proof in PR body: The PR body reports a live Gateway run with deterministic flush-provider HTTP 500 responses where all 12 WebSocket user messages completed with status=ok after the fix. (964b684b00c3)

Likely related people:

  • steipete: GitHub history and local blame show repeated recent work in reply/session recovery, auth-profile selection, and this maintainer replacement branch. (role: recent area contributor and likely follow-up owner; confidence: high; commits: 443c582949e3, 6df0fb818d67, 2c31af136429; files: src/auto-reply/reply/agent-runner.ts, src/auto-reply/reply/agent-runner-memory.ts, src/agents/auth-profiles/session-override.ts)
  • jalehman: GitHub file history shows recent lifecycle/session accessor refactors on the reset and session storage seams affected by this PR. (role: recent session-state contributor; confidence: medium; commits: 49e6f5a524bc, 127e174c9e4d; files: src/auto-reply/reply/agent-runner-session-reset.ts, src/auto-reply/reply/agent-runner-memory.ts)
  • LZY3538: GitHub file history shows recent compaction preservation work in the memory runner, which is adjacent to the changed memoryFlush helper behavior. (role: recent memory-compaction contributor; confidence: medium; commits: e6ec74c25430; files: src/auto-reply/reply/agent-runner-memory.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. labels Jul 6, 2026
@steipete

steipete commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Land-ready maintainer proof:

  • Reviewed the complete reply/memory/session-reset surface against current main, including callers, failure counters, completed-compaction rebinding, queued-session refresh, reset persistence, and notice delivery.
  • Improved the inherited patch before landing: explicit closed memory-flush outcome, shared maintenance-notice path, resolved error payloads counted as failures, completed transcript rotations preserved before failure recording, and duplicate mocked tests removed.
  • Sanitized direct AWS Crabbox: provider aws, lease cbx_882f0c2cb4bb, run run_c7e91a20d30f; hydration disabled, public networking, scrubbed launch environment, temporary HOME, checksum-verified Node 24 bootstrap. Seven focused files / 330 tests passed; lease auto-stopped.
  • Final regression: node scripts/run-vitest.mjs src/auto-reply/reply/agent-runner-memory.test.ts — 47 tests passed.
  • Formatting/lint: targeted oxfmt, oxlint src/auto-reply/reply/agent-runner-memory.test.ts, and git diff --check passed.
  • Fresh Codex autoreview converged after accepted findings; final result had no actionable findings.
  • Exact head 964b684b00c3e596fd11ef64a1f9b9d0ac61d4e5: CI run 28769952375 and Workflow Sanity run 28769952397 passed.

Known proof gap: no live third-party provider outage was induced. The contributor's deterministic Gateway proof used repeated mock-provider HTTP 500 responses and showed all 12 normal replies completing; maintainer proof exercised the same recovery state machine locally and in isolated AWS.

@steipete
steipete merged commit dd972b8 into main Jul 6, 2026
146 of 150 checks passed
@steipete
steipete deleted the codex/fix-memory-flush-death-loop-85645 branch July 6, 2026 05:41
@steipete

steipete commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Merged via squash.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 6, 2026
* fix(reply): recover from memory flush exhaustion

Co-authored-by: 忻役 <[email protected]>

Co-authored-by: Jerry-Xin <[email protected]>

* test(reply): remove unused memory flush result

---------

Co-authored-by: Jerry-Xin <[email protected]>
giodl73-repo pushed a commit to giodl73-repo/openclaw that referenced this pull request Jul 8, 2026
* fix(reply): recover from memory flush exhaustion

Co-authored-by: 忻役 <[email protected]>

Co-authored-by: Jerry-Xin <[email protected]>

* test(reply): remove unused memory flush result

---------

Co-authored-by: Jerry-Xin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Improvements or additions to documentation gateway Gateway runtime maintainer Maintainer-authored PR merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. size: L status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

memoryFlush has no escalation when assistant process dies mid-flush — session permanently over threshold, every subsequent message wedges

1 participant