-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
[Feature]: Dreaming phase crash recovery — detect and record interrupted runs #97692
Copy link
Copy link
Open
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.ClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.Good issue quality with a plausible reproduction path needing some confirmation.
Description
Metadata
Metadata
Assignees
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.ClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.Good issue quality with a plausible reproduction path needing some confirmation.
Type
Fields
Priority
None yet
Summary
Add crash recovery for dreaming phases so that when a phase is interrupted (process crash, OOM, timeout mid-execution), the system can identify the partial run and resume or cleanly discard it on next startup.
Problem to solve
When a dreaming phase crashes mid-execution (e.g., process killed during deep dreaming), the following state is left behind:
memory.dream.completedevent is never emitted for the crashed run (related: [Bug]: Dreaming phase completion events lack outcome normalization (no success/failure/partial status) #97690). The event log shows a gap — the previous successful run, then nothing until the next scheduled run succeeds.runningAtMsmarker persists in SQLite. On next startup, the stuck marker is cleared bySTUCK_RUN_MSmaintenance (src/cron/service/jobs.ts:579-591), but this happens silently and the interrupted phase result is lost.Currently,
DEFAULT_MEMORY_DEEP_DREAMING_RECOVERY_ENABLED = true(insrc/memory-host-sdk/dreaming.ts:45) controls memory recovery — re-promoting forgotten memories when health drops below 0.35. This is a different concern from crash recovery — detecting and handling a dreaming phase that was interrupted.Proposed solution
memory.dream.started), analogous to how agent runs emit start events. This creates a "pending" marker in the event log.memory.dream.startedexists without a correspondingmemory.dream.completedfor the same phase and day, the phase was interrupted.memory.dream.completedwithoutcome: "interrupted"for orphaned runs, so the event log is self-healing and downstream consumers can distinguish interrupted from successful runs.This approach follows the pattern already established by:
agent-run-terminal-outcome.tsnormalizing agent run terminal states (7 distinct reasons)runningAtMsstuck-marker cleanup inservice/jobs.ts:579-591Alternatives considered
runningAtMsmarkers, but this only affects the cron scheduler — it does not recover or record the interrupted dreaming result. Partial memory writes remain.recoveryconfig: TheDEFAULT_MEMORY_DEEP_DREAMING_RECOVERY_*settings address memory health degradation, not run interruption. They are complementary, not overlapping.Impact
Affected: All users with dreaming enabled, especially those running long deep/REM dreaming phases on resource-constrained systems (e.g., Raspberry Pi, small VPS) where OOM or process kills are more likely.
Severity: Medium (interrupted dreaming runs currently leave orphaned state with no diagnostic trail).
Frequency: Intermittent — only when a dreaming phase is interrupted, which depends on system stability and dreaming duration.
Consequence: Without crash recovery, interrupted dreaming runs are invisible in monitoring, partial memory writes accumulate without annotation, and the next successful run has no awareness that the previous attempt was interrupted.
Evidence/examples
extensions/memory-core/src/dreaming-narrative.ts:90-95— narrative generation has a 60s timeout but no crash recovery mechanism.extensions/memory-core/src/dreaming-narrative.ts:960—"Narrative generation is best-effort — never fail the parent phase."— failures are swallowed, not recorded.src/cron/service/jobs.ts:579-591— cron stuck-marker cleanup shows the existing pattern for detecting interrupted runs, but only at the cron level, not the dreaming level.src/agents/agent-run-terminal-outcome.ts:18-25—AgentRunTerminalReasonnormalizes 7 distinct agent run terminal states. Dreaming has no equivalent.Additional information
Related to #97690 (dreaming outcome normalization). That issue covers the missing
outcomefield in the event type; this issue covers the crash detection and recovery mechanism. Both are needed for complete observability of dreaming phases.