Skip to content

[Feature]: Dreaming phase crash recovery — detect and record interrupted runs #97692

Description

@lg320531124

Summary

Add crash recovery for dreaming phases so that when a phase is interrupted (process crash, OOM, timeout mid-execution), the system can identify the partial run and resume or cleanly discard it on next startup.

Problem to solve

When a dreaming phase crashes mid-execution (e.g., process killed during deep dreaming), the following state is left behind:

  1. No outcome record: The memory.dream.completed event is never emitted for the crashed run (related: [Bug]: Dreaming phase completion events lack outcome normalization (no success/failure/partial status) #97690). The event log shows a gap — the previous successful run, then nothing until the next scheduled run succeeds.
  2. Stale cron marker: The cron runningAtMs marker persists in SQLite. On next startup, the stuck marker is cleared by STUCK_RUN_MS maintenance (src/cron/service/jobs.ts:579-591), but this happens silently and the interrupted phase result is lost.
  3. Partial memory writes: If a dreaming phase wrote some inline memory before crashing, the partial write remains on disk with no indication it was produced by an interrupted run. Subsequent dreaming phases may treat it as valid.

Currently, DEFAULT_MEMORY_DEEP_DREAMING_RECOVERY_ENABLED = true (in src/memory-host-sdk/dreaming.ts:45) controls memory recovery — re-promoting forgotten memories when health drops below 0.35. This is a different concern from crash recovery — detecting and handling a dreaming phase that was interrupted.

Proposed solution

  1. Write a phase-start event before execution begins (memory.dream.started), analogous to how agent runs emit start events. This creates a "pending" marker in the event log.
  2. On startup, check for orphaned started events — if a memory.dream.started exists without a corresponding memory.dream.completed for the same phase and day, the phase was interrupted.
  3. Emit a memory.dream.completed with outcome: "interrupted" for orphaned runs, so the event log is self-healing and downstream consumers can distinguish interrupted from successful runs.
  4. Mark partial inline memory writes with a generation/sequence counter so subsequent dreaming phases can detect and optionally reprocess partial writes.

This approach follows the pattern already established by:

  • agent-run-terminal-outcome.ts normalizing agent run terminal states (7 distinct reasons)
  • The cron runningAtMs stuck-marker cleanup in service/jobs.ts:579-591

Alternatives considered

  • Rely on cron stuck-marker cleanup alone: The cron system already clears stale runningAtMs markers, but this only affects the cron scheduler — it does not recover or record the interrupted dreaming result. Partial memory writes remain.
  • Write-phase checkpoint files: Could write intermediate checkpoints during dreaming, but this adds I/O overhead to a hot path and the dreaming phases are designed to be idempotent (re-running a phase on the same data produces equivalent results).
  • Use the existing recovery config: The DEFAULT_MEMORY_DEEP_DREAMING_RECOVERY_* settings address memory health degradation, not run interruption. They are complementary, not overlapping.

Impact

Affected: All users with dreaming enabled, especially those running long deep/REM dreaming phases on resource-constrained systems (e.g., Raspberry Pi, small VPS) where OOM or process kills are more likely.
Severity: Medium (interrupted dreaming runs currently leave orphaned state with no diagnostic trail).
Frequency: Intermittent — only when a dreaming phase is interrupted, which depends on system stability and dreaming duration.
Consequence: Without crash recovery, interrupted dreaming runs are invisible in monitoring, partial memory writes accumulate without annotation, and the next successful run has no awareness that the previous attempt was interrupted.

Evidence/examples

  • extensions/memory-core/src/dreaming-narrative.ts:90-95 — narrative generation has a 60s timeout but no crash recovery mechanism.
  • extensions/memory-core/src/dreaming-narrative.ts:960"Narrative generation is best-effort — never fail the parent phase." — failures are swallowed, not recorded.
  • src/cron/service/jobs.ts:579-591 — cron stuck-marker cleanup shows the existing pattern for detecting interrupted runs, but only at the cron level, not the dreaming level.
  • src/agents/agent-run-terminal-outcome.ts:18-25AgentRunTerminalReason normalizes 7 distinct agent run terminal states. Dreaming has no equivalent.

Additional information

Related to #97690 (dreaming outcome normalization). That issue covers the missing outcome field in the event type; this issue covers the crash detection and recovery mechanism. Both are needed for complete observability of dreaming phases.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions