Skip to content

[Bug]: Deleted transcripts ingested by dreaming + incomplete heartbeat filter leaks heartbeat noise into dream corpus #103720

Description

@btrailor

Summary

The dreaming ingestion pipeline has two compounding code-level issues that cause heartbeat poll responses to leak into the dream corpus as low-confidence (0.58) snippets. These snippets become the primary content for Light and REM dream phases, producing illegible "A memory trace surfaced, but details were unavailable in this run." fallback entries — 2 ghost entries per night, with only the Deep phase producing real content.

Environment

  • OpenClaw: 2026.6.10 (aa69b12)
  • Dreaming: enabled via agents.defaults.dreaming.enabled: true
  • Dreaming model: ollamapro/gemma4:31b
  • Heartbeat: every 5 minutes, useHeartbeatResponseTool: false (default, visibleReplies not set to message_tool)
  • Memory backend: builtin SQLite
  • Agent count: 60
  • Session transcript files: ~12,500 total (.jsonl + .jsonl.deleted.* + .jsonl.reset.*)

Issue 1: Deleted transcripts are ingested by dreaming

Root cause

isUsageCountedSessionTranscriptFileName in paths-BcGVZo_k.js returns true for files with .jsonl.deleted.* and .jsonl.reset.* suffixes. shouldSkipTranscriptFileForDreaming in engine-qmd-BaDqrBde.js does NOT exclude these deleted/reset files — it only skips compaction checkpoints and non-usage-counted archives.

Impact

listSessionFilesForAgent returns ALL .jsonl, .jsonl.deleted.*, and .jsonl.reset.* files. In our setup, 4,619 deleted/reset transcripts were on disk, with 459 falling within the 7-day REM lookback window. All are read and processed during every dreaming sweep.

For orphaned deleted files (not in sessions.json), the cron-run classification (if entry.generatedByCronRun → skip) never fires because classifySessionTranscriptFromSessionStore returns null. The fallback content-based classification checks isCronRunGeneratedRecord (which looks for sessionKey matching cron:...:run:... in record metadata) and isGeneratedCronPromptMessage (which matches [cron:...] prefix in user messages) — but heartbeat sessions have neither sessionKey in their records nor [cron:...] prefix in their user messages (their user message is [OpenClaw heartbeat poll]).

Evidence

  • 4,577 .jsonl.deleted.* files + 42 .jsonl.reset.* files found in ~/.openclaw/agents/main/sessions/
  • isUsageCountedSessionTranscriptFileName confirmed to return true for .deleted.* and .reset.* suffixes
  • shouldSkipTranscriptFileForDreaming confirmed to NOT filter these files
  • 4,402 of 4,577 deleted files are orphaned (not in sessions.json)

Suggested fix

shouldSkipTranscriptFileForDreaming should exclude .jsonl.deleted.* and .jsonl.reset.* files, OR listSessionFilesForAgent should not return deleted/reset files for dreaming ingestion. Deleted transcripts have already been garbage-collected from the session store; reading them during dreaming is unintended.

Related: #72611 (covers the deleted transcript leak more broadly — this issue adds the heartbeat-specific filter gap as a second compounding bug).

Issue 2: Incomplete heartbeat response filter in dreaming ingestion

Root cause

sanitizeSessionText in engine-qmd-BaDqrBde.js (the dreaming ingestion module) filters heartbeat content using:

  1. isHeartbeatUserMessage(text, role) — filters user messages containing [OpenClaw heartbeat poll]
  2. Manual check: role === "assistant" && normalized === "HEARTBEAT_OK"exact string match only

The codebase already has a more robust filter: isHeartbeatOkResponse in heartbeat-filter-2_xjQ2D1.js, which uses stripHeartbeatToken with a configurable ackMaxChars (default 300) to catch longer heartbeat acknowledgments that contain "HEARTBEAT_OK" or are within the acknowledgment character budget. This function is used by doctor-state-integrity-BbdG0V0k.js, session-transcript-path-*.js, and heartbeat-filter-2_xjQ2D1.js itself — but is NOT imported by engine-qmd-BaDqrBde.js.

engine-qmd-BaDqrBde.js only imports isHeartbeatUserMessage from heartbeat-filter-2_xjQ2D1.js:

import { r as isHeartbeatUserMessage } from "./heartbeat-filter-2_xjQ2D1.js";

The isHeartbeatOkResponse export (n) is not imported.

Impact

When the model generates any heartbeat acknowledgment other than the literal string "HEARTBEAT_OK" (e.g., "Heartbeat received. Main is active. No pending user request in this cron poll."), the response passes through sanitizeSessionText unfiltered and becomes a snippet in the session corpus at confidence 0.58. These snippets dominate the Light and REM dream phases because they outnumber real conversation snippets.

In our setup, 0 out of 50 sampled heartbeat assistant responses contained the exact string "HEARTBEAT_OK". All were natural language acknowledgments. Local models (ollamapro/glm-5.2, kimi-k2.7-code) do not reliably follow the "reply HEARTBEAT_OK" format instruction, especially when workspace files (HEARTBEAT.md, SOUL.md) contain conflicting guidance about whether to reply or stay silent.

Evidence

  • grep "isHeartbeatOkResponse" engine-qmd-BaDqrBde.js → no matches (not imported)
  • grep "isHeartbeatOkResponse" heartbeat-filter-2_xjQ2D1.js → function exists at line 184, uses stripHeartbeatToken with ackMaxChars
  • grep "isHeartbeatOkResponse" doctor-state-integrity-BbdG0V0k.js → imported and used (line 23, line 94)
  • 50/50 sampled heartbeat responses did NOT contain "HEARTBEAT_OK"
  • Dream corpus files (memory/.dreams/session-corpus/*.txt) contain heartbeat assistant responses as snippets with confidence: 0.58

Suggested fix

Import and use isHeartbeatOkResponse in sanitizeSessionText instead of the manual exact-string match. This would catch heartbeat acknowledgments within the ackMaxChars budget, which is already configurable at agents.defaults.heartbeat.ackMaxChars (default 300).

Compound effect

When both issues are active:

  1. Deleted heartbeat session transcripts are read (Issue 1)
  2. Heartbeat assistant responses are not filtered (Issue 2)
  3. Unfiltered responses become low-confidence snippets in the session corpus
  4. Light and REM dream phases receive mostly heartbeat noise as input
  5. Dreaming subagent receives garbage snippets, fails to synthesize meaningful narrative
  6. appendFallbackNarrativeEntry writes "A memory trace surfaced, but details were unavailable in this run." to DREAMS.md
  7. Result: 2 ghost entries per night (Light + REM), only Deep phase produces real content

Config-level mitigations applied (does not fix the code)

  1. Purged 4,619 deleted/reset transcript files from disk
  2. Updated HEARTBEAT.md to remove conflicting "silence is the default" instruction
  3. Applied allowModelOverride: true + allowedModels: ["*"] for dreaming subagent retry fallback

These reduce symptom severity but do not fix the underlying code issues.

Related issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.no-staleExclude from stale automation

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions