fix(memory): corpus dreaming eligibility contract + heartbeat provenance test#110571
fix(memory): corpus dreaming eligibility contract + heartbeat provenance test#110571headbouyJB wants to merge 3 commits into
Conversation
…integration tests for heartbeat provenance filtering
Add `eligibleForDreaming: boolean` to `SessionTranscriptCorpusEntry` so
archive artifacts (.jsonl.deleted.* / .jsonl.reset.*) carry an explicit
`false` signal rather than relying on corpus consumers to check
`artifactKind === "archive-artifact"` ad-hoc. Dreaming phases now gate
on `!entry.eligibleForDreaming` instead, making the eligibility boundary
a first-class corpus concern.
Also add the missing `session-transcript-corpus.test.ts` (clawsweeper
acceptance criterion) verifying that active sessions carry `true` and all
archive artifact variants (deleted, reset, orphaned, cron-lineage) carry
`false`.
Extend `seedDreamingSessionTranscript` to propagate an optional
`provenance` field, and add an end-to-end dreaming sweep test
demonstrating that a heartbeat turn with runtime provenance
`{ kind: "internal_system", sourceTool: "heartbeat" }` suppresses the
paired natural-language assistant response (not "HEARTBEAT_OK") from the
session corpus output — the path added by openclaw#109403 / commit 571f1dc.
Acceptance criteria satisfied:
- pnpm test packages/memory-host-sdk/src/host/session-files.test.ts (41/41)
- pnpm test packages/memory-host-sdk/src/host/session-transcript-corpus.test.ts (5/5 new)
- pnpm test extensions/memory-core/src/dreaming-phases.test.ts (47/47, +1 new)
Fixes openclaw#103720
Co-Authored-By: Claude Fable 5 <[email protected]>
openclaw-barnacle
Bot
added
extensions: memory-core
clawsweeper
Bot
added
rating: 🦪 silver shellfish
|
Codex review: needs maintainer review before merge. Reviewed July 18, 2026, 5:44 AM ET / 09:44 UTC. Summary PR surface: Source +12, Tests +289. Total +301 across 5 files. Reproducibility: yes. in source, with medium confidence: archive corpus entries and provenance-tagged heartbeat turns have a concrete seeded path through the corpus reader and Dreaming sweep, and the contributor supplied before/after runtime output; this review did not independently execute that path. Review metrics: none identified. Stored data model Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Next step before merge
Security Review detailsBest possible solution: Land a single bounded change that keeps Dreaming eligibility on the corpus-entry contract, preserves archive availability for Do we have a high-confidence way to reproduce the issue? Yes in source, with medium confidence: archive corpus entries and provenance-tagged heartbeat turns have a concrete seeded path through the corpus reader and Dreaming sweep, and the contributor supplied before/after runtime output; this review did not independently execute that path. Is this the best way to solve the issue? Yes: placing the live-versus-archive decision on the corpus entry avoids repeated consumer-side AGENTS.md: unclear because the file could not be read completely. Codex review notes: model internal, reasoning high; reviewed against fceae4d8aa04. Label changesLabel changes:
Label justifications:
Evidence reviewedPR surface: Source +12, Tests +289. Total +301 across 5 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
Review history (1 earlier review cycle)
|
Real behavior proof — eligibleForDreaming corpus gating confirmedEnvironment: PR worktree Entry-level evidence (three markers):
Honest delta: The PR does not change runtime exclusion behavior for archive artifacts — both builds correctly exclude them. The PR's value is (1) an explicit Test results (PR build):
Data-model / serialized-state question: Repro (repo root, either build): node scripts/run-vitest.mjs packages/memory-host-sdk/src/host/session-transcript-corpus.test.ts extensions/memory-core/src/dreaming-phases.test.ts(The entry-level probe seeds an SQLite-backed active session plus a
|
clawsweeper
Bot
added
proof: sufficient
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: fix(memory): corpus dreaming eligibility contract + heartbeat provenance test This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
1 participant
What Problem This Solves
Issue #103720 identified two corpus-contamination paths in the Dreaming pipeline.
A. Archive artifacts (deleted/reset transcripts) re-ingested by Dreaming
.jsonl.deleted.*and.jsonl.reset.*rotation files are retained in the shared corpus formemory_searchcontinuity, but Dreaming must learn only from the live session corpus. The runtime guard (da50a450d2) correctly skips these, but it operated via an ad-hocartifactKind === "archive-artifact"check inside the consuming phase —SessionTranscriptCorpusEntrycarried no expliciteligibleForDreamingsignal. Any future corpus consumer had to re-implement the exclusion rule independently, and the clawsweeper review required a first-class field, a dedicated test file, and the consuming phase to gate on the field rather than repeating the ad-hoc check.B. Natural-language heartbeat responses leaking into the dream corpus
buildSessionEntry()gained provenance-basedinsideHeartbeatTurnsuppression in571f1dcef9, correctly excluding assistant responses paired with{ kind: "internal_system", sourceTool: "heartbeat" }user messages. No integration test exercised that path through a real dreaming sweep — specifically the case where the assistant responds with natural-language text rather than the exactHEARTBEAT_OKtoken.Clawsweeper acceptance criteria (from issue #103720):
SessionTranscriptCorpusEntrymust carryeligibleForDreaming: boolean; archive artifactsfalse, active sessionstruesession-transcript-corpus.test.tsunderpackages/memory-host-sdk/src/host/covering active, deleted, reset, orphaned-deleted, and cron-lineage variantscollectSessionIngestionBatches()indreaming-phases.tsmust gate on!entry.eligibleForDreamingrather than repeatingartifactKind === "archive-artifact"provenance: { kind: "internal_system", sourceTool: "heartbeat" }user message suppresses the paired natural-language assistant response through a full dreaming sweepAll four clawsweeper acceptance criteria are met by this PR.
What This PR Does
Adds
eligibleForDreaming: booleantoSessionTranscriptCorpusEntry— set at construction time (truefor active sessions,falsefor all archive artifacts) so any corpus consumer can honour the exclusion without re-examiningartifactKind.Adds
session-transcript-corpus.test.ts(new file, 5 unit tests) covering: SQLite-backed active session (true),.deletedarchive (false),.resetarchive (false), orphaned-deleted archive with no sessions.json entry (false), and cron-lineage archive preservinggeneratedByCronRun: truealongsideeligibleForDreaming: false.Updates
dreaming-phases.tsto gate on!entry.eligibleForDreaminginstead ofentry.artifactKind === "archive-artifact", making the dreaming skip a consequence of the corpus-level contract rather than a second ad-hoc check.Adds dreaming-phases integration test that seeds a session with a provenance-authenticated heartbeat user message followed by a natural-language assistant response ("Looks like you're all set for the morning"), runs a full dreaming sweep, and asserts the heartbeat turn and its paired response are absent from corpus output while a normal user/assistant exchange in the same session is retained.
Files changed:
packages/memory-host-sdk/src/host/session-transcript-corpus.tseligibleForDreamingfield added to type;trueintoSessionStoreCorpusEntry(),falseintoArtifactCorpusEntry()packages/memory-host-sdk/src/host/session-transcript-corpus.test.tspackages/memory-host-sdk/src/host/session-files.test.tseligibleForDreaming: falseextensions/memory-core/src/dreaming-phases.tsartifactKind === "archive-artifact"to!entry.eligibleForDreamingextensions/memory-core/src/dreaming-phases.test.tsseedDreamingSessionTranscript()accepts optionalprovenance; new e2e heartbeat-provenance sweep test addedEvidence
Rebased HEAD:
beedbb7076c69eb04ae1a8ccb2d492086ded182bCurrent upstream/main:
78b4a4afedf33e39a1b64eee07ffa7543abe79efMutation check — BEFORE (src reverted to upstream state)
Command:
Output:
All 5 tests fail with
eligibleForDreamingabsent — the field did not exist before this PR.Mutation check — AFTER (fix commit
beedbb7076)Command:
Output:
Full test suite on fix commit
Commands:
session-transcript-corpus.test.ts(NEW)dreaming-phases.test.ts(+1 new e2e)session-files.test.tsTypecheck (
node scripts/run-tsgo.mjs): the only errors are pre-existing onupstream/main(packages/net-policy/src/ip.tslines 67/69 —"benchmarking"/"orchid2"type mismatches); not caused by this change.git diff --check: clean.Notes
Legacy-provenance window.
insideHeartbeatTurnsuppression inbuildSessionEntry()relies on provenance metadata being present in persisted JSONL/SQLite records. Session files written before571f1dcef9(2026-07-17) were not rewritten with provenance, so natural-language heartbeat responses from those sessions may remain eligible for Dreaming ingestion until they rotate out naturally. This PR does not backfill legacy sessions: backfilling would require rewriting all existing JSONL/SQLite records to inject provenance that was never captured, risking data loss on sessions that have genuine content mixed with heartbeat turns and no reliable way to distinguish them retrospectively. The gap is transient — sessions rotate on their normal schedule — and does not affect new sessions from571f1dcef9onward.No text-fallback reintroduced.
571f1dcef9deliberately removed user-spoofable text matching (seesession-files.test.ts: "does not couple user-spoofed heartbeat text to the next assistant response"). This PR does not reintroduce text-based detection.AI-assisted development. This change was developed with Claude Code assistance (claude-sonnet-4-6). All logic reviewed and validated manually; mutation test run and outputs verified by hand.
Fixes #103720
Real behavior proof (entry-level, both builds — added after triage)
Probe on PR head
beedbb7076vs merge-base78b4a4afed: seeded an SQLite-backed active session, a.jsonl.deleted.archive artifact, and a heartbeat-provenance turn; calledlistSessionTranscriptCorpusEntriesForAgentdirectly on each build.eligibleForDreaming: trueartifactKindcheckeligibleForDreaming: false, excluded via field gatebuildSessionEntryfilterHonest delta: runtime exclusion behavior is unchanged — the PR contributes the explicit contract, the property-driven gate, and the coverage (per the four acceptance criteria above).
Serialized-state flag resolved:
SessionTranscriptCorpusEntryis a runtime-computed view, never persisted; the persisted dreaming state (writeSessionIngestionState,dreaming-phases.ts:587) is a separate{version, files, seenMessages}type, andJSON.stringify.*eligibleForDreaminggreps to zero in production code. No migration surface.Full probe detail in the comment below.