fix(memory-core): exclude archived transcripts from Dreaming session corpus by xiaobao-k8s · Pull Request #90433 · openclaw/openclaw

xiaobao-k8s · 2026-06-04T18:14:19Z

Summary

Archived session transcripts (.jsonl.reset.* and .jsonl.deleted.*) are
currently ingested into the Dreaming corpus, polluting analysis with
reset/deleted cron-spawned subagent noise. On a cron-heavy deployment,
this causes 67-91% of the daily corpus to be dominated by automation
noise, suppressing pattern discovery. This is particularly visible on
workspaces where cron jobs frequently spawn subagents whose transcripts get
reset/deleted but remain in the session directory as usage-counted archives.

Changes

Reuse the SDK-exported isSessionArchiveArtifactName classifier so Dreaming
follows the canonical archive filename contract instead of carrying a local regex
Add archive-exclusion filter in collectSessionIngestionBatches so
Dreaming corpus assembly skips archived transcripts
Preserve memory_search archive indexing path unchanged (only Dreaming
ingestion is affected)
Add corpus-level regression coverage for fresh ordinary reset archives,
including archive timestamps without milliseconds (...T18-10-00Z)

Real behavior proof

Behavior or issue addressed: Before the fix, Dreaming session-corpus assembly could ingest usage-counted reset/deleted archive transcripts and, after refreshing against current main, could also fail to preserve skipped-transcript checkpoint semantics for generated dreaming/isolated-cron transcripts. After the fix, Dreaming excludes session archive artifacts from the session corpus, keeps generated dreaming/cron run transcripts out of corpus output, preserves skipped checkpoints with zero ingested content lines, and keeps legacy JSON sessions.json classification compatible with the current SQLite session-store loader.
Real environment tested: Repository openclaw/openclaw, branch fix/dreaming-exclude-archives, commit 61fd2d6f9cc5, local Linux worktree /datad/github/openclaw-issue90313 refreshed against current origin/main, using real OpenClaw session storage under /root/.openclaw/agents for the patched Dreaming storage proof and a temp workspace/plugin-state store for output isolation.
Exact steps or command run after this patch:

git diff --check
pnpm exec oxlint extensions/memory-core/src/dreaming-phases.ts packages/memory-host-sdk/src/host/session-files.ts
OPENCLAW_TEST_FAST=1 node scripts/test-projects.mjs extensions/memory-core/src/dreaming-phases.test.ts packages/memory-host-sdk/src/host/session-files.test.ts
./node_modules/.bin/tsx /tmp/openclaw-90433-real-storage-proof.ts > /tmp/openclaw-90433-real-storage-proof-clean.json
jq '{realStateRoot,totalJsonl,archiveCount,liveCount,agentCount,corpusFileCount:(.corpusFiles|length),liveLineCount,archiveAbsence: [.archiveAbsence[] | {agent,file,absent}][0:5]}' /tmp/openclaw-90433-real-storage-proof-clean.json

Evidence after fix: Terminal output from the verification:

=== diff --check ===
=== oxlint changed files ===
Found 0 warnings and 0 errors.
Finished in 49ms on 2 files with 208 rules using 8 threads.
=== focused tests quick rerun ===
packages/memory-host-sdk/src/host/session-files.test.ts: 12 passed
extensions/memory-core/src/dreaming-phases.test.ts: 45 passed
[test] passed 2 Vitest shards in 112.86s

Real storage proof summary:
{
  "realStateRoot": "/root/.openclaw/agents",
  "totalJsonl": 627,
  "archiveCount": 73,
  "liveCount": 554,
  "agentCount": 7,
  "corpusFileCount": 17,
  "liveLineCount": 240,
  "archiveAbsence": [
    { "agent": "hr-agent", "file": "30d9dd9e-8281-48eb-bf65-c2d4086d7163.jsonl.reset.2026-05-31T02-45-36.108Z", "absent": true },
    { "agent": "hr-agent", "file": "37325983-4da0-49cc-8747-1bdcf4b0342a.jsonl.reset.2026-06-03T08-50-31.129Z", "absent": true },
    { "agent": "hr-agent", "file": "3b72e1cc-a0c0-4378-92fa-473dfe21551a.jsonl.reset.2026-05-23T13-55-57.715Z", "absent": true },
    { "agent": "hr-agent", "file": "43fb8e14-c895-46c0-a5eb-4eae0014cbbd.jsonl.reset.2026-05-29T23-16-37.488Z", "absent": true },
    { "agent": "hr-agent", "file": "53c23768-32e0-4772-9f8a-0944bde75f52.jsonl.reset.2026-05-29T00-41-25.765Z", "absent": true }
  ]
}

Observed result after fix: The refreshed branch builds a Dreaming session corpus from live real session storage while sampled .jsonl.reset.* archive transcripts remain absent from the corpus; generated dreaming and isolated cron transcript checkpoints are retained as zero ingested content lines; focused regression suites pass on the refreshed main branch.
What was not tested: Full repository test suite and full production gateway run were not executed; validation was limited to changed-file lint, focused memory-core/SDK regressions, and a patched Dreaming run over real local session storage with redacted/archive-only file identifiers.

Test Plan

git diff --check
pnpm exec oxlint extensions/memory-core/src/dreaming-phases.ts extensions/memory-core/src/dreaming-phases.test.ts — 0 warnings, 0 errors
OPENCLAW_TEST_FAST=1 node scripts/test-projects.mjs extensions/memory-core/src/dreaming-phases.test.ts — 45 tests, all passing
OPENCLAW_TEST_FAST=1 node scripts/test-projects.mjs packages/memory-host-sdk/src/host/session-files.test.ts — 12 tests, all passing

Risk

Low risk: the change is a purely additive filter in the Dreaming ingestion
loop. Archive files are still indexed for memory_search through the
unchanged listSessionFilesForAgent → buildSessionEntry path used by
manager-sync-ops.ts and qmd-manager.ts.
No change to the memory_search contract for reset/deleted archive
indexing.

Related Issue

Refs #90313 (archive transcript Dreaming corpus pollution).
Note: the broader cron-descended subagent classification issue mentioned in
the linked report is outside this patch's scope and remains tracked separately.

Follow-up after ClawSweeper review

Removed the runtime legacy sessions.json fallback from packages/memory-host-sdk/src/host/session-files.ts; SQLite session metadata remains the canonical runtime source.
Kept the Dreaming archive transcript exclusion and checkpoint-alignment changes unchanged.
Re-ran focused validation after the fallback removal:
- git diff --check — passed
- pnpm exec oxlint extensions/memory-core/src/dreaming-phases.ts extensions/memory-core/src/dreaming-phases.test.ts — 0 warnings, 0 errors
- OPENCLAW_TEST_FAST=1 node scripts/test-projects.mjs extensions/memory-core/src/dreaming-phases.test.ts — 45/45 passed
- OPENCLAW_TEST_FAST=1 node scripts/test-projects.mjs packages/memory-host-sdk/src/host/session-files.test.ts — 12/12 passed

…corpus Archived session transcripts (.jsonl.reset.* and .jsonl.deleted.*) are currently ingested into the Dreaming corpus, polluting analysis with reset/deleted cron-spawned subagent noise. On a cron-heavy deployment, this causes 67-91% of the daily corpus to be dominated by automation noise, suppressing pattern discovery. Add a Dreaming-specific archive-exclusion boundary: a new isSessionArchiveTranscriptPath helper detects .jsonl.reset.* and .jsonl.deleted.* archives, and collectSessionIngestionBatches skips them during corpus assembly. This preserves memory_search archive indexing via the existing listSessionFilesForAgent path, which is unchanged for non-Dreaming callers. Fixes openclaw#90313

clawsweeper · 2026-06-04T18:21:13Z

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Close: this PR is now superseded by the merged session-corpus metadata implementation, which covers the archive exclusion and the broader cron-lineage defect on current main.

Root-cause cluster
Relationship: superseded
Canonical: #96517
Summary: This PR is superseded by the merged maintainer-owned replacement that closed the canonical Dreaming corpus pollution issue and moved the fix to the session transcript corpus seam.

Members:

canonical: fix(memory-core): route dreaming corpus through session metadata #96517 - Merged replacement implements archive exclusion and cron lineage at the maintained corpus metadata seam.
fixed_by_candidate: Dreaming session-corpus: cron classification doesn't follow subagent parentage; archived (.deleted/.reset) transcripts re-ingested #90313 - The canonical issue reported both archive re-ingestion and cron-descended subagent classification and was closed by the merged replacement.
superseded: fix(memory-core): exclude archive transcripts from dreaming corpus and propagate cron parentage to subagents #93187 - The broader open attempt overlaps the same root cause but is also superseded by the merged maintainer-owned implementation.
adjacent_distinct: Dreaming needs supported guard to disable session transcript ingestion during QMD migrations #87689 - The operator guard request concerns disabling transcript ingestion during migrations, which is adjacent but not the same branch-level fix.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Canonical path: Keep the merged #96517 implementation as the canonical path and close this conflicting archive-only branch.

So I’m closing this here and keeping the remaining discussion on #96517.

Review details

Best possible solution:

Keep the merged #96517 implementation as the canonical path and close this conflicting archive-only branch.

Do we have a high-confidence way to reproduce the issue?

No for current main: source inspection shows Dreaming now skips archive-artifact corpus entries and classifies cron-descended session lineage before ingestion. The original pre-fix failure path is documented in the PR diff and linked issue, but it is superseded on current main.

Is this the best way to solve the issue?

No for this branch now. It was a plausible archive-only fix, but the best current solution is the merged session-transcript-corpus metadata seam from #96517 because it also handles cron lineage and avoids a duplicate Dreaming-local source of truth.

Security review:

Security review cleared: The PR changes memory-core TypeScript ingestion logic and tests only; no workflow, dependency, lockfile, script, credential, or third-party execution surface is changed.

AGENTS.md: found and applied where relevant.

What I checked:

Repository policy read: Root AGENTS.md and scoped extensions/AGENTS.md were read fully; their review policy mattered because this is bundled-plugin session-state work and the current implementation belongs at the SDK/plugin seam. (AGENTS.md:17, 6cb82eaab865)
Current main Dreaming archive filter: Current main no longer queues reset/delete archive artifacts for Dreaming: collectSessionIngestionBatches reads corpus entries and skips entry.artifactKind === "archive-artifact" before appending corpus lines. (extensions/memory-core/src/dreaming-phases.ts:851, 6cb82eaab865)
Current main corpus metadata seam: Current main keeps reset/delete archive artifacts in the shared corpus while classifying cron-descended children through spawnedBy/parentSessionKey closure, so Dreaming gets a richer maintained boundary than this archive-only branch. (packages/memory-host-sdk/src/host/session-transcript-corpus.ts:189, 6cb82eaab865)
Regression coverage on main: Current main has a Dreaming regression that seeds archive, checkpoint, cron, heartbeat, and ordinary transcript content and asserts only the useful ordinary content enters memory/.dreams/session-corpus. (extensions/memory-core/src/dreaming-phases.test.ts:1680, 6cb82eaab865)
Superseding merged PR: fix(memory-core): route dreaming corpus through session metadata #96517 is merged, closes Dreaming session-corpus: cron classification doesn't follow subagent parentage; archived (.deleted/.reset) transcripts re-ingested #90313, and states that it preserves shared archive indexing while preventing Dreaming archive re-ingestion and classifying cron-descended child transcripts. (da50a450d271)
Current PR diff is the old local shape: This PR's diff still adds a local Dreaming archive classifier/filter and test, which is now duplicated by the merged corpus-metadata seam on current main. (extensions/memory-core/src/dreaming-phases.ts:860, 3e587d859993)

Likely related people:

jalehman: Authored the merged replacement fix(memory-core): route dreaming corpus through session metadata #96517 and earlier session-store Dreaming skip work in fix(memory-core): skip dreaming transcript ingestion via session store #67315. (role: merged replacement author and adjacent feature contributor; confidence: high; commits: cc7fac26ee98, a1b01f0281d0; files: extensions/memory-core/src/dreaming-phases.ts, packages/memory-host-sdk/src/host/session-transcript-corpus.ts, packages/memory-host-sdk/src/host/session-files.ts)
xialonglee: Authored the broader open attempt fix(memory-core): exclude archive transcripts from dreaming corpus and propagate cron parentage to subagents #93187 that explored archive exclusion and cron parentage before the narrower merged replacement landed. (role: adjacent fix contributor; confidence: medium; commits: dbba69576117, d9b7c9c4db8e; files: packages/memory-host-sdk/src/host/session-transcript-corpus.ts, extensions/memory-core/src/dreaming-phases.ts)
rcrick: GitHub metadata and commit history tie reset/deleted session archives entering the usage-counted predicate to Usage: include reset and deleted session archives #43215. (role: introduced archive usage behavior; confidence: high; commits: 2fe1ff8ea8ab; files: src/config/sessions/artifacts.ts, src/infra/session-cost-usage.ts)

Codex review notes: model internal, reasoning high; reviewed against 6cb82eaab865; fix evidence: commit da50a450d271, main fix timestamp 2026-06-24T11:29:26-07:00.

…VE_TIMESTAMP_RE Make the milliseconds portion of the archive timestamp optional ((?:\.\d{3})?Z), matching the canonical contract in src/config/sessions/artifacts.ts. Previously the local regex required exactly three millisecond digits (\.\d{3}Z), which could miss valid usage-counted archive transcripts without the optional millisecond portion. The plugin SDK boundary prevents direct import of the host's isSessionArchiveArtifactName; this change aligns the local regex to the same contract.

xiaobao-k8s · 2026-06-05T02:33:40Z

@clawsweeper re-review

clawsweeper · 2026-06-05T02:33:43Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/26991862145
Updated: 2026-06-05T02:43:53.886Z

xiaobao-k8s · 2026-06-08T02:43:06Z

@clawsweeper re-review

clawsweeper · 2026-06-08T02:43:09Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/27113039206
Updated: 2026-06-08T02:51:33.763Z

…archives

xiaobao-k8s · 2026-06-08T08:35:52Z

Updated after the latest review and refreshed the branch against current origin/main.

Changes since the previous proof attempt:

Merged current main into fix/dreaming-exclude-archives.
Kept the archive exclusion behavior and added the refresh-required compatibility fix for skipped dreaming / isolated-cron transcript checkpoints.
Added legacy JSON sessions.json classification fallback for the memory-host SDK path now that the session store loader is SQLite-first.
Replaced the PR body proof with a patched Dreaming run over real local session storage (/root/.openclaw/agents) and redacted archive-absence evidence.

Local verification on head 61fd2d6f9cc5:

git diff --check
pnpm exec oxlint extensions/memory-core/src/dreaming-phases.ts packages/memory-host-sdk/src/host/session-files.ts → 0 errors
OPENCLAW_TEST_FAST=1 node scripts/test-projects.mjs extensions/memory-core/src/dreaming-phases.test.ts packages/memory-host-sdk/src/host/session-files.test.ts → 57 tests passed
Real storage proof: 627 jsonl files scanned, 73 archive files detected/excluded, 240 live corpus lines produced, sampled archive transcripts absent: true.

@clawsweeper re-review

clawsweeper · 2026-06-08T08:35:55Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/27125740852
Updated: 2026-06-08T08:45:08.993Z

xiaobao-k8s · 2026-07-07T02:08:55Z

Closing per ClawSweeper review: this PR has been superseded by #96517 (session-corpus metadata implementation). Thanks for the contribution!

openclaw-barnacle Bot added extensions: memory-core Extension: memory-core size: XS proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 4, 2026

openclaw-barnacle Bot added triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. and removed proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 5, 2026

clawsweeper Bot added the merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. label Jun 5, 2026

clawsweeper Bot mentioned this pull request Jun 6, 2026

fix(memory-core): exclude soft-deleted .jsonl.deleted paths from dreaming corpus (#90466) #90839

Closed

fix(memory-core): reuse canonical archive classifier for dreaming

d48c896

clawsweeper Bot removed the merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. label Jun 7, 2026

openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 7, 2026

xiaobao-k8s added 2 commits June 8, 2026 16:21

Merge remote-tracking branch 'origin/main' into fix/dreaming-exclude-…

519eca3

…archives

fix(memory-core): preserve skipped transcript checkpoints after refresh

61fd2d6

openclaw-barnacle Bot added size: S and removed size: XS labels Jun 8, 2026

fix(memory-core): remove legacy session fallback

3e587d8

openclaw-barnacle Bot added size: XS and removed size: S proof: sufficient ClawSweeper judged the real behavior proof convincing. labels Jun 9, 2026

clawsweeper Bot mentioned this pull request Jun 24, 2026

fix(memory-core): route dreaming corpus through session metadata #96517

Merged

xiaobao-k8s closed this Jul 7, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(memory-core): exclude archived transcripts from Dreaming session corpus#90433

fix(memory-core): exclude archived transcripts from Dreaming session corpus#90433
xiaobao-k8s wants to merge 6 commits into
openclaw:mainfrom
xiaobao-k8s:fix/dreaming-exclude-archives

xiaobao-k8s commented Jun 4, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 4, 2026 •

edited

Loading

Uh oh!

xiaobao-k8s commented Jun 5, 2026

Uh oh!

clawsweeper Bot commented Jun 5, 2026 •

edited

Loading

Uh oh!

xiaobao-k8s commented Jun 8, 2026

Uh oh!

clawsweeper Bot commented Jun 8, 2026 •

edited

Loading

Uh oh!

xiaobao-k8s commented Jun 8, 2026

Uh oh!

clawsweeper Bot commented Jun 8, 2026 •

edited

Loading

Uh oh!

xiaobao-k8s commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

xiaobao-k8s commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Real behavior proof

Test Plan

Risk

Related Issue

Follow-up after ClawSweeper review

Uh oh!

clawsweeper Bot commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xiaobao-k8s commented Jun 5, 2026

Uh oh!

clawsweeper Bot commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xiaobao-k8s commented Jun 8, 2026

Uh oh!

clawsweeper Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xiaobao-k8s commented Jun 8, 2026

Uh oh!

clawsweeper Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xiaobao-k8s commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

xiaobao-k8s commented Jun 4, 2026 •

edited

Loading

clawsweeper Bot commented Jun 4, 2026 •

edited

Loading

clawsweeper Bot commented Jun 5, 2026 •

edited

Loading

clawsweeper Bot commented Jun 8, 2026 •

edited

Loading

clawsweeper Bot commented Jun 8, 2026 •

edited

Loading