Skip to content

fix(memory-core): skip dreaming transcript ingestion via session store#67315

Merged
jalehman merged 3 commits into
mainfrom
openclaw-8f8/dreaming-session-store-ingestion
Apr 15, 2026
Merged

fix(memory-core): skip dreaming transcript ingestion via session store#67315
jalehman merged 3 commits into
mainfrom
openclaw-8f8/dreaming-session-store-ingestion

Conversation

@jalehman

Copy link
Copy Markdown
Contributor

What

This PR teaches session transcript ingestion to classify dreaming-narrative-* transcripts from the sibling sessions.json store before reading transcript content. That lets the memory pipeline skip dream diary prompt/prose transcripts even when ingestion sees the JSONL file before the trailing bootstrap record lands.

Why

Dreaming narrative sessions were being ingested into memory/.dreams/session-corpus/*.txt during that timing window, which polluted Light Sleep reflection candidates and made daily memory files accumulate recursive dreaming junk.

Changes

  • Detect dreaming transcripts from session store
  • Keep bootstrap marker detection as fallback
  • Add unit regression for pre-bootstrap classification
  • Add dreaming ingestion regression for the timing race

Testing

  • pnpm test src/memory-host-sdk/host/session-files.test.ts extensions/memory-core/src/dreaming-phases.test.ts
  • Expected: 9/9 and 22/22 passing

@aisle-research-bot

aisle-research-bot Bot commented Apr 15, 2026

Copy link
Copy Markdown

🔒 Aisle Security Analysis

We found 3 potential security issue(s) in this PR:

# Severity Title
1 🟠 High Prototype Pollution via untrusted keys in sessions.json loaded by loadSessionStore()
2 🟠 High Session-store poisoning can mark arbitrary transcripts as dreaming and suppress ingestion/audit
3 🟡 Medium Event-loop blocking / I/O amplification in buildSessionEntry via synchronous sessions.json reads
1. 🟠 Prototype Pollution via untrusted keys in sessions.json loaded by loadSessionStore()
Property Value
Severity High
CWE CWE-1321
Location src/config/sessions/store-load.ts:54-63

Description

loadSessionStore() parses sessions.json using JSON.parse into a plain object and then normalizes it by iterating keys and writing back into the same object (store[key] = normalized). If an attacker can influence sessions.json, they can include a "__proto__" (or similar) key. Assigning to store["__proto__"] will mutate the prototype of store (and potentially other objects), causing prototype pollution.

Why this is a security issue:

  • Input: attacker-controlled JSON keys from sessions.json
  • Mutation sink: store[key] = normalized where key can be "__proto__", "constructor", etc.
  • Impact: prototype pollution can lead to unexpected behavior, authorization bypasses, crashes/DoS, or (depending on downstream usage patterns) code execution.

Vulnerable code:

for (const [key, entry] of Object.entries(store)) {
  ...
  store[key] = normalized;
}

Recommendation

Harden session store parsing/normalization against special keys.

Options:

  1. Use a null-prototype object when storing parsed data and ignore dangerous keys:
const parsed = JSON.parse(raw);
const store: Record<string, SessionEntry> = Object.create(null);

for (const [k, v] of Object.entries(parsed as Record<string, unknown>)) {
  if (k === "__proto__" || k === "prototype" || k === "constructor") continue;
  (store as any)[k] = v;
}
  1. Never assign back into an object using untrusted keys; instead, build a new sanitized object:
export function normalizeSessionStore(input: Record<string, SessionEntry>): Record<string, SessionEntry> {
  const out: Record<string, SessionEntry> = Object.create(null);
  for (const [key, entry] of Object.entries(input)) {
    if (key === "__proto__" || key === "prototype" || key === "constructor") continue;
    if (!entry) continue;
    out[key] = normalizeSessionEntryDelivery(normalizeSessionRuntimeModelFields(entry));
  }
  return out;
}

Also consider validating that session keys match an expected format (e.g., strict regex) before accepting them.

2. 🟠 Session-store poisoning can mark arbitrary transcripts as dreaming and suppress ingestion/audit
Property Value
Severity High
CWE CWE-276
Location src/memory-host-sdk/host/session-files.ts:246-284

Description

buildSessionEntry() now treats a transcript as generatedByDreamingNarrative based on untrusted data from the sibling sessions.json session store. When this flag is true, the parser skips all message lines, resulting in empty content/lineMap and preventing downstream ingestion/indexing (validated by new tests).

Impact:

  • If an attacker (or untrusted plugin/tenant) can write/modify agents/<id>/sessions/sessions.json, they can add a key that matches the dreaming-narrative prefix and point sessionFile / sessionId at any transcript.
  • The targeted transcript will be treated as internal “dreaming narrative” output and its contents will be skipped, enabling activity hiding / retention & audit bypass.

Vulnerable behavior:

  • Input: sessions.json entries (sessionKey + sessionFile/sessionId)
  • Logic: isDreamingNarrativeTranscriptFromSessionStore()generatedByDreamingNarrative
  • Sink: if (generatedByDreamingNarrative) { continue; } inside transcript parsing loop (drops all message content)

Vulnerable code:

let generatedByDreamingNarrative =
  opts.generatedByDreamingNarrative ?? isDreamingNarrativeTranscriptFromSessionStore(absPath);
...
if (generatedByDreamingNarrative) {
  continue;
}

Recommendation

Treat sessions.json as untrusted for security decisions, or ensure it is protected by strong integrity controls.

Practical fixes (pick one or combine):

  1. Derive dreaming classification from the transcript itself (e.g., only via bootstrap/custom records), not from sessions.json.
  2. If sessions.json must be used, require both:
    • transcript evidence (bootstrap/runId markers), and
    • a session-store match that is constrained to the agent’s sessions directory.
  3. Add integrity/permission checks for sessions.json (owner/perms), and in multi-tenant/plugin contexts ensure untrusted parties cannot modify it.

Example (require transcript evidence before skipping ingestion):

let generatedByDreamingNarrative = false;// First pass: detect from transcript markers only
for (...) {
  ...
  if (isDreamingNarrativeGeneratedRecord(record)) {
    generatedByDreamingNarrative = true;
  }
  ...
}// Optional: use session store only as a hint, not an authority
if (!generatedByDreamingNarrative && opts.generatedByDreamingNarrative) {// do NOT skip ingestion based solely on store
}
3. 🟡 Event-loop blocking / I/O amplification in buildSessionEntry via synchronous sessions.json reads
Property Value
Severity Medium
CWE CWE-400
Location src/memory-host-sdk/host/session-files.ts:245-248

Description

buildSessionEntry() now implicitly classifies transcripts by calling isDreamingNarrativeTranscriptFromSessionStore(absPath), which loads ${dirname(absPath)}/sessions.json via loadSessionStore().

Security impact (DoS):

  • loadSessionStore() performs a synchronous fs.readFileSync() + JSON.parse() on sessions.json.
  • buildSessionEntry() is called in batch/concurrent loops (e.g., session indexing/export). With the new default, this can cause N synchronous reads/parses of the same sessions.json, blocking the Node.js event loop and dramatically increasing CPU and disk I/O.
  • A large sessions.json (or many session files) can therefore degrade responsiveness or cause timeouts (denial of service) during indexing/sync operations.

Vulnerable code:

let generatedByDreamingNarrative =
  opts.generatedByDreamingNarrative ?? isDreamingNarrativeTranscriptFromSessionStore(absPath);

which calls into:

const raw = fs.readFileSync(storePath, "utf-8");
const parsed = JSON.parse(raw);

Note: extensions/memory-core/src/dreaming-phases.ts was updated to precompute and pass generatedByDreamingNarrative, but other call sites (e.g. sync/export paths) still call buildSessionEntry(absPath) without opts, triggering the expensive sync load repeatedly.

Recommendation

Avoid implicit synchronous store loads inside buildSessionEntry().

Options:

  1. Require callers to pass generatedByDreamingNarrative (or a preloaded path set) and remove the default store lookup.
  2. Cache/preload the dreaming transcript path set once per sessionsDir and reuse it across calls.
  3. Make store loading async and/or ensure it happens outside tight loops.

Example (caller-side preload, similar to dreaming-phases):

const dreamingPaths = loadDreamingNarrativeTranscriptPathSetForSessionsDir(sessionsDir);
for (const absPath of files) {
  const entry = await buildSessionEntry(absPath, {
    generatedByDreamingNarrative: dreamingPaths.has(normalizeSessionTranscriptPathForComparison(absPath)),
  });
}

This prevents per-file synchronous disk reads/parses and avoids event-loop blocking.


Analyzed PR: #67315 at commit 87c09b2

Last updated on: 2026-04-15T20:10:56Z

@openclaw-barnacle openclaw-barnacle Bot added extensions: memory-core Extension: memory-core size: M maintainer Maintainer-authored PR labels Apr 15, 2026
@greptile-apps

greptile-apps Bot commented Apr 15, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a timing race in dreaming transcript ingestion: when the memory pipeline encountered a dreaming-narrative-* transcript before its bootstrap record had landed, the content was being incorrectly ingested into the session corpus. The fix adds a session-store lookup in buildSessionEntry (via isDreamingNarrativeTranscriptFromSessionStore) that classifies transcripts from sessions.json before reading JSONL content, keeping the existing bootstrap-record check as a fallback.

Confidence Score: 5/5

Safe to merge; the change is additive and well-tested with no correctness issues.

No P0 or P1 findings. The single P2 note (synchronous store read per-transcript instead of once per ingestion pass) is a performance nit mitigated by the existing loadSessionStore cache. All race-condition edge cases are covered by the new regression tests.

No files require special attention.

Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/memory-host-sdk/host/session-files.ts
Line: 118-133

Comment:
**Synchronous store read called per-transcript**

`isDreamingNarrativeTranscriptFromSessionStore` calls `loadSessionStore` (which uses `fs.readFileSync`) synchronously inside what is otherwise an async function. For agents with many transcripts, this blocks the event loop once per file even though `loadSessionStore` has caching. If the cache is cold (e.g. first call after `sessions.json` is updated), each transcript forces a full synchronous JSON parse of potentially a large store. Consider hoisting the store read (or its result) to the callsite in `buildSessionEntry` so the synchronous read happens at most once per ingestion pass rather than once per file.

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "openclaw-8f8: skip dreaming transcript i..." | Re-trigger Greptile

Comment on lines +118 to +133
function isDreamingNarrativeTranscriptFromSessionStore(absPath: string): boolean {
const sessionsDir = path.dirname(absPath);
const storePath = path.join(sessionsDir, "sessions.json");
const normalizedAbsPath = normalizeComparablePath(absPath);
const store = loadSessionStore(storePath);
for (const [sessionKey, entry] of Object.entries(store)) {
if (!isDreamingNarrativeSessionStoreKey(sessionKey)) {
continue;
}
const transcriptPath = resolveSessionStoreTranscriptPath(sessionsDir, entry);
if (transcriptPath === normalizedAbsPath) {
return true;
}
}
return false;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Synchronous store read called per-transcript

isDreamingNarrativeTranscriptFromSessionStore calls loadSessionStore (which uses fs.readFileSync) synchronously inside what is otherwise an async function. For agents with many transcripts, this blocks the event loop once per file even though loadSessionStore has caching. If the cache is cold (e.g. first call after sessions.json is updated), each transcript forces a full synchronous JSON parse of potentially a large store. Consider hoisting the store read (or its result) to the callsite in buildSessionEntry so the synchronous read happens at most once per ingestion pass rather than once per file.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/memory-host-sdk/host/session-files.ts
Line: 118-133

Comment:
**Synchronous store read called per-transcript**

`isDreamingNarrativeTranscriptFromSessionStore` calls `loadSessionStore` (which uses `fs.readFileSync`) synchronously inside what is otherwise an async function. For agents with many transcripts, this blocks the event loop once per file even though `loadSessionStore` has caching. If the cache is cold (e.g. first call after `sessions.json` is updated), each transcript forces a full synchronous JSON parse of potentially a large store. Consider hoisting the store read (or its result) to the callsite in `buildSessionEntry` so the synchronous read happens at most once per ingestion pass rather than once per file.

How can I resolve this? If you propose a fix, please make it concise.

@jalehman jalehman self-assigned this Apr 15, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eccf053a99

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +105 to +110
if (typeof entry?.sessionFile === "string" && entry.sessionFile.trim().length > 0) {
const sessionFile = entry.sessionFile.trim();
const resolved = path.isAbsolute(sessionFile)
? sessionFile
: path.resolve(sessionsDir, sessionFile);
return normalizeComparablePath(resolved);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Fallback to sessionId when sessionFile metadata is stale

This helper returns immediately when entry.sessionFile is present, so it never falls back to sessionId if that stored path is stale or no longer points at the current transcript. We already support stale sessionFile metadata elsewhere by deriving from sessionId, so this path can miss dreaming transcripts in exactly the pre-bootstrap window this change is trying to cover, allowing dream prompt/prose lines to be ingested as normal session corpus.

Useful? React with 👍 / 👎.

Comment thread src/memory-host-sdk/host/session-files.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b378118950

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +101 to +103
function normalizeComparablePath(pathname: string): string {
const resolved = path.resolve(pathname);
return process.platform === "win32" ? resolved.toLowerCase() : resolved;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Canonicalize transcript paths before dreaming-set comparisons

normalizeComparablePath only applies path.resolve (plus lowercasing on Windows), but session-store sessionFile paths can be canonicalized through realpath resolution (via resolveSessionFilePath), while listSessionFilesForAgent builds paths from the unresolved sessions directory string. In environments where the state/sessions directory is accessed through a symlink, the same transcript can appear as two different absolute strings (symlink path vs real path), so dreaming transcripts are not recognized from sessions.json and can still be ingested during the pre-bootstrap window this change is meant to prevent.

Useful? React with 👍 / 👎.

@jalehman
jalehman force-pushed the openclaw-8f8/dreaming-session-store-ingestion branch from d37bd30 to 006be87 Compare April 15, 2026 18:18

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 006be871cc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

const messageTimestampsMs: number[] = [];
let generatedByDreamingNarrative = false;
let generatedByDreamingNarrative =
opts.generatedByDreamingNarrative ?? isDreamingNarrativeTranscriptFromSessionStore(absPath);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Cache dreaming-store lookup in buildSessionEntry

buildSessionEntry now defaults to isDreamingNarrativeTranscriptFromSessionStore(absPath) when no option is provided, and that path loads sessions.json and iterates all store entries for every file. We still call buildSessionEntry without options in high-volume loops (extensions/memory-core/src/memory/manager-sync-ops.ts:821 and extensions/memory-core/src/memory/qmd-manager.ts:2035), so large reindex/export runs now add O(files × store entries) synchronous work and can noticeably stall indexing on workspaces with many sessions. Passing a precomputed lookup (or caching inside buildSessionEntry) is needed to avoid this regression.

Useful? React with 👍 / 👎.

@jalehman
jalehman force-pushed the openclaw-8f8/dreaming-session-store-ingestion branch from 006be87 to 87c09b2 Compare April 15, 2026 20:08
@jalehman
jalehman merged commit a1b01f0 into main Apr 15, 2026
27 checks passed
@jalehman
jalehman deleted the openclaw-8f8/dreaming-session-store-ingestion branch April 15, 2026 20:09
@jalehman

Copy link
Copy Markdown
Contributor Author

Merged via squash.

Thanks @jalehman!

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 87c09b2a75

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +93 to +95
if (firstSeparator < 0) {
return trimmed.startsWith(DREAMING_NARRATIVE_RUN_PREFIX);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restrict dreaming key match to internal run IDs

The new session-store classifier marks any key that starts with dreaming-narrative- as dreaming, including plain explicit session keys (firstSeparator < 0 path). Explicit keys are accepted verbatim by session-key resolution, so a user-chosen key like dreaming-narrative-notes will now be treated as an internal dreaming run and buildSessionEntry will skip all its message text, causing that conversation to disappear from session ingestion/export. Please match only the internal dreaming run-id shape (or other trusted internal metadata) instead of a broad prefix test on arbitrary session keys.

Useful? React with 👍 / 👎.

xudaiyanzi pushed a commit to xudaiyanzi/openclaw that referenced this pull request Apr 17, 2026
openclaw#67315)

Merged via squash.

Prepared head SHA: 87c09b2
Co-authored-by: jalehman <[email protected]>
Co-authored-by: jalehman <[email protected]>
Reviewed-by: @jalehman
kamskans pushed a commit to kamskans/openclaw that referenced this pull request Apr 17, 2026
openclaw#67315)

Merged via squash.

Prepared head SHA: 87c09b2
Co-authored-by: jalehman <[email protected]>
Co-authored-by: jalehman <[email protected]>
Reviewed-by: @jalehman
kvnkho pushed a commit to kvnkho/openclaw that referenced this pull request Apr 17, 2026
openclaw#67315)

Merged via squash.

Prepared head SHA: 87c09b2
Co-authored-by: jalehman <[email protected]>
Co-authored-by: jalehman <[email protected]>
Reviewed-by: @jalehman
lovewanwan pushed a commit to lovewanwan/openclaw that referenced this pull request Apr 28, 2026
openclaw#67315)

Merged via squash.

Prepared head SHA: 87c09b2
Co-authored-by: jalehman <[email protected]>
Co-authored-by: jalehman <[email protected]>
Reviewed-by: @jalehman
ogt-redknie pushed a commit to ogt-redknie/OPENX that referenced this pull request May 2, 2026
openclaw#67315)

Merged via squash.

Prepared head SHA: 87c09b2
Co-authored-by: jalehman <[email protected]>
Co-authored-by: jalehman <[email protected]>
Reviewed-by: @jalehman
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 9, 2026
openclaw#67315)

Merged via squash.

Prepared head SHA: 87c09b2
Co-authored-by: jalehman <[email protected]>
Co-authored-by: jalehman <[email protected]>
Reviewed-by: @jalehman
globalcaos pushed a commit to globalcaos/tinkerclaw that referenced this pull request May 13, 2026
openclaw#67315)

Merged via squash.

Prepared head SHA: 87c09b2
Co-authored-by: jalehman <[email protected]>
Co-authored-by: jalehman <[email protected]>
Reviewed-by: @jalehman
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
openclaw#67315)

Merged via squash.

Prepared head SHA: 87c09b2
Co-authored-by: jalehman <[email protected]>
Co-authored-by: jalehman <[email protected]>
Reviewed-by: @jalehman
jameslcowan pushed a commit to jameslcowan/openclaw that referenced this pull request Jun 2, 2026
openclaw#67315)

Merged via squash.

Prepared head SHA: 87c09b2
Co-authored-by: jalehman <[email protected]>
Co-authored-by: jalehman <[email protected]>
Reviewed-by: @jalehman
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: memory-core Extension: memory-core maintainer Maintainer-authored PR size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant