fix(memory-core): skip dreaming transcript ingestion via session store#67315
Conversation
🔒 Aisle Security AnalysisWe found 3 potential security issue(s) in this PR:
1. 🟠 Prototype Pollution via untrusted keys in sessions.json loaded by loadSessionStore()
Description
Why this is a security issue:
Vulnerable code: for (const [key, entry] of Object.entries(store)) {
...
store[key] = normalized;
}RecommendationHarden session store parsing/normalization against special keys. Options:
const parsed = JSON.parse(raw);
const store: Record<string, SessionEntry> = Object.create(null);
for (const [k, v] of Object.entries(parsed as Record<string, unknown>)) {
if (k === "__proto__" || k === "prototype" || k === "constructor") continue;
(store as any)[k] = v;
}
export function normalizeSessionStore(input: Record<string, SessionEntry>): Record<string, SessionEntry> {
const out: Record<string, SessionEntry> = Object.create(null);
for (const [key, entry] of Object.entries(input)) {
if (key === "__proto__" || key === "prototype" || key === "constructor") continue;
if (!entry) continue;
out[key] = normalizeSessionEntryDelivery(normalizeSessionRuntimeModelFields(entry));
}
return out;
}Also consider validating that session keys match an expected format (e.g., strict regex) before accepting them. 2. 🟠 Session-store poisoning can mark arbitrary transcripts as dreaming and suppress ingestion/audit
Description
Impact:
Vulnerable behavior:
Vulnerable code: let generatedByDreamingNarrative =
opts.generatedByDreamingNarrative ?? isDreamingNarrativeTranscriptFromSessionStore(absPath);
...
if (generatedByDreamingNarrative) {
continue;
}RecommendationTreat Practical fixes (pick one or combine):
Example (require transcript evidence before skipping ingestion): let generatedByDreamingNarrative = false;
// First pass: detect from transcript markers only
for (...) {
...
if (isDreamingNarrativeGeneratedRecord(record)) {
generatedByDreamingNarrative = true;
}
...
}
// Optional: use session store only as a hint, not an authority
if (!generatedByDreamingNarrative && opts.generatedByDreamingNarrative) {
// do NOT skip ingestion based solely on store
}3. 🟡 Event-loop blocking / I/O amplification in buildSessionEntry via synchronous sessions.json reads
Description
Security impact (DoS):
Vulnerable code: let generatedByDreamingNarrative =
opts.generatedByDreamingNarrative ?? isDreamingNarrativeTranscriptFromSessionStore(absPath);which calls into: const raw = fs.readFileSync(storePath, "utf-8");
const parsed = JSON.parse(raw);Note: RecommendationAvoid implicit synchronous store loads inside Options:
Example (caller-side preload, similar to dreaming-phases): const dreamingPaths = loadDreamingNarrativeTranscriptPathSetForSessionsDir(sessionsDir);
for (const absPath of files) {
const entry = await buildSessionEntry(absPath, {
generatedByDreamingNarrative: dreamingPaths.has(normalizeSessionTranscriptPathForComparison(absPath)),
});
}This prevents per-file synchronous disk reads/parses and avoids event-loop blocking. Analyzed PR: #67315 at commit Last updated on: 2026-04-15T20:10:56Z |
Greptile SummaryThis PR fixes a timing race in dreaming transcript ingestion: when the memory pipeline encountered a Confidence Score: 5/5Safe to merge; the change is additive and well-tested with no correctness issues. No P0 or P1 findings. The single P2 note (synchronous store read per-transcript instead of once per ingestion pass) is a performance nit mitigated by the existing No files require special attention. Prompt To Fix All With AIThis is a comment left during a code review.
Path: src/memory-host-sdk/host/session-files.ts
Line: 118-133
Comment:
**Synchronous store read called per-transcript**
`isDreamingNarrativeTranscriptFromSessionStore` calls `loadSessionStore` (which uses `fs.readFileSync`) synchronously inside what is otherwise an async function. For agents with many transcripts, this blocks the event loop once per file even though `loadSessionStore` has caching. If the cache is cold (e.g. first call after `sessions.json` is updated), each transcript forces a full synchronous JSON parse of potentially a large store. Consider hoisting the store read (or its result) to the callsite in `buildSessionEntry` so the synchronous read happens at most once per ingestion pass rather than once per file.
How can I resolve this? If you propose a fix, please make it concise.Reviews (1): Last reviewed commit: "openclaw-8f8: skip dreaming transcript i..." | Re-trigger Greptile |
| function isDreamingNarrativeTranscriptFromSessionStore(absPath: string): boolean { | ||
| const sessionsDir = path.dirname(absPath); | ||
| const storePath = path.join(sessionsDir, "sessions.json"); | ||
| const normalizedAbsPath = normalizeComparablePath(absPath); | ||
| const store = loadSessionStore(storePath); | ||
| for (const [sessionKey, entry] of Object.entries(store)) { | ||
| if (!isDreamingNarrativeSessionStoreKey(sessionKey)) { | ||
| continue; | ||
| } | ||
| const transcriptPath = resolveSessionStoreTranscriptPath(sessionsDir, entry); | ||
| if (transcriptPath === normalizedAbsPath) { | ||
| return true; | ||
| } | ||
| } | ||
| return false; | ||
| } |
There was a problem hiding this comment.
Synchronous store read called per-transcript
isDreamingNarrativeTranscriptFromSessionStore calls loadSessionStore (which uses fs.readFileSync) synchronously inside what is otherwise an async function. For agents with many transcripts, this blocks the event loop once per file even though loadSessionStore has caching. If the cache is cold (e.g. first call after sessions.json is updated), each transcript forces a full synchronous JSON parse of potentially a large store. Consider hoisting the store read (or its result) to the callsite in buildSessionEntry so the synchronous read happens at most once per ingestion pass rather than once per file.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/memory-host-sdk/host/session-files.ts
Line: 118-133
Comment:
**Synchronous store read called per-transcript**
`isDreamingNarrativeTranscriptFromSessionStore` calls `loadSessionStore` (which uses `fs.readFileSync`) synchronously inside what is otherwise an async function. For agents with many transcripts, this blocks the event loop once per file even though `loadSessionStore` has caching. If the cache is cold (e.g. first call after `sessions.json` is updated), each transcript forces a full synchronous JSON parse of potentially a large store. Consider hoisting the store read (or its result) to the callsite in `buildSessionEntry` so the synchronous read happens at most once per ingestion pass rather than once per file.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eccf053a99
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (typeof entry?.sessionFile === "string" && entry.sessionFile.trim().length > 0) { | ||
| const sessionFile = entry.sessionFile.trim(); | ||
| const resolved = path.isAbsolute(sessionFile) | ||
| ? sessionFile | ||
| : path.resolve(sessionsDir, sessionFile); | ||
| return normalizeComparablePath(resolved); |
There was a problem hiding this comment.
Fallback to sessionId when sessionFile metadata is stale
This helper returns immediately when entry.sessionFile is present, so it never falls back to sessionId if that stored path is stale or no longer points at the current transcript. We already support stale sessionFile metadata elsewhere by deriving from sessionId, so this path can miss dreaming transcripts in exactly the pre-bootstrap window this change is trying to cover, allowing dream prompt/prose lines to be ingested as normal session corpus.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b378118950
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| function normalizeComparablePath(pathname: string): string { | ||
| const resolved = path.resolve(pathname); | ||
| return process.platform === "win32" ? resolved.toLowerCase() : resolved; |
There was a problem hiding this comment.
Canonicalize transcript paths before dreaming-set comparisons
normalizeComparablePath only applies path.resolve (plus lowercasing on Windows), but session-store sessionFile paths can be canonicalized through realpath resolution (via resolveSessionFilePath), while listSessionFilesForAgent builds paths from the unresolved sessions directory string. In environments where the state/sessions directory is accessed through a symlink, the same transcript can appear as two different absolute strings (symlink path vs real path), so dreaming transcripts are not recognized from sessions.json and can still be ingested during the pre-bootstrap window this change is meant to prevent.
Useful? React with 👍 / 👎.
d37bd30 to
006be87
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 006be871cc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const messageTimestampsMs: number[] = []; | ||
| let generatedByDreamingNarrative = false; | ||
| let generatedByDreamingNarrative = | ||
| opts.generatedByDreamingNarrative ?? isDreamingNarrativeTranscriptFromSessionStore(absPath); |
There was a problem hiding this comment.
Cache dreaming-store lookup in buildSessionEntry
buildSessionEntry now defaults to isDreamingNarrativeTranscriptFromSessionStore(absPath) when no option is provided, and that path loads sessions.json and iterates all store entries for every file. We still call buildSessionEntry without options in high-volume loops (extensions/memory-core/src/memory/manager-sync-ops.ts:821 and extensions/memory-core/src/memory/qmd-manager.ts:2035), so large reindex/export runs now add O(files × store entries) synchronous work and can noticeably stall indexing on workspaces with many sessions. Passing a precomputed lookup (or caching inside buildSessionEntry) is needed to avoid this regression.
Useful? React with 👍 / 👎.
006be87 to
87c09b2
Compare
|
Merged via squash.
Thanks @jalehman! |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 87c09b2a75
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (firstSeparator < 0) { | ||
| return trimmed.startsWith(DREAMING_NARRATIVE_RUN_PREFIX); | ||
| } |
There was a problem hiding this comment.
Restrict dreaming key match to internal run IDs
The new session-store classifier marks any key that starts with dreaming-narrative- as dreaming, including plain explicit session keys (firstSeparator < 0 path). Explicit keys are accepted verbatim by session-key resolution, so a user-chosen key like dreaming-narrative-notes will now be treated as an internal dreaming run and buildSessionEntry will skip all its message text, causing that conversation to disappear from session ingestion/export. Please match only the internal dreaming run-id shape (or other trusted internal metadata) instead of a broad prefix test on arbitrary session keys.
Useful? React with 👍 / 👎.
openclaw#67315) Merged via squash. Prepared head SHA: 87c09b2 Co-authored-by: jalehman <[email protected]> Co-authored-by: jalehman <[email protected]> Reviewed-by: @jalehman
openclaw#67315) Merged via squash. Prepared head SHA: 87c09b2 Co-authored-by: jalehman <[email protected]> Co-authored-by: jalehman <[email protected]> Reviewed-by: @jalehman
openclaw#67315) Merged via squash. Prepared head SHA: 87c09b2 Co-authored-by: jalehman <[email protected]> Co-authored-by: jalehman <[email protected]> Reviewed-by: @jalehman
openclaw#67315) Merged via squash. Prepared head SHA: 87c09b2 Co-authored-by: jalehman <[email protected]> Co-authored-by: jalehman <[email protected]> Reviewed-by: @jalehman
openclaw#67315) Merged via squash. Prepared head SHA: 87c09b2 Co-authored-by: jalehman <[email protected]> Co-authored-by: jalehman <[email protected]> Reviewed-by: @jalehman
openclaw#67315) Merged via squash. Prepared head SHA: 87c09b2 Co-authored-by: jalehman <[email protected]> Co-authored-by: jalehman <[email protected]> Reviewed-by: @jalehman
openclaw#67315) Merged via squash. Prepared head SHA: 87c09b2 Co-authored-by: jalehman <[email protected]> Co-authored-by: jalehman <[email protected]> Reviewed-by: @jalehman
openclaw#67315) Merged via squash. Prepared head SHA: 87c09b2 Co-authored-by: jalehman <[email protected]> Co-authored-by: jalehman <[email protected]> Reviewed-by: @jalehman
openclaw#67315) Merged via squash. Prepared head SHA: 87c09b2 Co-authored-by: jalehman <[email protected]> Co-authored-by: jalehman <[email protected]> Reviewed-by: @jalehman
What
This PR teaches session transcript ingestion to classify
dreaming-narrative-*transcripts from the siblingsessions.jsonstore before reading transcript content. That lets the memory pipeline skip dream diary prompt/prose transcripts even when ingestion sees the JSONL file before the trailing bootstrap record lands.Why
Dreaming narrative sessions were being ingested into
memory/.dreams/session-corpus/*.txtduring that timing window, which polluted Light Sleep reflection candidates and made daily memory files accumulate recursive dreaming junk.Changes
Testing
pnpm test src/memory-host-sdk/host/session-files.test.ts extensions/memory-core/src/dreaming-phases.test.ts9/9and22/22passing