fix(memory,gateway): emit sessionTranscriptUpdate when archiving a live transcript so memory sync picks up new archives incrementally (AI-assisted)#76520
Conversation
|
Thanks for the context here. I did a careful shell check against current Close as implemented on main. Current main contains the equivalent landed patch, including the archive transcript event emit, reset/deleted archive delta-threshold bypass, regression tests, and changelog entry; no release tag contains the proof commit yet, so the provenance is main-only at 2026-05-03T09:15:20Z. So I’m closing this as already implemented rather than keeping a duplicate issue open. Review detailsBest possible solution: Keep the landed main implementation and close this now-redundant PR; any remaining session archive recovery work should continue in narrower follow-ups rather than by merging this duplicate branch. Do we have a high-confidence way to reproduce the issue? Yes. Source inspection gives a high-confidence path: archive rotation is a file rename, memory sync is driven by sessionTranscriptUpdate events, and the landed fix now emits that event and bypasses thresholds for one-shot reset/deleted archive paths. Is this the best way to solve the issue? Yes. The landed implementation uses the existing transcript event contract and existing archive classifiers, which is the narrow maintainable fix; merging this PR now would duplicate the same main-line change. Security review: Security review cleared: The PR diff is limited to in-process transcript signaling, memory classifier exports, tests, and changelog text; it adds no dependencies, permissions, secret handling, downloads, or code execution paths. What I checked:
Likely related people:
Codex review notes: model gpt-5.5, reasoning high; reviewed against 23fe3559e5a2; fix evidence: commit aba97a4c7cff, main fix timestamp 2026-05-03T09:15:20Z. |
2efe07e to
25e78ed
Compare
25e78ed to
45c392f
Compare
45c392f to
25e78ed
Compare
25e78ed to
17cdbe3
Compare
|
Closing: content of this PR already landed in |
|
Thanks @buyitsydney. Landed the equivalent patch on GitHub would not let maintainers update this fork branch ( Proof:
|
… delta threshold for usage-counted archives so new reset/deleted artifacts index incrementally
Two coupled gaps together meant post-reset `.jsonl.reset.<iso>` /
`.jsonl.deleted.<iso>` archives never entered `chunks` on the
incremental memory sync path, only on a manual
`openclaw memory index --force` full reindex.
1. `archiveFileOnDisk` renamed the live `.jsonl` onto disk but never
emitted a `sessionTranscriptUpdate` event, so the memory sync's
`ensureSessionListener` never learned the archived path existed.
Every other in-process session-file mutation already goes through
`emitSessionTranscriptUpdate` (appendMessage, compaction,
tool-result rewrite, chat transcript inject, command execution,
pi-embedded tool-result truncation, `config/sessions/transcript`
append/truncate plumbing, session tool-result guard); archive was
the sole emit gap.
2. Even with the emit in place, `processSessionDeltaBatch` would
forward the event to `updateSessionDelta` and gate it behind the
`deltaBytes` / `deltaMessages` thresholds (defaults
`agents.defaults.memorySearch.sync.sessions.deltaBytes: 100000` and
`deltaMessages: 50`). Those thresholds are designed for live
transcripts accumulating appended messages; a one-shot archive
rename below the threshold would never be marked dirty and never
trigger a session-delta sync, so the archive would still not
reindex.
Fix
- `src/gateway/session-transcript-files.fs.ts::archiveFileOnDisk`
emits `emitSessionTranscriptUpdate({ sessionFile: archived })` after
`fs.renameSync`. Same pattern as the eight existing emit points.
- `extensions/memory-core/src/memory/manager-sync-ops.ts::`
`processSessionDeltaBatch` now short-circuits the delta accounting
for usage-counted archive artifacts (classified by
`isSessionArchiveArtifactName` + `isUsageCountedSessionTranscriptFileName`),
marking the archive path dirty directly and triggering a session-delta
sync regardless of size. `.jsonl.bak.<iso>` is explicitly NOT treated
as usage-counted and therefore does not bypass \u2014 it stays opaque to
the indexer the same way `buildSessionEntry` already skips it.
- `packages/memory-host-sdk/src/engine-qmd.ts` re-exports the two
classification helpers so memory-core can consume them via the
existing plugin-sdk boundary.
Scope
- No change to indexing policy for live transcripts, the chokidar
watch set, force-reindex behavior, visibility filtering, the
downstream `onSessionTranscriptUpdate` contract, or the archive
`.bak`/compaction-checkpoint skip list. The authoritative visibility
guard remains authoritative.
Tests
- `src/gateway/session-transcript-files.fs.archive-events.test.ts`
asserts archive emit for reset / deleted / bak.
- `extensions/memory-core/src/memory/manager-sync-ops.archive-delta-bypass.test.ts`
locks in the classification invariants the bypass depends on
(reset+deleted bypass, bak explicitly does not, live `.jsonl`
explicitly does not, compaction checkpoint does not), plus an
end-to-end verification that `emitSessionTranscriptUpdate` delivers
the archived path verbatim through the transcript-events bus.
Refs openclaw#56131. Follow-up to openclaw#76311 (socket retry, merged) and openclaw#76452
(archive content indexing + visibility stem). Together with those two
PRs this closes the archive-path memory indexing story: archive files
are read (`openclaw#76452`), mapped back to the live transcript stem
(`openclaw#76452`), the archive event is emitted (this PR), and the memory
sync incremental path now actually reindexes the archive instead of
dropping the event at the delta gate (this PR).
AI-assisted: Prepared with Claude (Opus 4.7). Validated locally with
`pnpm check` EXIT 0 and the two new test files (6 + 8 passed across
all three gateway test environments + extension-memory environment).
17cdbe3 to
04a8aa1
Compare
The session transcript listener dropped every archive artifact before scheduling, so .jsonl.reset and .jsonl.deleted archives written by /reset and session delete were no longer indexed on the live path and only surfaced in memory search after the next gateway restart. Remove the archive guard so in-agent archives fall through to scheduleSessionDirty, which reaches the usage-counted-archive branch in processSessionDeltaBatch. Regression from openclaw#89912; restores the incremental archive indexing added in openclaw#76520 that originally fixed openclaw#57334.
Summary
Session archive rotation (
reset/deleted/bak) renames the live.jsonlon disk but never emits asessionTranscriptUpdateevent, so the memory sync incremental path is left in the dark about the new archived path. The only way the index learns about the archive today is a fullmemory index --forcereindex — which is what users actually have to do after a/resetto regain search over post-reset history.*.jsonl.reset.<iso>/*.jsonl.deleted.<iso>files land on disk but never enterchunksuntil a full reindex.memory_searchreturns empty for any query that would land on post-reset history for hours / days afterwards./reset, every auto-reset, every gateway maintenance archive). Losing incremental index coverage on this path meansmemory_searchquietly goes blind to user-visible session history, with no indication. Downstream work that depends on the archive showing up in the index (the archive content reader + visibility stem extractor landed in fix(memory): resolve archived transcript hits during session visibility filtering (follow-up to #76311, AI-assisted) #76452) is wasted until the file is actually pulled in.archiveFileOnDisknow emits a session transcript update afterfs.renameSync, pointing at the new archived path. This is the exact same signal every other in-process session-file mutation already emits (appendMessage, compaction, tool-result rewrite, chat transcript inject, command execution, pi-embedded tool-result truncation, transcript append/truncate plumbing, session tool-result guard — eight existing emit points insrc/today). Archive was the sole mutation point that silently renamed without notifying subscribers.onSessionTranscriptUpdatelisteners (memory sync, sessions-history HTTP, runtime subscriptions) already know how to route a plain{ sessionFile }payload.Change Type
Scope
Linked Issue/PR
Root Cause
archiveFileOnDiskinsrc/gateway/session-transcript-files.fs.tsonly callsfs.renameSync(filePath, archived)and returns. Memory-core's incremental session sync reacts toonSessionTranscriptUpdateevents (seeensureSessionListenerinextensions/memory-core/src/memory/manager-sync-ops.ts), and the chokidar file watcher set up there coversMEMORY.md+memory/+extraPathsonly — not the agent sessions directory. Without an event emit, the new archive path is invisible to both signalling channels, sosessionPendingFilesnever sees it andshouldSyncSessionsForReindexonly returns true ifforce=true(i.e. manualmemory index --force).append,compaction, etc.) grew up alongside the event bus and emits consistently; archive rotation was introduced separately and never got wired through.Regression Test Plan
src/gateway/session-transcript-files.fs.archive-events.test.tsreset,deleted,bak), callingarchiveFileOnDiskrenames the file on disk and emits exactly onesessionTranscriptUpdatewhosesessionFilematches the new archived path — and nothing else (no stalemessage/messageIdpayload).User-visible / Behavior Changes
/reset, auto-reset, or maintenance archive, memory sync sees the new archive path on its next tick and indexes it incrementally — same timing as any other session mutation (appends already reachchunksin seconds). Users no longer needmemory index --forceto recovermemory_searchcoverage of freshly archived transcripts.Diagram
Security Impact
filterMemorySearchHitsBySessionVisibilityremains authoritative, as introduced by fix(memory): resolve archived transcript hits during session visibility filtering (follow-up to #76311, AI-assisted) #76452). Emitting for an archive path cannot leak information because the emit target is the same file that just landed on disk in its agent-owned session store.Repro + Verification
Environment
Steps
sessions.chunks./reset(or any code path hittingarchiveSessionTranscriptsForSession).memory/SQLite:SELECT COUNT(*) FROM files WHERE path LIKE '%.jsonl.reset.%';SELECT COUNT(*) FROM chunks WHERE path LIKE '%.jsonl.reset.%';Before this PR:
files/chunksnever include the just-archived path (untilmemory index --forceruns).After this PR: next memory sync tick (within seconds) picks up the archive,
filesandchunksboth reflect it,memory_searchcan surface post-reset history normally.Evidence
Local validation on upstream
mainat this commit SHA (Linux, Node 24.14.0, pnpm 10.33.2):pnpm check— EXIT 0 (typecheck prod / oxlint / policy guards / import-cycle / temp-path guardrails all green)npx vitest run src/gateway/session-transcript-files.fs.archive-events.test.ts→ 6 passed (2 cases × 3 gateway test environments: core / server / client)npx vitest run src/gateway/session-archive.imports.test.ts(existing) → 6 passed (no regression on the imports contract)Human Verification
resetarchive emits exactly one update pointing at the new path;deletedandbakbehave identically.message/messageId).session-archive.imports.test.tsstill holds (archive runtime not eagerly loaded byreply session module).fs.renameSyncthrows before the emit, so no ghost event for a nonexistent path..bakreason participates in the emit identically (useful for thesessions.tsbak rotation flow atsrc/gateway/server-methods/sessions.ts:1952).src/sessions/transcript-events.ts), so the emit never escapes the process.pnpm testsweep on upstreammainat this exact SHA — only the targeted test files were exercised locally; CI will cover the rest on the PR.Review Conversations
Compatibility / Migration
sessionFilepayload.Risks and Mitigations
.jsonl.reset.<iso>/.jsonl.deleted.<iso>/.jsonl.bak.<iso>), and existing listeners treatsessionFileas an opaque string that downstream code classifies via the usage-counted helpers insrc/config/sessions/artifacts.ts. The memory sync listener is the primary consumer and is the one that should receive this signal.renameSyncfailure still fires the emit and points at a non-existent path.fs.renameSyncthrows on failure, so execution never reaches the emit in that case. The test plan locks this in by asserting the file actually exists at the archived path when the emit fires.AI-assisted PR: Prepared with Claude (Opus 4.7). Lightly tested — the underlying behavior is in active validation on a downstream fleet; upstream version is unit-test-validated locally in all three gateway test environments, and
pnpm checkis EXIT 0 on this SHA.