Skip to content

fix(memory,gateway): emit sessionTranscriptUpdate when archiving a live transcript so memory sync picks up new archives incrementally (AI-assisted)#76520

Closed
buyitsydney wants to merge 1 commit into
openclaw:mainfrom
buyitsydney:fix/archive-emit-transcript-update
Closed

fix(memory,gateway): emit sessionTranscriptUpdate when archiving a live transcript so memory sync picks up new archives incrementally (AI-assisted)#76520
buyitsydney wants to merge 1 commit into
openclaw:mainfrom
buyitsydney:fix/archive-emit-transcript-update

Conversation

@buyitsydney

Copy link
Copy Markdown
Contributor

Summary

Session archive rotation (reset / deleted / bak) renames the live .jsonl on disk but never emits a sessionTranscriptUpdate event, so the memory sync incremental path is left in the dark about the new archived path. The only way the index learns about the archive today is a full memory index --force reindex — which is what users actually have to do after a /reset to regain search over post-reset history.

  • Problem: After a session reset, *.jsonl.reset.<iso> / *.jsonl.deleted.<iso> files land on disk but never enter chunks until a full reindex. memory_search returns empty for any query that would land on post-reset history for hours / days afterwards.
  • Why it matters: Reset is a hot path (every /reset, every auto-reset, every gateway maintenance archive). Losing incremental index coverage on this path means memory_search quietly goes blind to user-visible session history, with no indication. Downstream work that depends on the archive showing up in the index (the archive content reader + visibility stem extractor landed in fix(memory): resolve archived transcript hits during session visibility filtering (follow-up to #76311, AI-assisted) #76452) is wasted until the file is actually pulled in.
  • What changed: archiveFileOnDisk now emits a session transcript update after fs.renameSync, pointing at the new archived path. This is the exact same signal every other in-process session-file mutation already emits (appendMessage, compaction, tool-result rewrite, chat transcript inject, command execution, pi-embedded tool-result truncation, transcript append/truncate plumbing, session tool-result guard — eight existing emit points in src/ today). Archive was the sole mutation point that silently renamed without notifying subscribers.
  • What did NOT change (scope boundary): Emission only. No change to indexing policy, the chokidar watch set, force-reindex behavior, visibility filtering, or any downstream subscriber. onSessionTranscriptUpdate listeners (memory sync, sessions-history HTTP, runtime subscriptions) already know how to route a plain { sessionFile } payload.

Change Type

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

Root Cause

  • Root cause: archiveFileOnDisk in src/gateway/session-transcript-files.fs.ts only calls fs.renameSync(filePath, archived) and returns. Memory-core's incremental session sync reacts to onSessionTranscriptUpdate events (see ensureSessionListener in extensions/memory-core/src/memory/manager-sync-ops.ts), and the chokidar file watcher set up there covers MEMORY.md + memory/ + extraPaths only — not the agent sessions directory. Without an event emit, the new archive path is invisible to both signalling channels, so sessionPendingFiles never sees it and shouldSyncSessionsForReindex only returns true if force=true (i.e. manual memory index --force).
  • Missing detection / guardrail: There was no unit test asserting that archive rotation emits, so the asymmetry with the other eight emit sites was invisible at the contract layer.
  • Contributing context: Archive is the newest of the session-file mutation points. The rest of the surface (append, compaction, etc.) grew up alongside the event bus and emits consistently; archive rotation was introduced separately and never got wired through.

Regression Test Plan

  • Coverage level that should have caught this:
    • Unit test
    • Seam / integration test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file: src/gateway/session-transcript-files.fs.archive-events.test.ts
  • Scenario the test should lock in: For each archive reason (reset, deleted, bak), calling archiveFileOnDisk renames the file on disk and emits exactly one sessionTranscriptUpdate whose sessionFile matches the new archived path — and nothing else (no stale message / messageId payload).
  • Why this is the smallest reliable guardrail: The contract is pure side-effect: a file rename + an event emit. A unit-level assertion is sufficient and cheap.
  • Existing test that already covers this: None.
  • If no new test is added, why not: N/A — 2 new cases added, exercising all three archive reasons.

User-visible / Behavior Changes

  • After a session /reset, auto-reset, or maintenance archive, memory sync sees the new archive path on its next tick and indexes it incrementally — same timing as any other session mutation (appends already reach chunks in seconds). Users no longer need memory index --force to recover memory_search coverage of freshly archived transcripts.
  • No config / env / defaults changed.

Diagram

Before (session reset):

  archiveFileOnDisk(filePath, "reset")
    └─ fs.renameSync(filePath, archived)
    └─ return archived                         ← NO event

  memory sync:
    ensureSessionListener → onSessionTranscriptUpdate  ← never fires for archived
    scheduleSessionDirty / sessionPendingFiles         ← never adds archived
    shouldSyncSessionsForReindex                       ← force=true only

  Result: archived file sits on disk, invisible to the incremental index
          until `memory index --force` runs.

After:

  archiveFileOnDisk(filePath, "reset")
    └─ fs.renameSync(filePath, archived)
    └─ emitSessionTranscriptUpdate({ sessionFile: archived })   ← NEW
    └─ return archived

  memory sync:
    onSessionTranscriptUpdate → scheduleSessionDirty(archived)
    → sessionPendingFiles.add(archived)
    → processSessionDeltaBatch → sessionsDirtyFiles.add(archived)
    → shouldSyncSessionsForReindex → true → targeted reindex

  Result: archive is indexed on the next memory sync tick,
          same signalling path as append / compaction / rewrite.

Security Impact

  • New permissions/capabilities? No
  • Secrets/tokens handling changed? No
  • New/changed network calls? No
  • Command/tool execution surface changed? No
  • Data access scope changed? No. The event carries only the new path. Listeners already gate their downstream actions against their own authorization model (the memory search visibility guard in filterMemorySearchHitsBySessionVisibility remains authoritative, as introduced by fix(memory): resolve archived transcript hits during session visibility filtering (follow-up to #76311, AI-assisted) #76452). Emitting for an archive path cannot leak information because the emit target is the same file that just landed on disk in its agent-owned session store.

Repro + Verification

Environment

  • OS: Linux (container)
  • Runtime: Node 24.14.0, pnpm 10.33.2
  • Model/provider: any
  • Config: default

Steps

  1. Start a fresh session with memory source including sessions.
  2. Append a few messages so the session file has real content indexed into chunks.
  3. Trigger /reset (or any code path hitting archiveSessionTranscriptsForSession).
  4. Inspect memory/ SQLite:
    • SELECT COUNT(*) FROM files WHERE path LIKE '%.jsonl.reset.%';
    • SELECT COUNT(*) FROM chunks WHERE path LIKE '%.jsonl.reset.%';

Before this PR: files/chunks never include the just-archived path (until memory index --force runs).

After this PR: next memory sync tick (within seconds) picks up the archive, files and chunks both reflect it, memory_search can surface post-reset history normally.

Evidence

Local validation on upstream main at this commit SHA (Linux, Node 24.14.0, pnpm 10.33.2):

  • pnpm checkEXIT 0 (typecheck prod / oxlint / policy guards / import-cycle / temp-path guardrails all green)
  • npx vitest run src/gateway/session-transcript-files.fs.archive-events.test.ts6 passed (2 cases × 3 gateway test environments: core / server / client)
  • npx vitest run src/gateway/session-archive.imports.test.ts (existing) → 6 passed (no regression on the imports contract)
  • Screenshot / recording — not applicable (no UI change)

Human Verification

  • Verified scenarios:
    • reset archive emits exactly one update pointing at the new path; deleted and bak behave identically.
    • Payload contract matches the compaction-only emit pattern (sessionFile only, no stale message / messageId).
    • Imports graph under session-archive.imports.test.ts still holds (archive runtime not eagerly loaded by reply session module).
  • Edge cases checked:
    • Rename failure from fs.renameSync throws before the emit, so no ghost event for a nonexistent path.
    • .bak reason participates in the emit identically (useful for the sessions.ts bak rotation flow at src/gateway/server-methods/sessions.ts:1952).
    • Listener set is process-local (src/sessions/transcript-events.ts), so the emit never escapes the process.
  • What I did NOT verify: Full pnpm test sweep on upstream main at this exact SHA — only the targeted test files were exercised locally; CI will cover the rest on the PR.

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

  • Backward compatible? Yes — strictly additive. Any listener that did not care about archive events simply receives an extra notification for a file path; listeners already handle any sessionFile payload.
  • Config/env changes? No
  • Migration needed? No

Risks and Mitigations

  • Risk: A subscriber that explicitly wants to ignore archived paths (e.g., treats archive as opaque) now receives the emit.
    • Mitigation: Archived paths are already distinguishable by suffix (.jsonl.reset.<iso> / .jsonl.deleted.<iso> / .jsonl.bak.<iso>), and existing listeners treat sessionFile as an opaque string that downstream code classifies via the usage-counted helpers in src/config/sessions/artifacts.ts. The memory sync listener is the primary consumer and is the one that should receive this signal.
  • Risk: A renameSync failure still fires the emit and points at a non-existent path.
    • Mitigation: fs.renameSync throws on failure, so execution never reaches the emit in that case. The test plan locks this in by asserting the file actually exists at the archived path when the emit fires.

AI-assisted PR: Prepared with Claude (Opus 4.7). Lightly tested — the underlying behavior is in active validation on a downstream fleet; upstream version is unit-test-validated locally in all three gateway test environments, and pnpm check is EXIT 0 on this SHA.

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime size: S labels May 3, 2026
@clawsweeper

clawsweeper Bot commented May 3, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I did a careful shell check against current main, and this is already implemented.

Close as implemented on main. Current main contains the equivalent landed patch, including the archive transcript event emit, reset/deleted archive delta-threshold bypass, regression tests, and changelog entry; no release tag contains the proof commit yet, so the provenance is main-only at 2026-05-03T09:15:20Z.

So I’m closing this as already implemented rather than keeping a duplicate issue open.

Review details

Best possible solution:

Keep the landed main implementation and close this now-redundant PR; any remaining session archive recovery work should continue in narrower follow-ups rather than by merging this duplicate branch.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection gives a high-confidence path: archive rotation is a file rename, memory sync is driven by sessionTranscriptUpdate events, and the landed fix now emits that event and bypasses thresholds for one-shot reset/deleted archive paths.

Is this the best way to solve the issue?

Yes. The landed implementation uses the existing transcript event contract and existing archive classifiers, which is the narrow maintainable fix; merging this PR now would duplicate the same main-line change.

Security review:

Security review cleared: The PR diff is limited to in-process transcript signaling, memory classifier exports, tests, and changelog text; it adds no dependencies, permissions, secret handling, downloads, or code execution paths.

What I checked:

Likely related people:

  • buyitsydney: Authored the proof commit now on main and the recent archive transcript visibility/content-safety work that this PR builds on. (role: recent archive-memory implementation author; confidence: high; commits: aba97a4c7cff, 2ffdb5d248a1; files: src/gateway/session-transcript-files.fs.ts, extensions/memory-core/src/memory/manager-sync-ops.ts, packages/memory-host-sdk/src/host/session-files.ts)
  • vincentkoc: Committed the landed main patch and appears in recent current-main history for the memory host SDK contract surface used by this fix. (role: maintainer/merger and adjacent SDK contract owner; confidence: medium; commits: aba97a4c7cff, 2e593d07f706; files: packages/memory-host-sdk/src/engine-qmd.ts, src/gateway/session-transcript-files.fs.ts, extensions/memory-core/src/memory/manager-sync-ops.ts)
  • solodmd: Recently touched the memory sync watcher path in manager-sync-ops on current main, so they are an adjacent routing candidate if follow-up watcher behavior regresses. (role: recent adjacent maintainer; confidence: low; commits: d1365fef16d8; files: extensions/memory-core/src/memory/manager-sync-ops.ts)

Codex review notes: model gpt-5.5, reasoning high; reviewed against 23fe3559e5a2; fix evidence: commit aba97a4c7cff, main fix timestamp 2026-05-03T09:15:20Z.

@buyitsydney
buyitsydney force-pushed the fix/archive-emit-transcript-update branch from 2efe07e to 25e78ed Compare May 3, 2026 07:32
@openclaw-barnacle openclaw-barnacle Bot added extensions: memory-core Extension: memory-core size: M and removed size: S labels May 3, 2026
@buyitsydney
buyitsydney force-pushed the fix/archive-emit-transcript-update branch from 25e78ed to 45c392f Compare May 3, 2026 08:26
@openclaw-barnacle openclaw-barnacle Bot added size: S and removed extensions: memory-core Extension: memory-core size: M labels May 3, 2026
@buyitsydney
buyitsydney force-pushed the fix/archive-emit-transcript-update branch from 45c392f to 25e78ed Compare May 3, 2026 08:28
@openclaw-barnacle openclaw-barnacle Bot added extensions: memory-core Extension: memory-core size: M and removed size: S labels May 3, 2026
@vincentkoc
vincentkoc force-pushed the fix/archive-emit-transcript-update branch from 25e78ed to 17cdbe3 Compare May 3, 2026 08:47
@vincentkoc vincentkoc self-assigned this May 3, 2026
@buyitsydney

Copy link
Copy Markdown
Contributor Author

Closing: content of this PR already landed in upstream/main as commit 25e78ed56d (same title/author/diff). CI was red because PR base predated that commit, causing duplicate CHANGELOG conflict + checks-fast-contracts-plugins staleness. No further action needed on this PR.

@buyitsydney buyitsydney closed this May 3, 2026
@vincentkoc

Copy link
Copy Markdown
Member

Thanks @buyitsydney. Landed the equivalent patch on main as aba97a4.

GitHub would not let maintainers update this fork branch (maintainer_can_modify=false), and the remaining red check was stale-main noise from the plugin-sdk channel-facade classification that had already landed on main. I rebased locally, resolved the stale-main issue, and verified the rebased patch before landing it directly.

Proof:

  • pnpm test:serial extensions/memory-core/src/memory/manager-sync-ops.archive-delta-bypass.test.ts packages/memory-host-sdk/src/host/session-files.test.ts src/gateway/session-transcript-files.fs.archive-events.test.ts extensions/memory-core/src/memory/manager.session-reindex.test.ts extensions/memory-core/src/memory/manager-targeted-sync.test.ts src/plugins/contracts/plugin-sdk-package-contract-guardrails.test.ts
  • OPENCLAW_TESTBOX=1 pnpm check:changed on tbx_01kqphn0fsma91nkbcw9m4x7gx

… delta threshold for usage-counted archives so new reset/deleted artifacts index incrementally

Two coupled gaps together meant post-reset `.jsonl.reset.<iso>` /
`.jsonl.deleted.<iso>` archives never entered `chunks` on the
incremental memory sync path, only on a manual
`openclaw memory index --force` full reindex.

1. `archiveFileOnDisk` renamed the live `.jsonl` onto disk but never
   emitted a `sessionTranscriptUpdate` event, so the memory sync's
   `ensureSessionListener` never learned the archived path existed.
   Every other in-process session-file mutation already goes through
   `emitSessionTranscriptUpdate` (appendMessage, compaction,
   tool-result rewrite, chat transcript inject, command execution,
   pi-embedded tool-result truncation, `config/sessions/transcript`
   append/truncate plumbing, session tool-result guard); archive was
   the sole emit gap.

2. Even with the emit in place, `processSessionDeltaBatch` would
   forward the event to `updateSessionDelta` and gate it behind the
   `deltaBytes` / `deltaMessages` thresholds (defaults
   `agents.defaults.memorySearch.sync.sessions.deltaBytes: 100000` and
   `deltaMessages: 50`). Those thresholds are designed for live
   transcripts accumulating appended messages; a one-shot archive
   rename below the threshold would never be marked dirty and never
   trigger a session-delta sync, so the archive would still not
   reindex.

Fix

- `src/gateway/session-transcript-files.fs.ts::archiveFileOnDisk`
  emits `emitSessionTranscriptUpdate({ sessionFile: archived })` after
  `fs.renameSync`. Same pattern as the eight existing emit points.

- `extensions/memory-core/src/memory/manager-sync-ops.ts::`
  `processSessionDeltaBatch` now short-circuits the delta accounting
  for usage-counted archive artifacts (classified by
  `isSessionArchiveArtifactName` + `isUsageCountedSessionTranscriptFileName`),
  marking the archive path dirty directly and triggering a session-delta
  sync regardless of size. `.jsonl.bak.<iso>` is explicitly NOT treated
  as usage-counted and therefore does not bypass \u2014 it stays opaque to
  the indexer the same way `buildSessionEntry` already skips it.

- `packages/memory-host-sdk/src/engine-qmd.ts` re-exports the two
  classification helpers so memory-core can consume them via the
  existing plugin-sdk boundary.

Scope

- No change to indexing policy for live transcripts, the chokidar
  watch set, force-reindex behavior, visibility filtering, the
  downstream `onSessionTranscriptUpdate` contract, or the archive
  `.bak`/compaction-checkpoint skip list. The authoritative visibility
  guard remains authoritative.

Tests

- `src/gateway/session-transcript-files.fs.archive-events.test.ts`
  asserts archive emit for reset / deleted / bak.
- `extensions/memory-core/src/memory/manager-sync-ops.archive-delta-bypass.test.ts`
  locks in the classification invariants the bypass depends on
  (reset+deleted bypass, bak explicitly does not, live `.jsonl`
  explicitly does not, compaction checkpoint does not), plus an
  end-to-end verification that `emitSessionTranscriptUpdate` delivers
  the archived path verbatim through the transcript-events bus.

Refs openclaw#56131. Follow-up to openclaw#76311 (socket retry, merged) and openclaw#76452
(archive content indexing + visibility stem). Together with those two
PRs this closes the archive-path memory indexing story: archive files
are read (`openclaw#76452`), mapped back to the live transcript stem
(`openclaw#76452`), the archive event is emitted (this PR), and the memory
sync incremental path now actually reindexes the archive instead of
dropping the event at the delta gate (this PR).

AI-assisted: Prepared with Claude (Opus 4.7). Validated locally with
`pnpm check` EXIT 0 and the two new test files (6 + 8 passed across
all three gateway test environments + extension-memory environment).
@buyitsydney buyitsydney reopened this May 3, 2026
@buyitsydney
buyitsydney force-pushed the fix/archive-emit-transcript-update branch from 17cdbe3 to 04a8aa1 Compare May 3, 2026 09:18
@clawsweeper clawsweeper Bot closed this May 3, 2026
yetval added a commit to yetval/openclaw that referenced this pull request Jun 23, 2026
The session transcript listener dropped every archive artifact before
scheduling, so .jsonl.reset and .jsonl.deleted archives written by /reset
and session delete were no longer indexed on the live path and only
surfaced in memory search after the next gateway restart. Remove the
archive guard so in-agent archives fall through to scheduleSessionDirty,
which reaches the usage-counted-archive branch in processSessionDeltaBatch.

Regression from openclaw#89912; restores the incremental archive indexing added in
openclaw#76520 that originally fixed openclaw#57334.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: memory-core Extension: memory-core gateway Gateway runtime size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants