Skip to content

ambient transcript watermark survives session reset, permanently hiding archived messages #99373

Description

@obviyus

What happened

3ad465d (#99306) introduced per-chat ambient transcript watermarks (SessionEntry.ambientTranscriptWatermarks): room-event rows persist into the session transcript and every prompt-facing raw-window consumer filters out messages at/before the watermark, so the per-turn chat window only carries the un-persisted gap.

The watermark survives session resets, but the transcript rows it vouches for do not. Observed in production: a group session was reset (transcript archived to <id>.jsonl.reset.<ts>, new sessionId minted); the ambient watermark carried over on the session entry; the next turn's chat window filtered out every message at/before the watermark — but those rows now live only in the archived transcript the model never sees. The new session's transcript demonstrably begins exactly at watermark+1; everything older is permanently invisible to the agent (not in the window, not in the transcript, never compactable).

Root cause

Watermark validity is a fact about a specific transcript (sessionId), but it is stored and read as a fact about the chat. Any flow that mints a new sessionId (manual reset, daily reset, session delete) silently converts "persisted, deduped" into "dropped".

Expected

Bind watermark validity to the transcript it describes: stamp the current sessionId into the watermark record on write, and treat a watermark whose sessionId differs from the entry's current sessionId as absent on read. Self-invalidating on every reset path — no per-reset-flow cleanup to forget. Existing shipped watermarks without a sessionId are cache-like state: treat as invalid and let the window backfill once (no doctor migration needed).

Also observed (needs diagnosis)

The same production group session reset twice in one night with default reset config (mode daily, DEFAULT_RESET_AT_HOUR=4): one transcript archived as .deleted. at 03:00:18 UTC and one as .reset. at 04:23:04 UTC (session was ~37 min old). Two intra-day resets are not explainable by the documented daily policy; worth root-causing while in this code (separate fix if it is a distinct bug).

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.bugSomething isn't workingclawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.maintainerMaintainer-authored PR

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions