Skip to content

Session transcript integrity: .jsonl silently drops early user messages; .trajectory.jsonl FIFO window makes neither file independently authoritative #104323

Description

@iceteaSA

Two distinct loss shapes in paired session files (<id>.jsonl + <id>.trajectory.jsonl), observed on 2026.7.1-beta.2. Each file loses history the other retains, so any consumer that treats one file as "the transcript" (backup, archive, history reassembly, memory extraction, incident forensics) works from an incomplete record without knowing it.

Shape 1 — .jsonl head-drop

A session's .jsonl was missing its first 7 user-message entries while the paired .trajectory.jsonl recorded all 14 turns (session.started events) correctly. The .jsonl retained session-startup events (model_change, thinking_level_change) but the early user messages themselves never landed. Nothing in either file marks the gap.

Detection signature: jsonl.first_user_message_ts > trajectory.first_session_started_ts. In our case the gap was +3313s (~55 min of conversation missing from the runtime transcript).

From reading the dist source: appendJsonlEntrySync (transcript-write-context) appends per-message with no retry; a failure between the trajectory recorder's event write and the jsonl flush leaves the trajectory with session.started + prompt.submitted while the jsonl permanently lacks the user message. Side note: the message content survived in the context-engine DB (lossless-claw's messages table), which is how we confirmed the content was real — but the runtime transcript is what downstream consumers read.

Shape 2 — trajectory FIFO window (expected, but undocumented as a pairing hazard)

trimJsonlWindow drops oldest trajectory events once the file approaches TRAJECTORY_RUNTIME_CAPTURE_MAX_BYTES (10 MiB). A 3-week-old session had its trajectory at 99.4% of cap holding only the last ~25 turns, while its .jsonl held the full history. This is by design — but combined with shape 1 it means neither file is a superset of the other, and there is no marker in either file saying "the other file has what I lost."

Why it matters

Repro

Synthetic fixtures for both shapes (head-drop and rotation-with-covered-head) are exercised by a standalone checker we wrote while diagnosing this: it classifies a session pair as no-loss / rotation-truncated / anomalous / indeterminate from timestamps, counts, and coverage spans. Happy to share the script or fold it into a PR if useful.

Suggested fixes (any one of these)

  1. Atomic paired flush on the prompt-submitted path (trajectory event + jsonl user-message entry under one mutex; replay from the trajectory seq cursor on crash).
  2. Embed the jsonl message.id into prompt.submitted.data.messageId so downstream consumers can do record-level alignment and detect gaps (today the id schemes are disjoint: per-message uuid vs per-turn seq/sourceSeq).
  3. On compaction/rotation, snapshot BOTH files as a pair (rotateTranscriptAfterCompaction currently handles only the jsonl).

Version: [email protected] · single-host LXC install · no client data in this report; timestamps and counts only.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:data-lossCan lose, corrupt, or silently drop user/session/config data.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions