fix(sessions): stream JSONL transcript scans instead of buffering whole files by davidv399 · Pull Request #80615 · openclaw/openclaw

davidv399 · 2026-05-11T09:54:46Z

The remaining whole-file transcript scans called out by ClawSweeper triage on #54296 still materialized the entire session JSONL file in memory before splitting on newlines. On long-running sessions where transcripts grow into the multi-MB / 100s-of-MB range, that scales peak RSS with file size and is the practical OOM risk in the report.

This PR adds shared streaming helpers in src/config/sessions/transcript-stream.ts and migrates every flagged whole-file scan to them. Forward scans use fs.createReadStream + readline (one line of memory at a time). Reverse scans now walk the transcript newest-first in bounded chunks from the file end, decoding complete JSONL lines without a fixed tail cap, so /tts latest and delivery-mirror dedupe keep exact latest-assistant semantics even when a large user/tool entry follows the assistant line.

Migrated scans:

src/config/sessions/transcript.ts -- latest/tail assistant reads, delivery-mirror equivalent-message lookup, and idempotency-key lookup.
src/gateway/server-methods/chat.ts -- chat-method append idempotency lookup, now also tolerant of malformed lines mid-scan.
src/gateway/session-compaction-checkpoints.ts -- compaction fork entry loading from the forward stream helper instead of one big fileHandle.readFile("utf-8").

Preserved invariants from ClawSweeper's remaining-risk note: parentId DAG, malformed-line tolerance, idempotency-key return semantics, newest-assistant lookup semantics, and delivery-mirror dedupe behavior.

Verification

pnpm test src/config/sessions/transcript-stream.test.ts src/config/sessions/transcript.test.ts -- 29/29 pass.
node scripts/run-tsgo.mjs -p tsconfig.core.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/core.tsbuildinfo -- pass.
pnpm check:changed on Testbox tbx_01krbb7zw876zaqv9hfvg90k01 -- core and core-test typechecks pass; still fails in pre-existing unrelated extensions/qqbot/src/engine/tools/remind-logic.test.ts(238,23) on deleteAfterRun.
git diff --check -- clean.

Real behavior proof

Behavior addressed: Peak resident memory growing with transcript size during idempotency lookups, latest-assistant scans, delivery-mirror dedupe, and compaction fork loading on long-running sessions. The fix must keep latest-assistant and dedupe behavior exact even when a large user/tool entry follows the assistant entry.

Real environment tested: Blacksmith Testbox via Crabbox on PR head 552470c29a4d29afe1015222420638b657444802, lease tbx_01krbcw4swx29ecnqnpewj5j50, slug amber-barnacle, Node 24 Testbox workflow environment. The command created a real OpenClaw session store and JSONL transcript on disk, used production session append/read helpers, appended an 8 MiB trailing user entry, verified latest-assistant reverse scan and delivery-mirror dedupe, verified idempotency forward scan, and forked a compaction checkpoint transcript.

Exact steps or command run after this patch: pnpm crabbox:run -- --provider blacksmith-testbox --blacksmith-org openclaw --blacksmith-workflow .github/workflows/ci-check-testbox.yml --blacksmith-job check --blacksmith-ref main --idle-timeout 90m --ttl 240m --timing-json --shell -- <script that runs pnpm exec tsx .crabbox/pr80615-transcript-e2e.mts>. The script imports src/config/sessions/transcript.ts, src/config/sessions/transcript-append.ts, and src/gateway/session-compaction-checkpoints.ts from the synced PR checkout and exercises real filesystem state under /tmp.

Evidence after fix: Copied Crabbox terminal output from the passing Testbox run:

CRABBOX_PHASE:e2e-transcript
PR80615_E2E_OK
sessionFileBytes=8389986
sourceLines=4
forkLines=4
latestAssistantId=8819b590-0325-4e3c-a4df-b04ddf69742a
mirrorDedupeId=8819b590-0325-4e3c-a4df-b04ddf69742a
idempotentMessageId=2128a785-436a-4703-94e3-746f8fdbba69
tempDir=/tmp/openclaw-pr80615-e2e-by3EWT
blacksmith run summary sync=delegated command=37.41s total=40.098s exit=0
{"provider":"blacksmith-testbox","leaseId":"tbx_01krbcw4swx29ecnqnpewj5j50","slug":"amber-barnacle","syncMs":0,"syncPhases":[{"name":"delegated","skipped":true,"reason":"blacksmith-testbox owns sync"}],"syncSkipped":false,"syncDelegated":true,"commandMs":37409,"commandPhases":[{"name":"user-command","ms":32806},{"name":"e2e-transcript","ms":4603}],"totalMs":40098,"exitCode":0}
Testbox tbx_01krbcw4swx29ecnqnpewj5j50 stopped (completed)

Observed result after fix: The real transcript file was 8,389,986 bytes with the newest entry being a large user message. readLatestAssistantTextFromSessionTranscript still found the earlier assistant id. appendAssistantMessageToSessionTranscript returned the same assistant id for delivery-mirror dedupe instead of appending a duplicate. Repeating an append with the same idempotency key returned the same idempotent message id. forkCompactionCheckpointTranscriptAsync streamed and forked the transcript with matching line count.

What was not tested: No live model/provider turn or channel delivery was exercised; this bug is in local session transcript scanning and compaction file handling, so the E2E uses real OpenClaw session files and production session helpers without provider credentials.

clawsweeper · 2026-05-11T09:59:27Z

Codex review: needs real behavior proof before merge.

Summary
The branch adds shared session transcript streaming helpers, migrates session and gateway whole-file JSONL scans to them, adds regression coverage, and updates the changelog.

Reproducibility: yes. at source level. Current main still reaches fs.promises.readFile or fileHandle.readFile on the affected transcript scan paths before splitting and scanning JSONL content.

Real behavior proof
Needs stronger real behavior proof before merge: The body includes terminal proof for forward streaming, but current head's reverse-scan behavior is covered only by tests; add redacted terminal/log/screenshot proof from current head and update the PR body to trigger a fresh ClawSweeper review, or ask a maintainer for @clawsweeper re-review.

Next step before merge
Not a repair-lane candidate because no narrow code defect was found; the remaining blocker is updated real behavior proof for the latest PR head.

Security
Cleared: The diff adds no dependencies, workflow changes, install hooks, secret handling, or external execution paths, and it narrows memory exposure while reading local transcript files.

Review details

Best possible solution:

Land the shared streaming helper approach after updated real behavior proof for the current head and normal maintainer/CI handling.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level. Current main still reaches fs.promises.readFile or fileHandle.readFile on the affected transcript scan paths before splitting and scanning JSONL content.

Is this the best way to solve the issue?

Yes for the code direction. Forward streaming plus a chunked newest-first reverse scanner is the narrow maintainable fix, but merge should wait for current-head real behavior proof of the reverse scan behavior.

What I checked:

Current main session transcript scans: Current main still reads the whole transcript with fs.promises.readFile before reverse latest/tail assistant scans and forward idempotency/delivery-mirror scans. (src/config/sessions/transcript.ts:148, c53d7aee0f90)
Current main gateway scan: The chat method idempotency helper reads the entire transcript with fs.promises.readFile and then splits it before scanning for the key. (src/gateway/server-methods/chat.ts:1359, c53d7aee0f90)
Current main compaction fork scan: The compaction checkpoint fork loader calls fileHandle.readFile("utf-8") and materializes all parsed entries before returning them. (src/gateway/session-compaction-checkpoints.ts:156, c53d7aee0f90)
PR helper and caller migration: The PR head adds streamSessionTranscriptLines and streamSessionTranscriptLinesReverse, then routes the session transcript, chat idempotency, and compaction fork callers through the helpers. (src/config/sessions/transcript-stream.ts:35, 552470c29a4d)
Regression coverage: The PR adds focused helper tests for forward streaming, missing and empty files, malformed lines, abort handling, long lines, reverse chunk boundaries, UTF-8 boundaries, and newest-first ordering, plus a delivery-mirror regression with a 5 MiB following user entry. (src/config/sessions/transcript-stream.test.ts:34, 552470c29a4d)
Current-head proof gap: The PR body contains terminal proof for a 200 MiB forward streaming scan, but the latest head commit changes reverse scan behavior and the body only cites tests for those latest-assistant/dedupe reverse paths. (552470c29a4d)

Likely related people:

steipete: Current-main blame in the shallow checkout points to Peter Steinberger on the affected transcript/gateway scan lines, and the latest PR head commit rewrites the reverse scan helper behavior. (role: recent area contributor; confidence: high; commits: 3ed9b23834, 552470c29a4d, 15cf49222f92; files: src/config/sessions/transcript.ts, src/gateway/server-methods/chat.ts, src/gateway/session-compaction-checkpoints.ts)
vincentkoc: Prior review history ties this person to recent gateway/session transcript hot-path work adjacent to the files being migrated. (role: recent adjacent transcript contributor; confidence: medium; commits: ec59af33863a, aec83af23d49, d122a3492d17; files: src/gateway/session-utils.fs.ts, src/gateway/server-methods/chat.ts)
andyylin: Prior review history identifies delivery-mirror dedupe work in src/config/sessions/transcript.ts, one of the reverse scans changed here. (role: delivery-mirror dedupe contributor; confidence: medium; commits: e95efa437381, d842ec417924; files: src/config/sessions/transcript.ts, src/config/sessions/transcript.test.ts)
amknight: Prior review history connects this person to async transcript I/O work around session compaction checkpoints, the fork loader surface changed by this PR. (role: compaction I/O contributor; confidence: medium; commits: 6f819280a302; files: src/gateway/session-compaction-checkpoints.ts)
scoootscooob: Prior review history identifies this person as introducing the compaction checkpoint area containing the fork loader changed here. (role: feature introducer; confidence: medium; commits: f4fcaa09a358; files: src/gateway/session-compaction-checkpoints.ts)

Remaining risk / open question:

The posted terminal proof does not demonstrate the current-head reverse-scan paths changed by the latest commit; tests are useful but supplemental for the external-PR proof gate.
No local tests were run because this was a read-only review of the checked-out repository.

Codex review notes: model gpt-5.5, reasoning high; reviewed against c53d7aee0f90.

davidv399 · 2026-05-11T10:01:04Z

@clawsweeper re-review

clawsweeper · 2026-05-11T10:02:31Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/25663513478
Updated: 2026-05-11T10:11:22.755Z

…le files (openclaw#54296) The remaining whole-file transcript scans flagged by ClawSweeper triage on splitting on newlines. On long-running sessions where transcripts grow into the multi-MB / 100s-of-MB range that scales peak RSS with file size and is the practical OOM risk in the report. Add a shared streaming helper module `src/config/sessions/transcript-stream.ts` exposing: - `streamSessionTranscriptLines(filePath, { signal? })`: forward async-iterator over trimmed non-empty lines using `fs.createReadStream` + `readline` with `crlfDelay: Infinity`. Bounded to one line of memory at a time and honours an abort signal between lines. - `readSessionTranscriptTailLines(filePath, { maxBytes? })`: tail-only read of the last `maxBytes` of a file (default 4 MiB, clamped to [1 KiB, 64 MiB]), returning trimmed non-empty lines in reverse order. Drops the leading line of the slice when the window does not start at byte zero so callers never see a partial-line suffix. Migrate every flagged whole-file scan to these helpers while preserving the malformed-line tolerance and idempotency-key return semantics callers depend on (see `Remaining risk / open question` on the issue): - `src/config/sessions/transcript.ts`: `readLatestAssistantTextFromSessionTranscript`, `readTailAssistantTextFromSessionTranscript`, and the delivery-mirror dedupe helper `findLatestEquivalentAssistantMessageId` now use the tail helper; `transcriptHasIdempotencyKey` uses the forward stream helper. - `src/gateway/server-methods/chat.ts`: the inline `transcriptHasIdempotencyKey` used by chat-method append idempotency now uses the forward stream helper and tolerates malformed lines mid-scan (matching the sibling helper in `config/sessions/transcript.ts`). - `src/gateway/session-compaction-checkpoints.ts`: `readTranscriptEntriesForForkAsync` builds the fork entry array from the forward stream helper instead of one big `fileHandle.readFile("utf-8")` call. Fixes openclaw#54296.

steipete · 2026-05-11T11:46:20Z

Landed via rebase onto main.

Local focused proof: pnpm test src/config/sessions/transcript-stream.test.ts src/config/sessions/transcript.test.ts src/gateway/session-compaction-checkpoints.test.ts src/gateway/server.chat.gateway-server-chat.test.ts (2 Vitest shards, 58 tests passed)
Crabbox E2E proof: tbx_01krbcw4swx29ecnqnpewj5j50 (PR80615_E2E_OK, 8,389,986-byte transcript, latest assistant/dedupe/idempotency/fork checks passed)
GitHub checks: fresh PR head a3f4379a56a0ad4cfbaad0bcd74d5586c56abb04 green, including Real behavior proof, prod/test types, lint, build, gateway/core/security checks.
Landed commits: 9654239af9, 5da239473c

Thanks @jack-stormentswe!

openclaw-barnacle Bot added app: web-ui App: web-ui gateway Gateway runtime size: M proof: supplied External PR includes structured after-fix real behavior proof. labels May 11, 2026

davidv399 force-pushed the fix/transcript-streaming-scans branch from cd2ad07 to 42662bc Compare May 11, 2026 09:55

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 11, 2026

clawsweeper Bot mentioned this pull request May 11, 2026

Bug: Non-atomic write of exec-approvals file + unbounded transcript readFileSync #54296

Closed

jack-stormentswe and others added 2 commits May 11, 2026 12:35

fix(sessions): stream transcript reverse scans

a3f4379

steipete force-pushed the fix/transcript-streaming-scans branch from 552470c to a3f4379 Compare May 11, 2026 11:38

steipete merged commit 5da2394 into openclaw:main May 11, 2026
110 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(sessions): stream JSONL transcript scans instead of buffering whole files#80615

fix(sessions): stream JSONL transcript scans instead of buffering whole files#80615
steipete merged 2 commits into
openclaw:mainfrom
davidv399:fix/transcript-streaming-scans

davidv399 commented May 11, 2026 •

edited by steipete

Loading

Uh oh!

clawsweeper Bot commented May 11, 2026 •

edited

Loading

Uh oh!

davidv399 commented May 11, 2026

Uh oh!

clawsweeper Bot commented May 11, 2026 •

edited

Loading

Uh oh!

Uh oh!

steipete commented May 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

davidv399 commented May 11, 2026 • edited by steipete Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Verification

Real behavior proof

Uh oh!

clawsweeper Bot commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidv399 commented May 11, 2026

Uh oh!

clawsweeper Bot commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

steipete commented May 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

davidv399 commented May 11, 2026 •

edited by steipete

Loading

clawsweeper Bot commented May 11, 2026 •

edited

Loading

clawsweeper Bot commented May 11, 2026 •

edited

Loading