Skip to content

fix(usage): read Claude transcripts over 50 MB a page at a time - #3498

Merged
rileycrane merged 2 commits into
mainfrom
rileycrane/usage-large-transcripts
Nov 13, 2026
Merged

fix(usage): read Claude transcripts over 50 MB a page at a time#3498
rileycrane merged 2 commits into
mainfrom
rileycrane/usage-large-transcripts

Conversation

@rileycrane

@rileycrane rileycrane commented Nov 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Problem

Settings → Usage shows an error for an agent instead of its usage when any one of its Claude Code transcripts is over 50 MB:

Couldn't read the sessions of : ~/.claude/projects///subagents/agent-.jsonl is larger than 50 MB.

The app's native file read refuses files over 50 MB (src-tauri/src/host_files.rs). readUsage in src/bundled/claude-code/usage.ts passed every read error up except "no such file". So one long subagent transcript failed the read for all of that agent's sessions. Long-running subagents pass 50 MB in practice. The one that surfaced this was 74 MB.

Fix

When the app's read reports a file is over 50 MB, readUsage reads it instead with the plugin's existing read-page process (dd + base64, already in the manifest and used for large chat transcripts in transcript.ts). It reads 8 MiB pages and digests whole lines a page at a time:

  • The file's usage is still counted in full, not skipped or reported as a gap.
  • Each line is measured before it is decoded. A line over 50 MB (the app's whole-file limit) is never decoded or parsed. It counts as an unreadable line, so the session reports a gap and the rest of the file still counts. A line longer than the limit plus a page is dropped as it is read, so its bytes are never all held.
  • At most one page plus one line the app could read whole (≤ 50 MB) is held at once, before decoding and parsing copies. This is the same per-line bound a whole-file read already allows. Peak memory is higher than one page.
  • The old one-shot digest becomes an incremental digester(). A whole file goes through it once, giving the same start, title, times, observation keys and order, unreadable-line count and spawned names as before.
  • A file deleted before its pages are read is a "transcript is gone" gap, the same as for the app's read.
  • The paged read is not cancelled by a caller's signal, matching the app's own read. The cached promise is shared by every agent's concurrent read, so one caller's abort must not reject it for the others.

Non-goal: Codex rollouts over 50 MB (src/bundled/codex/usage.ts) have the same limit through a different reader. That fix is left for a separate change.

Tests

  • New usage.test.ts cases, run against the manifest's real processes:
    • Paged read: a subagent transcript over 8 MiB, with a line of two-byte characters across the first page's end. A streamed request's usage repeats on a later page and is counted once, with the later counts. There are no gaps. This fails on origin/main with the "larger than 50 MB" error.
    • Oversized lines: the limit is mocked to 12 MiB in this file. Three cases: a line over the limit completed within a full page, one unfinished at EOF, and one longer than the limit plus a page. Each reports one unreadable line and counts the lines around it. This fails on the first commit, 0be4861, which decoded those lines.
    • File gone: a transcript deleted before its pages are read is reported as gone. This fails without the gap handling.
  • bin/pnpm check passes.
  • bin/pnpm exec vitest run (full suite): 21,782 passed, 1 failed, at a8d65b6. The failure is browser-host/media-preparation.integration.test.mjs (tiled HEIC preparation). It does not reach the changed files, and it also failed on an earlier origin/main run (17da600).
  • Ran readUsage read-only against a real ~/.claude/projects holding a 74 MB subagent transcript, with the app's 50 MB limit simulated:
    • On origin/main it fails with the reported error.
    • With this change it reads all 57 of the agent's sessions in about 1.7 s, and the large session counts all 1,081 of its requests with no gaps.
  • With the real 50 MiB limit, synthetic single-record files of 51 MiB (with a newline and unterminated) and 120 MiB each report one unreadable line. A 49 MiB record is read. Node RSS sampling during these runs ranged from 38 to 746 MiB for the same inputs depending on test order, which mostly reflects GC timing, so I'm not claiming a tight peak. Desktop/WebKit memory is not measured.

Not yet exercised in the running desktop app's Usage page.

🤖 Generated with Claude Code

@rileycrane @claude
fix(usage): read Claude transcripts over 50 MB a page at a time
0be4861 
The Usage page failed every read of an agent's Claude Code sessions when
one transcript, usually a long subagent's, was larger than the app's
50 MB file read: the reader passed that error up, so the agent showed
"Couldn't read the sessions" instead of its usage.

Such a file is now read with the plugin's declared read-page process in
8 MiB pages and digested a page of whole lines at a time, so its usage
still counts and only a page's text is held at once. A file gone before
its pages are read is a gap, as for the app's read.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Riley Crane <[email protected]>
@rileycrane
rileycrane requested a review from a team as a code owner November 13, 2026 19:34
rileycrane
rileycrane commented Nov 13, 2026
View reviewed changes

@rileycrane rileycrane left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Independent accounting/failure review at 0be4861: no must-fix defect found in this scope. Incremental parsing preserves session attribution, keyed request replacement/order, unkeyed requests, titles, times, and gaps. Six synthetic probes with the actual usage module and manifest processes passed: paged/full equality for three EOF shapes with cross-page duplicate updates and multibyte text; permission/I/O failures on page two reject; deletion on page two discards partial subagent counts and reports a gap. This does not validate native desktop rendering or peak memory/resource bounds. One optional regression-coverage suggestion is inline.

Comment thread src/bundled/claude-code/usage.test.ts Resolved
rileycrane
rileycrane commented Nov 13, 2026
View reviewed changes

@rileycrane rileycrane left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Carl's adversarial resource/boundary review at 0be4861: one MUST-FIX line-bound defect is inline. Reading transcripts larger than 50 MiB is reasonable, and reusing the existing read-page process is a sound scoped approach. The shared 50 MiB cap is an application policy around whole-buffer reads, originally derived from upload policy (96360af, 52615f7); I found no hard Tauri ceiling at precisely that size in those owners/history. Raising it globally would increase allocation exposure across all plugin readers and change the documented host.readFile contract, so it is unnecessary here. The old upload-limit explanation is also stale as a universal rule: ordinary files now allow 100 MiB and the global limit is 500 MiB.

The one-page memory claim needs correcting: carryover can approach 50 MiB, with additional combined buffers, decoded strings, parsed JSON, base64 intermediates, and retained observations. A synthetic 51 MiB single-record file reached about 662 MiB maximum RSS in a fresh Node process running the actual usage module and manifest commands; the fixture was written in reused 1 MiB buffers. This is Node evidence, not a desktop/WebKit measurement, and eight readers can run together. No desktop Usage-screen or full-suite run was performed in this read-only review. Minimalness/elegance: 9/10; correctness: 8/10 until the inline guard defect is repaired.

Comment thread src/bundled/claude-code/usage.ts Resolved
@rileycrane @claude
fix(usage): leave a transcript line over 50 MB unread
a8d65b6 
The paged read checked the 50 MB line limit only on the bytes carried to
the next page, so a line completed within a page, or left unfinished at
the end of the file, was decoded and parsed at any size.

Each line is now measured before it is decoded. One larger than the
app's read is left unread and counted as an unreadable line, so the
session reports a gap and the rest of the file still counts; a line
longer than a page past the limit is dropped as it is read rather than
held.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Riley Crane <[email protected]>
wpfleger96
wpfleger96 approved these changes Nov 13, 2026
View reviewed changes

@wpfleger96 wpfleger96 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Reviewed at a8d65b6. No blocking findings.

  • Paging leaves results unchanged: seen and unkeyed carry across pages, and the made-up keys for requests without ids stay unique because seen.size only grows. Session start, title, times and spawned subagent names come out the same as from a whole-file read. Text is decoded only on whole-line boundaries, so a multi-byte character can't be split across pages.
  • The line limit holds: only the line carried in from the previous page can be over READ_LIMIT, and that case is checked before any text is decoded.
  • Page boundaries: I added throwaway tests for a file that ends exactly on an 8 MiB page boundary (macOS dd returns an empty last page there), a dropped oversized line whose newline is the first byte of a page, and a line exactly at the limit. All three behave correctly.

@rileycrane
rileycrane merged commit e4e3f78 into main Nov 13, 2026
64 checks passed
@rileycrane
rileycrane deleted the rileycrane/usage-large-transcripts branch November 13, 2026 21:19
joshfriend added a commit that referenced this pull request Nov 13, 2026
@joshfriend
Merge remote-tracking branch 'origin/main' into jfriend/github-merge-…
ab041fd 
…when-ready

* origin/main: (21 commits)
  fix(agents): report a conversation wake that fails before its turn starts (#2192)
  fix(usage): read Claude transcripts over 50 MB a page at a time (#3498)
  feat(agents): preview where interrupted agents were working before resume (#3494)
  perf(messages): close the editor and delete dialog on submit (#3493)
  feat(messages): quote selected message text into the composer (#2392)
  perf(toast): scope docked clearance to the toast host (#3489)
  fix(unread): undo exact captured ordinary reads conditionally (#3473)
  fix(solo): keep navigation responsive during chat deletion (#3488)
  fix(channels): keep custom thread tab names until the tab closes (#3458)
  perf(relay): verify cached history in the native host (#3426)
  fix(settings): refine claude and typesafe key setup dialogs (#3391)
  Update bundled help for Bestie and agent settings (#3194)
  feat(profiles): share and show a local time zone (#3332)
  Revert "feat(link-previews): unfurl web links below messages by default" (#3480)
  Show disabled plugin chats in Solo (#3456)
  perf(messages): host one context menu per message list (#3465)
  test(claude): keep the quit-release test copy above the minimum version (#3477)
  fix(profiles): restore verified names from the device cache (#3382)
  fix(mentions): show addressed agents first in Solo audience picker (#2856)
  perf: count member access once and scope find highlights to painting (#3464)
  ...
joshfriend added a commit that referenced this pull request Nov 13, 2026
@joshfriend
Merge remote-tracking branch 'origin/main' into jfriend/github-pr-ope…
1cc40d6 
…n-on-github

* origin/main: (21 commits)
  fix(agents): report a conversation wake that fails before its turn starts (#2192)
  fix(usage): read Claude transcripts over 50 MB a page at a time (#3498)
  feat(agents): preview where interrupted agents were working before resume (#3494)
  perf(messages): close the editor and delete dialog on submit (#3493)
  feat(messages): quote selected message text into the composer (#2392)
  perf(toast): scope docked clearance to the toast host (#3489)
  fix(unread): undo exact captured ordinary reads conditionally (#3473)
  fix(solo): keep navigation responsive during chat deletion (#3488)
  fix(channels): keep custom thread tab names until the tab closes (#3458)
  perf(relay): verify cached history in the native host (#3426)
  fix(settings): refine claude and typesafe key setup dialogs (#3391)
  Update bundled help for Bestie and agent settings (#3194)
  feat(profiles): share and show a local time zone (#3332)
  Revert "feat(link-previews): unfurl web links below messages by default" (#3480)
  Show disabled plugin chats in Solo (#3456)
  perf(messages): host one context menu per message list (#3465)
  test(claude): keep the quit-release test copy above the minimum version (#3477)
  fix(profiles): restore verified names from the device cache (#3382)
  fix(mentions): show addressed agents first in Solo audience picker (#2856)
  perf: count member access once and scope find highlights to painting (#3464)
  ...
joshfriend added a commit that referenced this pull request Nov 13, 2026
@joshfriend
Merge remote-tracking branch 'origin/main' into composer-focus-on-click
7b4e498 
* origin/main: (31 commits)
  Let plugin panels name their tab with panels.useTabTitle (#3251)
  fix(agents): report a conversation wake that fails before its turn starts (#2192)
  fix(usage): read Claude transcripts over 50 MB a page at a time (#3498)
  feat(agents): preview where interrupted agents were working before resume (#3494)
  perf(messages): close the editor and delete dialog on submit (#3493)
  feat(messages): quote selected message text into the composer (#2392)
  perf(toast): scope docked clearance to the toast host (#3489)
  fix(unread): undo exact captured ordinary reads conditionally (#3473)
  fix(solo): keep navigation responsive during chat deletion (#3488)
  fix(channels): keep custom thread tab names until the tab closes (#3458)
  perf(relay): verify cached history in the native host (#3426)
  fix(settings): refine claude and typesafe key setup dialogs (#3391)
  Update bundled help for Bestie and agent settings (#3194)
  feat(profiles): share and show a local time zone (#3332)
  Revert "feat(link-previews): unfurl web links below messages by default" (#3480)
  Show disabled plugin chats in Solo (#3456)
  perf(messages): host one context menu per message list (#3465)
  test(claude): keep the quit-release test copy above the minimum version (#3477)
  fix(profiles): restore verified names from the device cache (#3382)
  fix(mentions): show addressed agents first in Solo audience picker (#2856)
  ...
philazar added a commit that referenced this pull request Nov 13, 2026
@philazar
Merge remote-tracking branch 'origin/main' into coder3/identity-archi…
98159c5 
…ve-client

* origin/main:
  feat(solo): smooth chat recency reordering (#3502)
  Reject self-mentions that cannot wake another session (#3513)
  feat(status): offer don't clear for every status (#3515)
  Let plugin panels name their tab with panels.useTabTitle (#3251)
  fix(agents): report a conversation wake that fails before its turn starts (#2192)
  fix(usage): read Claude transcripts over 50 MB a page at a time (#3498)
  feat(agents): preview where interrupted agents were working before resume (#3494)
  perf(messages): close the editor and delete dialog on submit (#3493)
  feat(messages): quote selected message text into the composer (#2392)
  perf(toast): scope docked clearance to the toast host (#3489)
  fix(unread): undo exact captured ordinary reads conditionally (#3473)

Signed-off-by: Philip Azar <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wpfleger96 wpfleger96 wpfleger96 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@rileycrane @wpfleger96