Skip to content

fix: avoid unexpected EOF on huge untracked files - #5

Open
ry2009 wants to merge 5 commits into
devfrom
fix/unexpected-eof-large-file
Open

ry2009 wants to merge 5 commits into
devfrom
fix/unexpected-eof-large-file

Conversation

@ry2009

@ry2009 ry2009 commented Dec 9, 2025 •

Copy link
Copy Markdown
Owner

Summary

  • stat untracked files and count lines only when size <= 5MB; above that, report 0/0 to avoid huge reads that caused "Unexpected EOF" on startup
  • keeps status data useful for normal-sized files while safeguarding against multi-GB files

Benchmark (single 200MB untracked file)

  • 1 MB text: first 13.5 ms, second 13.3 ms; lines=524,289 (inline + cached).

    • 10 MB text: first 82.6 ms, second 68.4 ms; lines initially 0 while background stream runs (fills cache
      shortly after).
    • 50 MB text: first 293.8 ms, second 304.9 ms; lines 0 initially, background stream (under 100 MB cap) fills
      cache after.
    • 100 MB text: ~9 ms both calls; background stream kicks off (at cap) to fill later.
    • 150 MB text: ~8.5 ms both calls; no background count (over 100 MB cap) so lines stay 0 by design.
    • 150 MB binary: ~9–10 ms; lines stay 0 (skipped).

    What changed vs the prior guardrail we moved from:

    • ≤5 MB: inline count once, cached.
    • 5–100 MB text: fast return with 0, background stream counts and caches; later status shows real lines.
    • 100 MB or binary: stat-only (0/0) to stay safe.

    • Cache (path+size+mtime) persisted to .opencode-line-cache.json in the project dir; no repeated reads once
      counted.

Testing

  • PATH=$HOME/.bun/bin:$PATH bun run packages/opencode/typecheck
  • bun turbo typecheck

@ry2009

ry2009 commented Dec 9, 2025 •

Copy link
Copy Markdown
Owner Author

bandaid guard rail fix for benchmarking as I dive deeper into mem/RAM

guardrail is only faster bc it doesn't execute the whole path and limits things not a real fix imo

@ry2009
ry2009 force-pushed the fix/unexpected-eof-large-file branch from fa6edbd to 4c1335f Compare December 9, 2025 23:28
@ry2009

ry2009 commented Dec 9, 2025

Copy link
Copy Markdown
Owner Author
  • Changed File.status to stat untracked files instead of reading them; tracked files remain
    unchanged.

    • removes the crash/hang path; status still lists the files (0/0 lines for untracked).
    • 200 MB untracked file → old 1.86s/~967 MB RSS vs new ~50 ms/~85 MB RSS.

    No user-facing limits; tracked large files are fine. Lmk any questions or concerns :)

Comment thread packages/opencode/src/file/index.ts Resolved
Comment thread packages/opencode/src/file/index.ts Outdated
if (!stat) continue
changedFiles.push({
path: filepath,
added: 0,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm losing # of lines kinda stinks

@ry2009 ry2009 Dec 9, 2025 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we reintroduce line counts, large untracked files put us back on the crash path unless we cap by size. Which then we would have the perf fix + a guardrail but could be a compromise

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I imagine this is a "solved problem" there is prolly something we can do to get line counts for most cases and have the handling for the ones we don't right?

@ry2009 ry2009 Dec 10, 2025 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed it to where we keep line counts for normal files and avoids the crash path for multi‑GB files.

This is the most optimal fix I saw from a deep dive and looking in stack overflow, we still also can:

  1. For large files, show size (from stat) so users still get useful metadata
  2. If we insist on line counts for huge files, add an opt‑in flag to run a streaming counter (e.g., wc -l) knowing it may be slow.
  3. configurable size threshold?

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possible deep cache and higher perf approach but not sure it will improve perf much if at all.

@ry2009 ry2009 Dec 10, 2025 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rekram1-node
Built custom untracked-line caches that meets the “get line counts for most cases and handle the ones we don’t”:

  • small/medium files keep counts
  • large text eventually gets counts
  • huge/binary stays safe.
  • Hot path remains fast; the only “slow” work s the one-off background stream for ≤100 MB text.

Let me know if this recent commit or
tweak: count lines for small untracked files
is preferred

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants