-
Notifications
You must be signed in to change notification settings - Fork 2.8k
feat: add token-budget based file reading with intelligent preview #8789
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implements a simple, token-budget based file reading system that prevents context window overflow and tokenizer crashes. Problem: - Files could fill entire context window causing issues - tiktoken crashes with 'unreachable' error on files >5MB - PR #6667's approach was too complex with magic numbers Solution - Multi-Layer Defense: 1. Fast path: Files <100KB skip validation (no overhead) 2. Token validation: 100KB-5MB files use real token counting - Budget: (contextWindow - currentTokens) * 0.6 - Smart truncation if exceeds budget 3. Preview mode: Files >5MB get 100KB preview (prevents crashes) 4. Error recovery: Catch tokenizer 'unreachable' errors gracefully Key Features: - No magic numbers - dynamic based on actual context - Real token counting using existing tokenizer - 100KB previews for large files (perfect size for structure visibility) - Graceful error handling prevents conversation crashes - Simple implementation (~160 lines vs complex heuristics) Testing: - 17 comprehensive tests covering all scenarios - All tests passing including edge cases and error conditions Files: - src/core/tools/helpers/fileTokenBudget.ts: Core validation logic - src/core/tools/helpers/__tests__/fileTokenBudget.spec.ts: Test suite - src/core/tools/readFileTool.ts: Integration into read file tool
Contributor
|
Starting my review of this PR—comments incoming soon! 🚀 Follow Along on Roo Code Cloud Issues to AddressAll previously flagged issues have been resolved:
Latest Review: No new issues found. All previous concerns have been properly addressed. |
Improvements: - Preview files (>5MB) now use token counting to respect budget - Read only 100KB preview initially, then validate with tokenizer - If preview exceeds budget, truncate accordingly - Better error handling with conservative character-based estimation - All 17 tests passing
- Added getTokenUsage mock to createMockCline for readFileTool tests - Added contextWindow to model info mock - Updated fileTokenBudget test expectations for error handling - All 59 tests now passing (42 readFileTool + 17 fileTokenBudget)
…ncation - Previously used original file totalLines, causing mismatch after truncation - Now computes displayedLines from truncated content and sets lines="1-N" - Prevents LLM referencing non-existent line numbers - All tests passing (59/59)
…ation - Count all lines (including empty) when computing lines="1-N" - Prevents under-reporting when truncated preview contains blank lines - Tests remain green (42/42 readFileTool, 17/17 fileTokenBudget)
jr
approved these changes
Oct 23, 2025
Integrated countFileLinesAndTokens into validateFileTokenBudget: - Streams file once with chunked token estimation (256-line chunks) - Early exits when budget exceeded (saves I/O and memory) - Preserves all existing safety checks: - Fast path for <100KB files - Preview mode for >5MB files - Error handling for tokenizer crashes - Fallback to full read if streaming fails Benefits: - Single file pass with early exit vs full read + tokenize - Prevents loading large files into memory unnecessarily - Conservative fallback on tokenizer errors (2 chars = 1 token) - All existing tests passing (59/59) Files: - src/integrations/misc/line-counter.ts: Added countFileLinesAndTokens() - src/core/tools/helpers/fileTokenBudget.ts: Integrated streaming - src/integrations/misc/__tests__/line-counter.spec.ts: Basic tests
Two fixes: 1. Line counting off-by-one: Files ending with \n now count correctly - "line1\nline2\n" now correctly shows lines="1-2" not lines="1-3" - Consistent with countFileLines() behavior - Prevents LLM confusion about line numbers 2. Fixed line-counter.spec.ts mocking: - Use proper Readable stream instead of mock object - Properly mock fs.createReadStream with stream interface - All 63 tests passing (42 readFileTool + 17 fileTokenBudget + 4 line-counter) Files changed: - src/core/tools/readFileTool.ts: Handle trailing newline in line count - src/integrations/misc/__tests__/line-counter.spec.ts: Fix stream mocking
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
mrubens
approved these changes
Oct 23, 2025
mini2s
added a commit
to zgsm-ai/costrict
that referenced
this pull request
Oct 24, 2025
* fix: skip flaky Windows test in custom-system-prompt.spec.ts (RooCodeInc#8023) Co-authored-by: roomote[bot] <219738659+roomote[bot]@users.noreply.github.com> Co-authored-by: Roo Code <[email protected]> Co-authored-by: Matt Rubens <[email protected]> * feat: add GDPR-compliant cookie consent banner (RooCodeInc#8022) Co-authored-by: Roo Code <[email protected]> Co-authored-by: Bruno Bergher <[email protected]> Co-authored-by: Matt Rubens <[email protected]> * Release: v1.76.0 (RooCodeInc#8030) * Types 1.77.0 (RooCodeInc#8031) * feat(web): update homepage headline and tagline for remote control capabilities (RooCodeInc#7911) * Merge remote-tracking branch 'upstream/main' into roo-to-main * ux: Responsive Auto-Approve (RooCodeInc#8032) Co-authored-by: roomote[bot] <219738659+roomote[bot]@users.noreply.github.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * feat(cloud): Add telemetry retry queue for network resilience (RooCodeInc#7597) * feat(cloud): Add telemetry retry queue for network resilience - Implement RetryQueue class with workspace-scoped persistence - Queue failed telemetry events for automatic retry - Retry events every 60 seconds with fresh auth tokens - FIFO eviction when queue reaches 100 events - Persist queue across VS Code restarts This ensures telemetry data isn't lost during network failures or temporary server issues. Migrated from RooCodeInc/Roo-Code-Cloud#744 * fix: address PR review feedback for retry queue - Fix retry order to use consistent FIFO processing - Add retry limit enforcement with max retries check - Add configurable request timeout (default 30s) - Add comprehensive tests for retryAll() method - Add request-max-retries-exceeded event - Fix timeout test to avoid timing issues * fix: resolve TypeScript errors in RetryQueue tests * fix(cloud): Address PR feedback for telemetry retry queue - Handle HTTP error status codes (500s, 401/403, 429) as failures that trigger retry - Remove queuing of backfill operations since they're user-initiated - Fix race condition in concurrent retry processing with isProcessing flag - Add specialized retry logic for 429 with Retry-After header support - Clean up unnecessary comments - Add comprehensive tests for new status code handling - Add temporary debug logs with emojis for testing * refactor: address PR feedback for telemetry retry queue - Remove unused X-Organization-Id header from auth header provider - Simplify enqueue() API by removing operation parameter - Fix error retry logic: only retry 5xx, 429, and network failures - Stop retrying 4xx client errors (400, 401, 403, 404, 422) - Implement queue-wide pause for 429 rate limiting - Add auth state management integration: - Pause queue when not in active-session - Clear queue on logout or user change - Preserve queue when same user logs back in - Remove debug comments - Fix ESLint no-case-declarations error with proper block scope - Update tests for all new behaviors * Docs: Overhaul README (RooCodeInc#8005) * docs: overhaul README for clarity and impact * fix: update badge links and styles in README for improved visibility * docs(README): stronger CTAs for Discord/Reddit; restore Modes/Roles incl. cloud agents * docs(readme): add support CTA; refine modes section; clarify custom modes wording - add centered help/links CTA under badges: [README.md](README.md:8-10) - expand feature list to explicitly include modes: [README.md](README.md:45) - rename section header “Modes and Roles” → “Modes”: [README.md](README.md:52) - update wording to “Custom Modes: build specialized modes for your team or workflow”: [README.md](README.md:60) * docs: implement @jdilla1277 suggestions for English README - Updated header to "Your AI-Powered Dev Team, Right in Your Editor" (removed Cline reference) - Changed line 44 to "Generate Code from natural language descriptions and specs" for better SEO - Removed "not the other way around" phrase from line 54 - Replaced "Cloud Agents" with "Roomote Control" on line 61 As requested by @hannesrudolph, these changes are only applied to the English README. * docs: clarify Roomote Control description and update related links in README * Reworked the contributor list. Made translastion corrections. --------- Co-authored-by: Roo Code <[email protected]> * Add organizationFeaturesSchema with roomoteControlEnabled (RooCodeInc#8085) feat: add organizationFeaturesSchema with roomoteControlEnabled - Add organizationFeaturesSchema with optional roomoteControlEnabled boolean - Integrate features property into organizationSettingsSchema as optional - Add comprehensive tests for new schema validation - Maintain backward compatibility with existing organization settings Co-authored-by: Roo Code <[email protected]> * Release: v1.78.0 (RooCodeInc#8089) chore: bump version to v1.78.0 * Get the model id property for a given provider (RooCodeInc#8009) * Fix: Transform keybindings in nightly build to fix command+y shortcut (RooCodeInc#8070) * fix: transform keybindings command references in nightly build The keybindings section was not being transformed during the nightly build process, causing command+y keybinding to reference the wrong command name (roo-cline.addToContext instead of roo-code-nightly.addToContext). - Added keybindings schema to types.ts - Updated generatePackageJson to transform keybindings command references - This ensures keybindings work correctly in the nightly build * fix: only include keybindings in output when they exist Updated generatePackageJson to conditionally add keybindings to avoid including undefined values in the generated package.json. Fixed eslint-disable comment placement. --------- Co-authored-by: Roo Code <[email protected]> * Add a list of sub-processors (RooCodeInc#8163) * Add cookie policy (RooCodeInc#8165) * ux: Redesigned Message Feed (RooCodeInc#7985) Co-authored-by: Roo Code <[email protected]> Co-authored-by: Matt Rubens <[email protected]> * Merge branch 'main' of github.com:zgsm-ai/costrict into roo-to-main * bug: Incorrect copy for "Run" command when approving execution (RooCodeInc#8179) * test: set default language to English in prompt tests * Supernova (RooCodeInc#8175) Co-authored-by: Chris Estreich <[email protected]> * chore: add changeset for v3.28.4 (RooCodeInc#8182) * Changeset version bump (RooCodeInc#8183) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Matt Rubens <[email protected]> * feat(chat): enhance user info display and file path highlighting * fix: support dash prefix in parseMarkdownChecklist for todo lists (RooCodeInc#8055) - Updated regex pattern to support optional dash prefix (e.g., "- [ ] Task") - Added comprehensive test coverage for both formats - Fixes issue where todo lists with dash prefixes were not being parsed correctly Fixes RooCodeInc#8054 Co-authored-by: Roo Code <[email protected]> * Updating the privacy policy to allow us to send occasional emails (RooCodeInc#8180) * Updating the privacy policy to allow us to send onboarding and occasional marketing emails. * docs: Clarified unsubscribe behavior and updated the Last Updated date * docs: update Last Updated date in Privacy Policy page * fix: apply tiered pricing for Gemini models via Vertex AI (RooCodeInc#8018) * fix: apply tiered pricing for Gemini models via Vertex AI - Modified calculateCost method to handle models where cacheReadsPrice is only defined in tiers - Added comprehensive tests for Vertex AI tiered pricing calculation - Fixes issue where local cost calculation always showed highest tier rates Fixes RooCodeInc#8017 * Delete src/api/providers/__tests__/vertex-tiered-pricing.spec.ts --------- Co-authored-by: Roo Code <[email protected]> Co-authored-by: Daniel <[email protected]> * update sambanova models (RooCodeInc#8186) * update sambanova models * fix typo * Fix duplicate rehydrate during reasoning; centralize rehydrate and preserve cancel metadata (RooCodeInc#8171) Co-authored-by: daniel-lxs <[email protected]> * Wrap code blocks by default (RooCodeInc#8194) * Merge remote-tracking branch 'upstream/main' into roo-to-main * Add an announcement for Supernova (RooCodeInc#8197) * Add an announcement for Supernova * Remove duplicate keys * refactor(chat): optimize ChatRow rendering and user info handling * Release v3.28.5 (RooCodeInc#8198) chore: add changeset for v3.28.5 * Changeset version bump (RooCodeInc#8199) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Remove language selection and word wrap toggle from CodeBlock (RooCodeInc#8208) * feat: Add keyboard shortcut for toggling auto-approve (Cmd/Ctrl+Alt+A) (RooCodeInc#8214) Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> Co-authored-by: Roo Code <[email protected]> Co-authored-by: Matt Rubens <[email protected]> * Bare metal evals fixes (RooCodeInc#8224) Co-authored-by: Roo Code <[email protected]> * Fix: Improve reasoning block formatting for better readability (RooCodeInc#7868) * fix: prevent checkpoint text from wrapping in non-English languages (RooCodeInc#8207) Co-authored-by: Roo Code <[email protected]> * fix: respect Ollama Modelfile num_ctx configuration (RooCodeInc#7798) Co-authored-by: Roo Code <[email protected]> Co-authored-by: Hannes Rudolph <[email protected]> Co-authored-by: daniel-lxs <[email protected]> * feat: add package.nls.json checking to find-missing-translations script (RooCodeInc#8255) Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> Co-authored-by: Roo Code <[email protected]> Co-authored-by: daniel-lxs <[email protected]> Co-authored-by: Matt Rubens <[email protected]> * feat(types): add GPT-5-Codex model (RooCodeInc#8260) * chore: add changeset for v3.28.6 (RooCodeInc#8261) * Follow-up questions should trigger the "interactive" state (RooCodeInc#8259) * chore: add changeset for v3.28.6 (RooCodeInc#8263) * Changeset version bump (RooCodeInc#8262) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Matt Rubens <[email protected]> * Add image for 3.28.5 (RooCodeInc#8266) * feat: add zai-org/GLM-4.5-turbo model to Chutes provider (RooCodeInc#8157) Co-authored-by: Roo Code <[email protected]> * Cloud account switcher (RooCodeInc#8223) * Cloud account switcher * Bare metal evals fixes (RooCodeInc#8224) Co-authored-by: Roo Code <[email protected]> * Rounded icons in chat account switcher * Visual tweaks to CloudView * Remove hardcoded timeout * Safer check * PR feedback * Fix test * Check for org mismatches in handleCredentialsChange * Cloud: use the existing auth event flow to handle org switching * Cleanup: broadcast() might be confusingly named. --------- Co-authored-by: Chris Estreich <[email protected]> Co-authored-by: Roo Code <[email protected]> Co-authored-by: Bruno Bergher <[email protected]> Co-authored-by: John Richmond <[email protected]> * fix: checkpoint restore popover positioning issue (RooCodeInc#8219) (RooCodeInc#8220) Co-authored-by: Roo Code <[email protected]> Co-authored-by: daniel-lxs <[email protected]> * ux: Collapse thinking blocks by default (but control all of them with a keyboard shortcut) (RooCodeInc#8254) Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> Co-authored-by: Roo Code <[email protected]> Co-authored-by: Matt Rubens <[email protected]> * Remove org switcher tooltip (RooCodeInc#8269) * chore: add changeset for v3.28.7 (RooCodeInc#8268) * Changeset version bump (RooCodeInc#8271) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Matt Rubens <[email protected]> * Merge remote-tracking branch 'upstream/main' into roo-to-main * fix(i18n): spelling mistake in zh-CN chat.json * web: Opengraph image (RooCodeInc#8285) Co-authored-by: Matt Rubens <[email protected]> * Set port 3446 for web-evals in production mode (RooCodeInc#8288) * Show cloud switcher and option to add a team when logged in (RooCodeInc#8291) * fix: include initial ask in condense summarization (RooCodeInc#8293) (RooCodeInc#8298) * fix: include initial ask in condense summarization (RooCodeInc#8293) --------- Co-authored-by: Matt Rubens <[email protected]> * Fix frequent "No tool used" errors by clarifying tool-use rules (RooCodeInc#8292) * Add a couple more free models to the Roo provider (RooCodeInc#8304) * Release v3.28.8 (RooCodeInc#8305) * Changeset version bump (RooCodeInc#8306) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Matt Rubens <[email protected]> * feat: log out from cloud when resetting extension state (RooCodeInc#8312) Co-authored-by: Roo Code <[email protected]> Co-authored-by: Matt Rubens <[email protected]> * Merge remote-tracking branch 'upstream/main' into roo-to-main * feat: Add telemetry tracking to DismissibleUpsell component (RooCodeInc#8309) * feat: add telemetry tracking to DismissibleUpsell component - Added UPSELL_DISMISSED and UPSELL_CLICKED events to TelemetryEventName enum - Updated DismissibleUpsell component to track clicks and dismissals with telemetry - Added telemetry tests to DismissibleUpsell test suite - Events include upsellId in the payload for tracking specific upsells * refactor(webview): make handleDismiss synchronous in DismissibleUpsell test(webview): add scenario where dismissOnClick=true without onClick tracks only UPSELL_DISMISSED; update tests; all tests passing locally --------- Co-authored-by: Roo Code <[email protected]> * Correct tool use suggestion to improve model adherence to suggestion (RooCodeInc#8315) * Correct tool use suggestion to improve model adherence to suggestion * tweak * removing user hint when refreshing models (RooCodeInc#7710) * feat(zgsm): add quota info display and periodic updates * Show the Roo provider on the welcome screen (RooCodeInc#8317) * wip: Website Improvements (RooCodeInc#8303) Co-authored-by: Matt Rubens <[email protected]> * refactor: remove pr-reviewer mode (RooCodeInc#8222) * feat: add animated quota display with staggered transitions * Merge remote-tracking branch 'upstream/main' into roo-to-main * web: More website copy tweaks (RooCodeInc#8326) Co-authored-by: Matt Rubens <[email protected]> * fix: remove <thinking> tags from prompts for cleaner output and fewer tokens (RooCodeInc#8319) Co-authored-by: Roo Code <[email protected]> Co-authored-by: Hannes Rudolph <[email protected]> * Upgrade Supernova (RooCodeInc#8330) * chore: add changeset for v3.28.9 (RooCodeInc#8336) * Changeset version bump (RooCodeInc#8337) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Matt Rubens <[email protected]> * Track when telemetry settings change (RooCodeInc#8339) * fix: use max_completion_tokens for GPT-5 models in LiteLLM provider (RooCodeInc#6980) Co-authored-by: Roo Code <[email protected]> Co-authored-by: daniel-lxs <[email protected]> * Make chat icons shrink-0 (RooCodeInc#8343) * web: Testimonials (RooCodeInc#8360) * Adds lots of testimonials, 5-stars from marketplace * Fits more testimonials in one page * Testimonial heading tweak * ci: refresh contrib.rocks cache workflow (RooCodeInc#8083) * feat: add Claude 4.5 Sonnet model across all providers (RooCodeInc#8368) * chore: add changeset for v3.28.10 (RooCodeInc#8369) * Changeset version bump (RooCodeInc#8370) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Matt Rubens <[email protected]> * fix: correct AWS Bedrock Claude Sonnet 4.5 model identifier (RooCodeInc#8372) Fixes RooCodeInc#8371 - Updates the model ID from anthropic.claude-4.5-sonnet-v1:0 to anthropic.claude-sonnet-4-5-20250929-v1:0 to match AWS Bedrock naming convention Co-authored-by: Roo Code <[email protected]> * fix: correct Claude Sonnet 4.5 model ID format (RooCodeInc#8373) * chore: add changeset for v3.28.11 (RooCodeInc#8374) * Changeset version bump (RooCodeInc#8375) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Matt Rubens <[email protected]> * fix: Anthropic Sonnet 4.5 model id + Bedrock 1M context checkbox (RooCodeInc#8384) fix(anthropic): use claude-sonnet-4-5 id fix(bedrock): enable 1M context checkbox for Sonnet 4.5 via shared list closes RooCodeInc#8379 closes RooCodeInc#8381 * chore: add changeset for v3.28.12 (RooCodeInc#8385) * Changeset version bump (RooCodeInc#8376) * changeset version bump * Revise changelog for version 3.28.12 Updated version number and consolidated patch notes. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Matt Rubens <[email protected]> * Fix Vertex Sonnet 4.5 (RooCodeInc#8391) * fix: remove topP parameter from Bedrock inference config (RooCodeInc#8388) Co-authored-by: Matt Rubens <[email protected]> * chore: add changeset for v3.28.13 (RooCodeInc#8393) * Changeset version bump (RooCodeInc#8394) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Matt Rubens <[email protected]> * feat: add GLM-4.6 model support for z.ai provider (RooCodeInc#8408) Co-authored-by: Roo Code <[email protected]> * chore: add changeset for v3.28.14 (RooCodeInc#8413) * Changeset version bump (RooCodeInc#8414) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Matt Rubens <[email protected]> * A couple more sonnet 4.5 fixes (RooCodeInc#8421) * chore: Remove unsupported Gemini 2.5 Flash Image Preview free model (RooCodeInc#8359) * Include reasoning messages in cloud tasks (RooCodeInc#8401) * fix: show send button when only images are selected in chat textarea (RooCodeInc#8423) Co-authored-by: Roo Code <[email protected]> Co-authored-by: Matt Rubens <[email protected]> * Add structured data to the homepage (RooCodeInc#8427) Co-authored-by: roomote[bot] <219738659+roomote[bot]@users.noreply.github.com> * fix(ui): disable send button when no input content and update tests * fix: Addresses overeager 'there are unsaved changes' dialog in settings (RooCodeInc#8410) Fixes overeager 'there are unsaved changes' dialog in settings * feat: add UsageStats schema and type (RooCodeInc#8441) feat: add UsageStats schema and type to cloud.ts Co-authored-by: Roo Code <[email protected]> * Release: v1.80.0 (RooCodeInc#8442) chore: bump version to v1.80.0 * feat: add new DeepSeek and GLM models with detailed descriptions to the Chutes provider (RooCodeInc#8467) * Deprecate free grok 4 fast (RooCodeInc#8481) Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * fix: improve save button activation in prompts settings (
Problem
Files can fill the entire context window, and tiktoken crashes with
RuntimeError: unreachableon files >5MB.Solution
Token-budget based file reading with multi-layer protection against context overflow and tokenizer crashes.
How It Works
Implementation
Layer 1: Fast Path (< 100KB)
Layer 2: Token Validation (100KB - 5MB)
(contextWindow - currentTokens) * 0.6Layer 3: Preview Mode (> 5MB)
line_rangefor targeted reading 👁️Layer 4: Error Recovery
unreachableerrors gracefullyBenefits
✅ Dynamic budget based on actual context (no magic numbers)
✅ Real token counting using existing tokenizer
✅ 100KB previews for large files
✅ Graceful error handling prevents conversation crashes
✅ Simple (~160 lines) vs complex heuristics
✅ 17 comprehensive tests covering all scenarios
Testing
All 17 tests passing: fast path, budget validation, preview mode, error recovery, edge cases, unicode support.
Related
Closes #6667
Important
Introduces token-budget based file reading in
readFileTool.tsto handle large files efficiently, with new functions for budget validation and content truncation, and comprehensive tests.readFileTool.tsto prevent context overflow and tokenizer crashes.validateFileTokenBudgetandtruncateFileContentinfileTokenBudget.tsfor handling large files.fileTokenBudget.spec.tswith 17 tests covering scenarios like small files, large files, budget validation, and error handling.line-counter.spec.tsto test line and token counting with error handling.line-counter.tsto include token estimation alongside line counting.readFileTool.spec.tsto test new file reading behavior.This description was created by
for 169fb35. You can customize this summary. It will automatically update as commits are pushed.