Skip to content

fix(fetch): preserve UTF-8 when truncating fetched content - #4025

Open
imharjot wants to merge 1 commit into
charmbracelet:mainfrom
imharjot:fix-fetch-utf8-truncation
Open

imharjot wants to merge 1 commit into
charmbracelet:mainfrom
imharjot:fix-fetch-utf8-truncation

Conversation

@imharjot

@imharjot imharjot commented Oct 4, 2026 •

Copy link
Copy Markdown

Fetching a large UTF-8 page can fail with Response content is not valid UTF-8 when the byte limit lands inside a character. For example, 102399 ASCII bytes followed by 界 trigger the error even though the response is valid UTF-8. Formatting the response as Markdown can also split a character at the output limit.

Read a small, bounded lookahead to validate the character at the boundary, then keep a complete UTF-8 prefix. This fixes both fetch and sub-agent web_fetch while preserving errors for malformed input. The existing truncation notice now reflects whether the response was actually cut, including when HTML extraction makes the result shorter; a complete response exactly at the limit no longer gets a notice.

Added 12 local HTTP regression cases covering two-, three-, and four-byte characters, Markdown output, exact limits, and malformed input. Kept upstream’s refreshed cassette and updated two recorded requests to include the truncation notices for the plain-text and HTML fetch results. The recorded responses were preserved and replayed locally.

Validation on macOS with Go 1.27.0, rebased onto 3b1f8604f0027cf5fa47f29a27655ef1aeed8d8f (head 4eeedcdd2a7230d380370ab76299a3bf5af230c1):

  • The UTF-8 regressions fail on the unpatched upstream helpers and pass with this fix.
  • go test -race ./internal/agent/tools -run TestFetch -count=1 and the DeepSeek fetch cassette replay passed.
  • go mod tidy left the module files unchanged and go build -race ./... passed. The full go test -race -failfast ./... rerun passed. The first run reported a race in the unchanged Docker MCP backend test; it did not reproduce in five targeted or six full backend-package runs on upstream.
  • CI-pinned golangci-lint v2.13.1, log-capitalization lint and git diff --check passed.

The rebase preserves the newer upstream fetch error handling and its tests. No dependency, workflow or unrelated source changes are included. GitHub Actions results are separate from these local checks.

@charmcli

charmcli commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@imharjot

imharjot commented Oct 4, 2026

Copy link
Copy Markdown
Author

I have read the Contributor License Agreement (CLA) and hereby sign the CLA.

@imharjot
imharjot force-pushed the fix-fetch-utf8-truncation branch from a592f25 to 58bbed1 Compare October 22, 2026 05:45
@imharjot

Copy link
Copy Markdown
Author

Rebased onto 17fccc0, keeping the newer fetch network-error and HTTP-status tests alongside the UTF-8 regressions. The complete race-enabled suite (63 test packages), build and CI-pinned golangci-lint v2.13.1 all pass locally. The CLA check also passed. The new build, security and lint workflows require maintainer approval; could someone approve those fork runs so the remote matrix can check this head?

@imharjot
imharjot force-pushed the fix-fetch-utf8-truncation branch from 58bbed1 to 4eeedcd Compare November 3, 2026 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants