Repository navigation
fix(mime): flush TextDecoder to catch truncated UTF-8 sequences at buffer boundary - #55946
smartworldarafath wants to merge 1 commit into
Conversation
The isText() function in mime.ts was calling TextDecoder.decode() with
{ stream: true } but never calling the flush (a bare .decode() call at
the end). A fatal TextDecoder in streaming mode buffers incomplete
multi-byte sequences silently — the error is only thrown during the
final flush.
This means a file ending with a partial multi-byte sequence (e.g. only
the first byte of a 2-byte or 3-byte UTF-8 codepoint) was incorrectly
classified as 'text/plain' instead of 'application/octet-stream'.
Fix: store the decoder instance, call decode() with stream:true, then
call decode() once more with no arguments to flush and surface any
incomplete sequence at the buffer boundary.
github-actions
Bot
added
needs:compliance
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
|
The following comment was made by an LLM, it may be inaccurate: |
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Looks correct. I ran Mime.detect on both checkouts with your examples: on the base, Hello followed by a cut-off 0xC3 or 0xE2 0x82 is classified as text/plain. With this change it's application/octet-stream, which matches how invalid UTF-8 in the middle of a buffer is already treated. Complete ASCII and UTF-8 input gives the same result as before. The read, read-filesystem and session-prompt tests and the packages/core typecheck pass.
I also checked the callers. The read tool passes only the first 256 KB of a file, but it uses detect only to recognise images and PDFs, so whether that chunk counts as text doesn't matter there. Prompt attachments pass the whole file, so flushing the decoder is correct.
The public API is unchanged.
github-actions
Bot
removed
needs:compliance
|
Thanks for updating your PR! It now meets our contributing guidelines. 👍 |
Issue for this PR
Closes #56183
Type of change
What does this PR do?
When
isBinarydecodes a chunk withdecoder.decode(chunk, { stream: true }), it tellsTextDecoderto expect more chunks. If the chunk ends mid-way through a multi-byte UTF-8 character (e.g.0xC3or0xE2 0x82),TextDecoderbuffers the trailing incomplete byte(s) without throwing an error in streaming mode.Because
isBinaryimmediately finishes its loop without flushing the decoder, truncated multi-byte UTF-8 sequences at chunk boundaries were silently accepted as valid text instead of binary.This PR adds
decoder.decode()to flush any buffered incomplete bytes before returningfalse, ensuring invalid or truncated trailing multi-byte sequences throwTypeErrorand correctly classify the input as binary (application/octet-stream).How did you verify your code works?
Mime.detecton buffers with incomplete multi-byte sequences (e.g. truncated0xC3/0xE2 0x82), verifying they are classified as binary.Screenshots / recordings
N/A (MIME detection logic)
Checklist