Repository navigation
fsutil: fix zero padding being sniffed in DetectContentType - #301
Merged
System-Glitch merged 1 commit intoOct 23, 2026
Merged
System-Glitch merged 1 commit into
System-Glitch merged 1 commit into
Conversation
Coverage Report for CI Build 39963282342Coverage remained the same at 97.144%Details
Uncovered ChangesNo uncovered changes found. Coverage RegressionsNo coverage regressions found. Coverage Stats
💛 - Coveralls |
DetectContentType read into a 512-byte buffer and passed the whole buffer to http.DetectContentType, ignoring the number of bytes read. For content shorter than 512 bytes the rest of the buffer is zeros, and 0x00 is a binary data byte for the MIME sniffing algorithm, so short text without a BOM was never detected as text. Uploaded text files under 512 bytes got "application/octet-stream" as MIMEType (ParseMultipartFiles has no extension fallback), and small files served with Response.File/Download lacked the charset parameter that the same content gets at 512 bytes or more. Only sniff buffer[:n], as net/http does in serveContent. Co-Authored-By: Claude Opus 5.5 <[email protected]>
System-Glitch
force-pushed
the
fix/detect-content-type-padding
branch
from
October 23, 2026 13:41
dc60484 to
63142be
Compare
Member
|
I made a minor change, added a test case and rebased (adding signature). LGTM. |
System-Glitch
approved these changes
Oct 23, 2026
Member
|
Merged. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
References
Issue(s): N/A
Discussion: N/A
Description
fsutil.DetectContentType(util/fsutil/fsutil.go:181on master) reads into a 512-byte buffer and passes the whole buffer tohttp.DetectContentType(line 194). It throws away the byte count thatReadreturns (line 183). When the content is shorter than 512 bytes, the rest of the buffer is still zeros.http.DetectContentTypeimplements the MIME Sniffing Standard, which treats 0x00 as a binary data byte. The padding alone is enough to stop text from matching the text signature, so short text with no BOM and no HTML or XML signature is sniffed asapplication/octet-stream.This shows up in two places:
ParseMultipartFiles(util/fsutil/file.go:153, used by the parse middleware) callsDetectContentType(f, ""), so there is no extension fallback.File.MIMETypeisapplication/octet-streamfor every text upload under 512 bytes.validation.MIME("text/plain")rejects a 100-byte.txtupload, but accepts the same kind of file once it reaches 512 bytes.Response.File,Response.Downloadand the static handler (response.go:363) give different results depending on file size. A 300-bytestyle.cssis served astext/css, and a 600-byte one astext/css; charset=utf-8. A small text file with an unknown extension is served asapplication/octet-stream.The fix keeps
nand sniffsbuffer[:n].net/httpdoes the same inserveContent:n, _ := io.ReadFull(content, buf[:])thenDetectContentType(buf[:n]). The same slice goes tohasSVGSignature, which gives the same result as before.The existing tests did not catch this because
test_file.txt,test_script.shandtest_script.jsstart with a UTF-8 BOM. The BOM signature matches before the text signature is checked, so the padding has no effect on them.Tests:
TestDetectContentType"text" (short text, no known extension) andTestParseMultipartFiles"small_text_file". On master, both returnapplication/octet-stream.TestGetMIMEType:.gitignore,config/config.test.jsonandtest.css. InTestDetectContentType: thefile.jsoncase. Its content, "non-utf-8 content", is plain ASCII and therefore valid UTF-8 text, so I changed it to "content without BOM". Also threeTestStaticHandlercases and the "Write file" case of the compress middleware test. All of them now get thecharset=utf-8parameter that content of 512 bytes or more already gets, and.gitignorebecomestext/plain; charset=utf-8.Verification: this change was not compiled or run locally, because I had no Go toolchain available. CI will be the first real run. To work out the expected values, I ported
http.DetectContentTypeandfsutil.DetectContentTypeto Python. The Go side comes fromnet/http/sniff.goin go1.26.7, which is identical tonet/http/internal/sniff.goin go1.27.1. I ran both versions on the real test inputs. The unpatched model reproduces all 31 current expectations on the affected paths (fsutil, static, response, compress). The patched model produces exactly the 8 updated values and the 2 new ones.Possible drawbacks
Text content under 512 bytes is now detected as text, as it already is from 512 bytes up. Small text files served by
Response.File,Response.Downloador the static handler gain; charset=utf-8, or becometext/plain; charset=utf-8instead ofapplication/octet-streamwhen the extension is unknown. Small text uploads now gettext/plain; charset=utf-8asMIMEType.A single
Readcan still return fewer bytes than are available for some readers. That case was already sniffing zeros before this change.io.ReadFullwould handle it, but it would also change how the test reader's customEOFerror is handled, so I kept the singleRead.🤖 Generated with Claude Code