Skip to content

fsutil: fix zero padding being sniffed in DetectContentType - #301

Merged
System-Glitch merged 1 commit into
go-goyave:masterfrom
MaxFreedomPollard:fix/detect-content-type-padding
Oct 23, 2026
Merged

System-Glitch merged 1 commit into
go-goyave:masterfrom
MaxFreedomPollard:fix/detect-content-type-padding

Conversation

@MaxFreedomPollard

Copy link
Copy Markdown
Contributor

References

Issue(s): N/A
Discussion: N/A

Description

fsutil.DetectContentType (util/fsutil/fsutil.go:181 on master) reads into a 512-byte buffer and passes the whole buffer to http.DetectContentType (line 194). It throws away the byte count that Read returns (line 183). When the content is shorter than 512 bytes, the rest of the buffer is still zeros. http.DetectContentType implements the MIME Sniffing Standard, which treats 0x00 as a binary data byte. The padding alone is enough to stop text from matching the text signature, so short text with no BOM and no HTML or XML signature is sniffed as application/octet-stream.

This shows up in two places:

  • ParseMultipartFiles (util/fsutil/file.go:153, used by the parse middleware) calls DetectContentType(f, ""), so there is no extension fallback. File.MIMEType is application/octet-stream for every text upload under 512 bytes. validation.MIME("text/plain") rejects a 100-byte .txt upload, but accepts the same kind of file once it reaches 512 bytes.
  • Response.File, Response.Download and the static handler (response.go:363) give different results depending on file size. A 300-byte style.css is served as text/css, and a 600-byte one as text/css; charset=utf-8. A small text file with an unknown extension is served as application/octet-stream.

The fix keeps n and sniffs buffer[:n]. net/http does the same in serveContent: n, _ := io.ReadFull(content, buf[:]) then DetectContentType(buf[:n]). The same slice goes to hasSVGSignature, which gives the same result as before.

The existing tests did not catch this because test_file.txt, test_script.sh and test_script.js start with a UTF-8 BOM. The BOM signature matches before the text signature is checked, so the padding has no effect on them.

Tests:

  • Two new cases: TestDetectContentType "text" (short text, no known extension) and TestParseMultipartFiles "small_text_file". On master, both return application/octet-stream.
  • Eight updated expectations that recorded the padded result. In TestGetMIMEType: .gitignore, config/config.test.json and test.css. In TestDetectContentType: the file.json case. Its content, "non-utf-8 content", is plain ASCII and therefore valid UTF-8 text, so I changed it to "content without BOM". Also three TestStaticHandler cases and the "Write file" case of the compress middleware test. All of them now get the charset=utf-8 parameter that content of 512 bytes or more already gets, and .gitignore becomes text/plain; charset=utf-8.

Verification: this change was not compiled or run locally, because I had no Go toolchain available. CI will be the first real run. To work out the expected values, I ported http.DetectContentType and fsutil.DetectContentType to Python. The Go side comes from net/http/sniff.go in go1.26.7, which is identical to net/http/internal/sniff.go in go1.27.1. I ran both versions on the real test inputs. The unpatched model reproduces all 31 current expectations on the affected paths (fsutil, static, response, compress). The patched model produces exactly the 8 updated values and the 2 new ones.

Possible drawbacks

Text content under 512 bytes is now detected as text, as it already is from 512 bytes up. Small text files served by Response.File, Response.Download or the static handler gain ; charset=utf-8, or become text/plain; charset=utf-8 instead of application/octet-stream when the extension is unknown. Small text uploads now get text/plain; charset=utf-8 as MIMEType.

A single Read can still return fewer bytes than are available for some readers. That case was already sniffing zeros before this change. io.ReadFull would handle it, but it would also change how the test reader's custom EOF error is handled, so I kept the single Read.

🤖 Generated with Claude Code

@coveralls

coveralls commented Oct 22, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 39963282342

Coverage remained the same at 97.144%

Details

  • Coverage remained the same as the base build.
  • Patch coverage: 2 of 2 lines across 1 file are fully covered (100%).
  • No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 5848
Covered Lines: 5681
Line Coverage: 97.14%
Coverage Strength: 278.51 hits per line

💛 - Coveralls

@System-Glitch
System-Glitch self-requested a review October 23, 2026 07:25
@System-Glitch System-Glitch added the bug Something isn't working label Oct 23, 2026
@System-Glitch System-Glitch self-assigned this Oct 23, 2026
DetectContentType read into a 512-byte buffer and passed the whole
buffer to http.DetectContentType, ignoring the number of bytes read.
For content shorter than 512 bytes the rest of the buffer is zeros,
and 0x00 is a binary data byte for the MIME sniffing algorithm, so
short text without a BOM was never detected as text.

Uploaded text files under 512 bytes got "application/octet-stream" as
MIMEType (ParseMultipartFiles has no extension fallback), and small
files served with Response.File/Download lacked the charset parameter
that the same content gets at 512 bytes or more.

Only sniff buffer[:n], as net/http does in serveContent.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@System-Glitch
System-Glitch force-pushed the fix/detect-content-type-padding branch from dc60484 to 63142be Compare October 23, 2026 13:41
@System-Glitch

System-Glitch commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

I made a minor change, added a test case and rebased (adding signature). LGTM.

@System-Glitch
System-Glitch merged commit 63142be into go-goyave:master Oct 23, 2026
5 checks passed
@System-Glitch

Copy link
Copy Markdown
Member

Merged. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants