Repository navigation
compress: fix quality value of 0 being ignored in getEncoder - #296
Conversation
Coverage Report for CI Build 36429482251Coverage increased (+0.003%) to 97.144%Details
Uncovered ChangesNo uncovered changes found. Coverage RegressionsNo coverage regressions found. Coverage Stats
💛 - Coveralls |
System-Glitch
left a comment
There was a problem hiding this comment.
Thanks again for this PR. Compliance with standards is very important to me.
As explained in the PR description, the language detection is incorrect as well. A simple fix would be to add the following and a quick test case:
if lang.Priority == 0 { // Not acceptable according to RFC 9110
continue
}I'd like to bundle this change in the MR as well so we fix all compliance problems in one patch release.
I would add a small comment to httputil.ParseMultiValuesHeader to tell users that a priority of 0 means that it's not acceptable too.
|
Pushed |
There was a problem hiding this comment.
Thank you for the changes and additional docs and tests, that's awesome. 👍🏻
I kept the rejected list for the wildcard only: with the filter alone, br;q=0, * falls through to Encoders[0], which is br, and three table rows cover that. Happy to drop it with those rows if you prefer.
I didn't think about this specific case at first; hence why I thought at first that the rejected list wasn't necessary. But that's good we should keep that.
Currently a bit busy but I'll merge that soon. Thanks a lot for your contribution!
`getEncoder` grouped the parsed `Accept-Encoding` values by priority but never compared that priority to 0, so an encoding the client explicitly refused with `q=0` was still selected and the response was compressed with it. The parsed values are sorted by descending priority, so the zero-priority ones are now split off the end of the slice before the encoders are matched, and the loop only sees encodings the client actually accepts. When nothing is left, the middleware passes through uncompressed. The wildcard branch used to return the first configured encoder unconditionally, which returned the rejected encoding for a header such as `gzip;q=0, *`. It now returns the first configured encoder that the header did not reject, and nil when every encoder was rejected. That unconditional `m.Encoders[0]` also panicked with an index out of range when `Encoders` was empty and the request sent `Accept-Encoding: *`; the lookup that replaces it returns nil instead.
getEncoder now filters the parsed Accept-Encoding values with lo.FilterReject and returns early when no encoder is configured. The wildcard still skips the rejected encodings, because "br;q=0, *" must not select br. DetectLanguage ignores the languages given a quality value of 0, and the ParseMultiValuesHeader documentation says what a priority of 0 means.
84821b4 to
82dedc0
Compare
|
Sorry for the delay. This is merged now and I'll tag v5.12.5 very soon. Thanks for the contribution! 🎉 |
References
Issue(s): N/A
Discussion: N/A
Description
Middleware.getEncoder(middleware/compress/compress.go:144on master) groups the parsedAccept-Encodingvalues by priority at line 152 and matches the configured encoders against each group, but it never compares a priority to 0. RFC 9110 section 12.4.2 defines a quality value of 0 as "not acceptable", and section 12.5.3 applies that rule toAccept-Encoding: a listed content coding is acceptable unless it is accompanied by a qvalue of 0. The middleware therefore compresses responses with an encoding the client explicitly refused.With
Encoders: []Encoder{&Gzip{Level: gzip.BestCompression}}:Accept-Encoding: gzip;q=0comes back withContent-Encoding: gzipand a gzip framed body. A client that sendsq=0for gzip is exactly the client that cannot decode gzip, so the body is unreadable to it.Accept-Encoding: *;q=0does the same, even though it says no content coding is acceptable at all.Accept-Encoding: gzip;q=0, *also returns gzip, because the wildcard branch at line 165 returnsm.Encoders[0]unconditionally. The wildcard matches the codings that are not explicitly listed, so gzip is the one encoding it must not pick here.That same unconditional
m.Encoders[0]panics withindex out of range [0] with length 0whenEncodersis empty and the request sendsAccept-Encoding: *. Every other header value already returns nil for an emptyEncodersslice, so only the wildcard group reaches the panic.httputil.ParseMultiValuesHeaderalready parses the quality value correctly since #295 and returns aPriorityof 0 for theq=0entries in the first three headers. Only the consumer was missing.The fix splits the parsed values before any matching. They come back sorted by descending priority, so the zero-priority values are always a suffix:
slices.IndexFunclocates the first one, everything before it is accepted and everything from it onwards is rejected. When nothing is accepted,getEncoderreturns nil and the middleware passes the response through uncompressed. The wildcard branch now returns the first configured encoder that is not in the rejected list, and nil when every configured encoder was rejected or whenEncodersis empty, which removes the panic. The priority ordering, the rule that the earliest encoder in the slice wins on equal priority, and the hijack andUpgradeshort circuits are untouched. The doc comment onMiddlewarenow describes the rejection rule and links both RFC sections.Thirteen cases were added to the existing
TestEncoderPrioritytable:gzip;q=0,gzip;q=0.000,gzip; Q = 0,gzip;q=0.001,gzip;q=0, br,br;q=0, zstd;q=0, gzip;q=0,*;q=0,*;q=0, gzip,br;q=0, *,br;q=0, *;q=0.5,br;q=0, zstd;q=0, gzip;q=0, *,gzip;q=0, *with a single gzip encoder, and*with an emptyEncodersslice. ARejected encodingsubtest inTestCompressMiddlewarechecks that a full request withAccept-Encoding: gzip;q=0returns the plain body and noContent-Encodingheader. Ten of the table rows and that subtest fail on master, the emptyEncodersrow by panicking atcompress.go:165; the other three rows pass on master and guard behavior that does not change.go test -race -coverprofile=coverage.txt -covermode=atomic -coverpkg=./... ./middleware/compress/...passes andgo tool cover -funcreportsgetEncoderat 100% of statements.gofmt -s -l .prints nothing andgo vet ./middleware/compress/...is clean. golangci-lint was not run locally.Possible drawbacks
The headline change is a behavior change: a response to
Accept-Encoding: gzip;q=0used to be gzip encoded and is now sent uncompressed. That is the intended change, RFC 9110 section 12.5.3 requires it, no exported signature changes, and applications keep compiling and working without developer action.Two further header shapes change with it.
parseQuality(util/httputil/httputil.go:85) returns 0 both for an explicitq=0and for a quality value it cannot parse, and its doc comment says so. A request sendingAccept-Encoding: gzip;q=abc,gzip;q=1e-2or an out of range value such asgzip;q=-1used to be compressed at lowest priority and is now not compressed.The wildcard part is a separate change from the plain
q=0part.Accept-Encoding: gzip;q=0, *used to select the first encoder in the slice and now skips the rejected ones. RFC 9110 section 12.5.3 justifies it independently: the wildcard matches the codings that are not listed in the field, and gzip is listed there. The change only excludes codings the header rejected withq=0; a coding listed with a non-zero quality value can still be selected through the wildcard, as before.lang.Languages.DetectLanguage(lang/lang.go:166) reads the same parser and has the sameq=0gap forAccept-Language; I left it out of this change to keep it to the compress middleware.