fix(deps): pin hyper to flush-before-shutdown fix for large-GET unexpected EOF#4797
Merged
Conversation
…ected EOF hyper <= 1.10.1 can call poll_shutdown() on an HTTP/1 socket while response bytes are still buffered (a prior poll_flush() returned Poll::Pending and the result was discarded), so a backpressured/slow-reading peer receives a graceful FIN before the full Content-Length body is flushed. Standard S3 clients (minio-go/warp) report this as sporadic `unexpected EOF` on large-object GET under load (rustfs/backlog#1232). This is the transport-layer bug from Cloudflare's "hyper-bug" writeup — distinct from the EC-reconstruct-desync EOF already fixed via lockstep decode; here the app layer delivers the full body and truncation is purely hyper->socket. Fixed upstream in hyperium/hyper#4018 (commit 72046cc7, "fix(http1): flush buffered data before shutdown"), which is not in any crates.io release yet as of hyper 1.10.1. Pin hyper via [patch.crates-io] to git rev ccc1e850 (a descendant of the fix that still declares version 1.10.1). Use [patch.crates-io], not a git+rev on the workspace `hyper` dependency: the server connection is driven by the transitive hyper-util (conn::auto / GracefulShutdown), and a git source would not unify with the crates.io hyper hyper-util resolves, leaving two hyper copies with the server path still on the buggy one. The patch rewrites the crates.io source globally, so the lockfile holds a single git-sourced hyper. Add rustfs/tests/hyper_h1_shutdown_flush_regression.rs, a deterministic guard (mirrors hyper's own h1_shutdown_while_buffered) that fails if the pin is dropped or hyper is downgraded below the fix. Its load-bearing assertion is a synchronously-set flag, so a slow runner can only under-detect, never false-red. Closes rustfs/backlog#1232.
Contributor
|
CLA requirements are satisfied for this pull request. |
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
overtrue
enabled auto-merge (squash)
July 13, 2026 14:43
…-shutdown pin cargo-deny's [sources] check denies unknown git sources. The hyper [patch.crates-io] pin added for rustfs/backlog#1232 uses a github.com/hyperium git source, so add it to allow-git with an owner/review note and a removal condition (drop once a released hyper > 1.10.1 carries commit 72046cc7).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Large-object (≥4 MiB) GET under high concurrency sporadically fails with
download error: unexpected EOFon standard S3 clients (minio-go / warp): theserver sends a valid
Content-Length, but the client receives a body shorterthan that and the connection closes. Frequency ~1e-5 (≈1 per 30k large GETs),
version-independent, reproducible on a 4-node erasure cluster under warp
get64-concurrency. Tracked in rustfs/backlog#1232.
Root cause — transport layer (hyper), not RustFS
The issue write-up localized this to the transport layer with a full evidence
chain: packet capture shows a graceful server-initiated FIN (zero RST); the
application streaming layer records zero anomalies (no short-read /
dropped-incomplete /
EOFDIAG);ecstore downstream_closedcount equals theEOF count and is a downstream effect of hyper closing the reader, not the
cause. The body was fully delivered to hyper; truncation happens hyper→socket.
This is the hyper bug described in Cloudflare's "hyper-bug" post: hyper's
HTTP/1 dispatch can call
poll_shutdown()on the socket while response bytesare still buffered — a prior
poll_flush()returnedPoll::Pendingand theresult was discarded. When a slow/backpressured peer fills the socket buffer,
hyper sends the FIN before the buffered tail is flushed → client
unexpected EOF. Fixed upstream in hyperium/hyper#4018 (commit72046cc7,"fix(http1): flush buffered data before shutdown").
This is distinct from the earlier EC-reconstruct-desync EOF (backlog#832,
already fixed via lockstep decode) — that one truncated at the reader; here
the reader delivers the full body and truncation is purely transport.
Fix
hyper 1.10.1(current) does not contain the fix and no crates.io releasecarries it yet. Pin hyper via
[patch.crates-io]to a git rev that does:ccc1e850is the current tip of official hyperium/hypermaster, a descendantof the fix commit, and still declares
version = "1.10.1"so it unifies thewhole tree.
Why
[patch.crates-io]and not agit+revon the workspacehyperdependency: the server connection is actually driven by the transitive
hyper-util(server::conn::auto+GracefulShutdown). A git dep on theworkspace
hyperline would be a different source than the crates.iohyperthat
hyper-utilresolves, so Cargo would keep two hyper copies and theserver path would still run the buggy one.
[patch.crates-io]rewrites thecrates.io source globally — the lockfile then has exactly one hyper, git-sourced
(verified). Git deps are already used in this workspace (datafusion), so this
adds no new build capability; the exact 40-char SHA in
Cargo.lockkeeps buildsreproducible.
Reproduction & regression guard
Deterministic reproduction (mirrors hyper's own
h1_shutdown_while_bufferedtest): wrap the server socket so it accepts one partial write then pends,
instrument
poll_shutdown. Against crates.io hyper 1.10.1 →shutdown w/ buffered = true, server FIN, client body 212893 < Content-Length 500000(BUG). Against the pinned rev →
shutdown w/ buffered = false, no FIN (CLEAN).Landed as
rustfs/tests/hyper_h1_shutdown_flush_regression.rs(runs in thedefault
-p rustfsnextest lane; note CI excludese2e_test). It fails if thepin is ever dropped / hyper downgraded below the fix — the one realistic
regression vector. The load-bearing assertion is a synchronously-set flag, so a
slow runner can only under-detect (false pass), never false-red.
Adversarial validation (6 roles — high risk: S3 API-visible transport behavior)
[patch.crates-io]isthe only mechanism that patches the hyper-util server path; distinct from the
EC-decode EOF; no regression in the 26-commit range.
net-hardens parsing (H2 CONNECT OOM + max_buf_size). Caveats: git-rev deps are
a cargo-audit/RUSTSEC blind spot (track manually, un-pin once released).
wedge (backpressured peer delaying graceful drain) is pre-existing and
hard-bounded by the 10s graceful-shutdown cap.
unchanged, single git-sourced hyper satisfies all
^1, wire- andcluster-backward-compatible.
O(1)/connection-close;
tracingdeliberately left off.Follow-ups (not in this PR)
before completion. The root cause is fixed here, so this only guards future
premature-abort causes; it touches app-layer streaming + logging governance
and is kept separate.
can stall a response-body write (pre-existing, bounded by the 10s graceful
cap). Consider a reverse-proxy write timeout or a per-connection wall-clock
timeout.
72046cc7.Closes rustfs/backlog#1232.