Skip to content

fix(deps): pin hyper to flush-before-shutdown fix for large-GET unexpected EOF#4797

Merged
overtrue merged 2 commits into
mainfrom
overtrue/jovial-bhabha-fbb3ac
Jul 13, 2026
Merged

fix(deps): pin hyper to flush-before-shutdown fix for large-GET unexpected EOF#4797
overtrue merged 2 commits into
mainfrom
overtrue/jovial-bhabha-fbb3ac

Conversation

@overtrue

Copy link
Copy Markdown
Collaborator

Problem

Large-object (≥4 MiB) GET under high concurrency sporadically fails with
download error: unexpected EOF on standard S3 clients (minio-go / warp): the
server sends a valid Content-Length, but the client receives a body shorter
than that and the connection closes. Frequency ~1e-5 (≈1 per 30k large GETs),
version-independent, reproducible on a 4-node erasure cluster under warp get
64-concurrency. Tracked in rustfs/backlog#1232.

Root cause — transport layer (hyper), not RustFS

The issue write-up localized this to the transport layer with a full evidence
chain: packet capture shows a graceful server-initiated FIN (zero RST); the
application streaming layer records zero anomalies (no short-read /
dropped-incomplete / EOFDIAG); ecstore downstream_closed count equals the
EOF count and is a downstream effect of hyper closing the reader, not the
cause. The body was fully delivered to hyper; truncation happens hyper→socket.

This is the hyper bug described in Cloudflare's "hyper-bug" post: hyper's
HTTP/1 dispatch can call poll_shutdown() on the socket while response bytes
are still buffered — a prior poll_flush() returned Poll::Pending and the
result was discarded. When a slow/backpressured peer fills the socket buffer,
hyper sends the FIN before the buffered tail is flushed → client unexpected EOF. Fixed upstream in hyperium/hyper#4018 (commit 72046cc7,
"fix(http1): flush buffered data before shutdown").

This is distinct from the earlier EC-reconstruct-desync EOF (backlog#832,
already fixed via lockstep decode) — that one truncated at the reader; here
the reader delivers the full body and truncation is purely transport.

Fix

hyper 1.10.1 (current) does not contain the fix and no crates.io release
carries it yet. Pin hyper via [patch.crates-io] to a git rev that does:

[patch.crates-io]
hyper = { git = "https://github.com/hyperium/hyper.git", rev = "ccc1e850dc0cda3e71b0acd11f60ca3d48d09034" }

ccc1e850 is the current tip of official hyperium/hyper master, a descendant
of the fix commit, and still declares version = "1.10.1" so it unifies the
whole tree.

Why [patch.crates-io] and not a git+rev on the workspace hyper
dependency:
the server connection is actually driven by the transitive
hyper-util (server::conn::auto + GracefulShutdown). A git dep on the
workspace hyper line would be a different source than the crates.io hyper
that hyper-util resolves, so Cargo would keep two hyper copies and the
server path would still run the buggy one. [patch.crates-io] rewrites the
crates.io source globally — the lockfile then has exactly one hyper, git-sourced
(verified). Git deps are already used in this workspace (datafusion), so this
adds no new build capability; the exact 40-char SHA in Cargo.lock keeps builds
reproducible.

Reproduction & regression guard

Deterministic reproduction (mirrors hyper's own h1_shutdown_while_buffered
test): wrap the server socket so it accepts one partial write then pends,
instrument poll_shutdown. Against crates.io hyper 1.10.1 → shutdown w/ buffered = true, server FIN, client body 212893 < Content-Length 500000
(BUG). Against the pinned rev → shutdown w/ buffered = false, no FIN (CLEAN).

Landed as rustfs/tests/hyper_h1_shutdown_flush_regression.rs (runs in the
default -p rustfs nextest lane; note CI excludes e2e_test). It fails if the
pin is ever dropped / hyper downgraded below the fix — the one realistic
regression vector. The load-bearing assertion is a synchronously-set flag, so a
slow runner can only under-detect (false pass), never false-red.

Adversarial validation (6 roles — high risk: S3 API-visible transport behavior)

  • Correctness — fix present & semantically correct; [patch.crates-io] is
    the only mechanism that patches the hyper-util server path; distinct from the
    EC-decode EOF; no regression in the 26-commit range.
  • Security/supply-chain — rev is immutable (40-char SHA) on official master;
    net-hardens parsing (H2 CONNECT OOM + max_buf_size). Caveats: git-rev deps are
    a cargo-audit/RUSTSEC blind spot (track manually, un-pin once released).
  • Concurrency/durability — transport-only, durability-orthogonal; the only
    wedge (backpressured peer delaying graceful drain) is pre-existing and
    hard-bounded by the 10s graceful-shutdown cap.
  • Compatibility/build — additive-only public API, MSRV/edition/version
    unchanged, single git-sourced hyper satisfies all ^1, wire- and
    cluster-backward-compatible.
  • Performance — data-plane zero regression; flush-before-shutdown is
    O(1)/connection-close; tracing deliberately left off.
  • Test-coverage — resolved by the regression test above.

Follow-ups (not in this PR)

  • Observability (issue item a): a server-side WARN when body streaming aborts
    before completion. The root cause is fixed here, so this only guards future
    premature-abort causes; it touches app-layer streaming + logging governance
    and is kept separate.
  • Operational: hyper HTTP/1 has no write/idle-write timeout, so a slow reader
    can stall a response-body write (pre-existing, bounded by the 10s graceful
    cap). Consider a reverse-proxy write timeout or a per-connection wall-clock
    timeout.
  • Drop this patch once a released hyper > 1.10.1 contains commit 72046cc7.

Closes rustfs/backlog#1232.

…ected EOF

hyper <= 1.10.1 can call poll_shutdown() on an HTTP/1 socket while response
bytes are still buffered (a prior poll_flush() returned Poll::Pending and the
result was discarded), so a backpressured/slow-reading peer receives a graceful
FIN before the full Content-Length body is flushed. Standard S3 clients
(minio-go/warp) report this as sporadic `unexpected EOF` on large-object GET
under load (rustfs/backlog#1232). This is the transport-layer bug from
Cloudflare's "hyper-bug" writeup — distinct from the EC-reconstruct-desync EOF
already fixed via lockstep decode; here the app layer delivers the full body and
truncation is purely hyper->socket.

Fixed upstream in hyperium/hyper#4018 (commit 72046cc7, "fix(http1): flush
buffered data before shutdown"), which is not in any crates.io release yet as of
hyper 1.10.1. Pin hyper via [patch.crates-io] to git rev ccc1e850 (a descendant
of the fix that still declares version 1.10.1).

Use [patch.crates-io], not a git+rev on the workspace `hyper` dependency: the
server connection is driven by the transitive hyper-util (conn::auto /
GracefulShutdown), and a git source would not unify with the crates.io hyper
hyper-util resolves, leaving two hyper copies with the server path still on the
buggy one. The patch rewrites the crates.io source globally, so the lockfile
holds a single git-sourced hyper.

Add rustfs/tests/hyper_h1_shutdown_flush_regression.rs, a deterministic guard
(mirrors hyper's own h1_shutdown_while_buffered) that fails if the pin is dropped
or hyper is downgraded below the fix. Its load-bearing assertion is a
synchronously-set flag, so a slow runner can only under-detect, never false-red.

Closes rustfs/backlog#1232.
Copilot AI review requested due to automatic review settings July 13, 2026 14:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

Copy link
Copy Markdown
Contributor

CLA requirements are satisfied for this pull request.

@github-actions

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@overtrue
overtrue enabled auto-merge (squash) July 13, 2026 14:43
…-shutdown pin

cargo-deny's [sources] check denies unknown git sources. The hyper
[patch.crates-io] pin added for rustfs/backlog#1232 uses a github.com/hyperium
git source, so add it to allow-git with an owner/review note and a removal
condition (drop once a released hyper > 1.10.1 carries commit 72046cc7).
@overtrue
overtrue merged commit 0f83a27 into main Jul 13, 2026
38 of 39 checks passed
@overtrue
overtrue deleted the overtrue/jovial-bhabha-fbb3ac branch July 13, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants