Backporting #10031

Description from #10031

The pusher supports fallback from HTTPS to HTTP using HTTPFallback. But the current implementation of HTTPFallback doesn't reset the request body on fallback so the body is used twice. When the pusher uploads the manifest using HTTPFallback, it results in io: read/write on closed pipe error because the request body is a *io.PipeReader that can't be used twice. This commit fixes this issue by modifying HTTPFallback to reset the request body on fallback, using http.Request.GetBody().

Reusing the request body didn't happen when uploading the layer because the pusher correctly selects the scheme based on Location header and avoids fallback. This commit also fixes the test to cover the code path of uploading the manifest with fallback.

Additional information

• The same error is observed also in v1.7.14 so I'll backport this patch to that branch after this is merged to the main branch.
• The error starts to happen after 2a25c08 according to bisect. *io.PipeReader is closed before fallback so reusing it results in the explicit error.
• This will also fix that ctr + --plain-http failed with io: read/write on closed pipe error (>= v1.7.14)
  • https://github.com/containerd/stargz-snapshotter/actions/runs/8436798962/job/23199432922?pr=1605#step:4:4116 : failed to copy: failed to do request: Put "https://registry-alt.test:5000/v2/alpine/manifests/sha256:01344c7e6bf6a4b063eee2c3c83f39ef515e6dea8af546a7a1fc786f3d1a82c0": readfrom tcp 172.18.0.2:40046->172.18.0.4:5000: io: read/write on closed pipe