fix(media): Range requests to /media return 500 when storage is Cloudflare R2

Describe the bug

GET /media/{sha256} with a Range header fails when media storage is Cloudflare R2. The relay advertises accept-ranges: bytes on the full response, so clients trust it and then get an error:

Range: bytes=0-1023 → 500 {"error":"internal error"}
Range: bytes=0-99 → 416

A request with no Range header returns 200 with the complete body, so the object and the key resolve correctly.

This breaks video playback entirely, since players use range requests to stream and seek. Images are unaffected because they are fetched whole.

Steps to reproduce

Configure the relay with BUZZ_S3_ENDPOINT pointing at a Cloudflare R2 bucket (https://<account>.r2.cloudflarestorage.com).
Upload a video through Buzz Desktop.
curl -r 0-1023 https://<relay>/media/<sha256> → 500.
curl https://<relay>/media/<sha256> (no Range) → 200, full body.

Expected behavior

206 Partial Content with Content-Range: bytes 0-1023/<total>, per the behaviour documented at crates/buzz-relay/src/api/media.rs (RFC 9110 §14.2).

Version and platform

Buzz version: relay ghcr.io/block/buzz:sha-63496cc
OS: Ubuntu 24.04, Docker Compose, media on Cloudflare R2

Logs / additional context

Isolated to the relay — R2 itself handles ranges correctly:

Check	Result
R2 ranged `GetObject` (aws-cli, same key/creds)	`ContentRange: bytes 0-1023/9896517`, 1024 bytes
R2 `HeadObject`	`ContentLength=9896517`, `AcceptRanges=bytes`
Sidecar `_meta/**.json`	`"size": 9896517` (correct)
Relay full `GET`	`200`, all 9896517 bytes
Relay ranged `GET`	`500`

Both relay paths resolve the same key via resolve_s3_key and both call head_with_metadata, so key resolution and total size are not the difference. The only divergence is in buzz-media::storage:

get_stream → bucket.get_object_stream(key) — works
get_range → bucket.get_object_range(key, start, Some(end)) — fails

A get_range failure surfaces as MediaError::StorageError, which matches the 500. This looks like a rust-s3 get_object_range incompatibility with R2 rather than a bug in the range parsing, which behaves correctly.

Relay log for the failing request:

level=ERROR message="response failed" classification="Status code: 500 Internal Server Error" latency="1206 ms" target=tower_http::trace::on_failure http.request.method=GET

Not reproduced against MinIO or AWS S3 — R2 is the only backend tested here, so this may be R2-specific S3 interop. Possibly adjacent to #2470 (S3-interop failure on GCS), though that is the git object store rather than media.

Picking this up. Clear repro + identified seam at crates/buzz-relay/src/api/media.rs. I'll check the Range parsing/satisfiability logic + the S3 client GetObject range forwarding, fail-first a test pinning the 500/416 paths, then ship the minimal fix. Intent: no behavior change on the happy path ( full body), only correct the ranged branch. Will comment with the PR once opened.

Diagnosis so far — need one log line to pin it

Thanks for the detailed report. I've traced this end-to-end on current main (d88313f36, identical relay/media code to your sha-63496cc deploy), and there's exactly one unresolved step.

What the relay actually does (crates/buzz-relay/src/api/media.rs serve_blob_for_tenant, range branch):

Sidecar gate → resolve key (your _meta/**.json read, size=9896517, is fine).
total = head_with_metadata(key) → HEAD on the object → reads Content-Length.
parse_byte_range("bytes=0-1023", total) → (0, 1023).
0 < total ⇒ not the 416 branch.
end is clamped to ≤ min(total-1, start+16MiB-1) ⇒ get_range(key, 0, 1023)
→ bucket.get_object_range sends Range: bytes=0-1023 (correctly SigV4-signed, range in SignedHeaders).
head_with_metadata's ? and get_range's ? are the only MediaError::StorageError producers in this path, and StorageError is the only variant that returns HTTP 500.

So on the code as written:

bytes=0-1023 can only 500 via step 6.
bytes=0-99 → 416 can only happen if parse_byte_range returned None, which for start=0, total=9896517 is impossible — unless total isn't actually 9896517 (i.e., the HEAD Content-Length isn't what the sidecar claims) or the 416 is being generated by something else.

The one thing I need: when you reproduce this, the relay logs a line:

ERROR ... "media storage error"  error=<the exact S3Error>

That error= string is the decisive evidence — it carries the HTTP status and R2's XML error body (e.g. HttpFailWithBody(416, "<Error>…InvalidRange…</Error>") vs a HEAD failure), which tells us whether the 500 is the GET or the HEAD, and whether R2 is rejecting the signed range or the HEAD is returning a wrong/zero Content-Length.

Could you paste that log line (with RUST_LOG=buzz_relay=debug if needed) for the bytes=0-1023 → 500 case, and confirm the response headers on the bytes=0-99 → 416 (specifically any Content-Range)?

I have a narrow fix staged (map HttpFailWithBody(416) → relay 416, and normalize the open-ended-range end before it hits R2), but I don't want to ship it blind against a guess about R2's wire behavior.

I've opened PR #4079 which I believe addresses this.

What's in the PR

Diagnostic fix first: the original report couldn't easily tell us why S3 range GETs were failing — rust-s3 wraps upstream non-2xx responses in S3Error::HttpFailWithBody(status, body), but buzz was collapsing that into a generic StorageError, producing the opaque 500 {"error":"internal error"} with no trace of the upstream status or body. That's why the request for logs in this issue didn't yield anything actionable. The PR now logs tracing::warn!(s3_status, s3_body, key, start, end) on that path, so any future R2/S3 range rejection is directly visible.
Fallback for the 500: when the S3 range GET fails for a reason other than 404, we now fall back to a full object GET and slice [start..(end+1).min(len)] in memory, returning the usual 206 + Content-Range (or 416 when out of bounds). This unwires the S3 range rejected → 500 fast-path entirely. Response headers, chunking, and the existing MAX_RANGE_CHUNK bound are unchanged on the happy path.
Tests: unit tests covering the slice math, clamping behaviour at MAX_RANGE_CHUNK, and a regression case matching your 9,896,517-byte object.

Given you verified the same range works against R2 via aws s3api directly, the most likely culprit was a non-range error (e.g. auth/expiry on that specific request, or R2 returning something non-2xx in an edge case) being silently discarded. With this PR it'll either (a) work transparently via the fallback and you'll see the reason in logs, or (b) surface the real upstream error. Happy to adjust the approach if maintainers prefer strict range-only fail-fast — the diagnostic logging is independently worth shipping either way.

The difference between the working and failing paths is two signed headers, and it sits in rust-s3 rather than in the range parsing.

Captured from rust-s3 0.37.2 (the version buzz-media pins) against a logging endpoint, same bucket, same key:

GET  plain    accept: application/octet-stream
              SignedHeaders=accept;host;x-amz-content-sha256;x-amz-date

GET  ranged   content-length: 0
              content-type: text/plain
              range: bytes=0-1023
              SignedHeaders=accept;content-length;content-type;host;range;x-amz-content-sha256;x-amz-date

A body-less GET is sent with content-length: 0 and content-type: text/plain, and both are signed. The seam is the header assembly in s3/src/request/request_trait.rs:

match self.command() {
    Command::CopyObject { from } => { … }
    Command::ListObjects { .. } => {}
    Command::ListObjectsV2 { .. } => {}
    Command::HeadObject => {}
    Command::GetObject => {}
    Command::GetObjectTagging => {}
    Command::GetBucketLocation => {}
    Command::ListBuckets => {}
    _ => {
        headers.insert(CONTENT_LENGTH, self.command().content_length()?.to_string().parse()?);
        headers.insert(CONTENT_TYPE, self.command().content_type().parse()?);
    }
}

Command::GetObject is exempt. Command::GetObjectRange is not, so it falls through to _ and picks up both headers, which command.rs supplies as 0 and "text/plain". That is why get_stream works and get_range does not, even though both resolve the same key and both call head_with_metadata first.

durch/rust-s3#466 reports the same shape for DeleteObject: AWS tolerates the extra signed headers, R2 enforces SigV4 strictly and answers 403 SignatureDoesNotMatch, so no object can ever be deleted there while PUT, GET and HEAD are fine. durch/rust-s3#470 asks for the general fix.

What I have not been able to check is the response R2 gives here, since I have no R2 bucket. A 403 from the range GET would reach serve_blob_for_tenant as MediaError::StorageError and surface as the 500 {"error":"internal error"} in this report, so the two are consistent, and the warn!(s3_status, s3_body, …) added in #4079 should print it directly. If it reads SignatureDoesNotMatch, the cause is the two headers above and the fix belongs upstream; the full-object fallback in that PR would keep working but would fetch a whole video to serve one range of it.

For what it is worth, Supabase Storage accepts the ranged GET with those headers signed (206, correct Content-Range), so the strictness is not universal among S3-compatible backends.

Separately, that same rust-s3 version cannot talk to Supabase Storage at all: Region::host keeps the endpoint path and host_header passes it to the Host header, so https://<project>.supabase.co/storage/v1/s3 produces host: <project>.supabase.co/storage/v1/s3 and a bare 400. Filed as durch/rust-s3#473 with a fix in durch/rust-s3#474, in case anyone here tries that backend.

fix(media): treat R2 403 SignatureDoesNotMatch as a range rejection

@eastriverlee

@eastriverlee — this is the answer, and it holds against the source this repo actually builds.

I checked your reading against the pinned fork rather than crates.io rust-s3 0.37.2, since buzz-media pins tlongwell-block/rust-s3 at c9fce362. Same shape there — s3/src/request/request_trait.rs, the header block after headers.insert(HOST, …):

Command::HeadObject => {}
Command::GetObject => {}
Command::GetObjectTagging => {}
Command::GetBucketLocation => {}
Command::ListBuckets => {}
_ => {
    headers.insert(CONTENT_LENGTH, self.command().content_length()?.to_string().parse()?);
    headers.insert(CONTENT_TYPE, self.command().content_type().parse()?);
}

Command::GetObjectRange is absent from the exempt arm, so it falls to _ and signs content-length: 0 and content-type: text/plain. Worth noting it is exempted in the other match in the same file (the one around the body/payload decision), which is probably why the omission survived — it reads as handled.

Your analysis exposed a real gap in #4079, which I have now fixed. That PR gated the full-download fallback on 400/416 only, on the reasoning that 401/403 are auth failures that would fail the full GET too. That reasoning is right in general and wrong here: R2 answers 403 SignatureDoesNotMatch for this, so the fallback never fired for the backend the issue was actually reported against. The PR logged the status and then returned the same opaque 500.

Fixed by gating that 403 on the error code in the body:

fn is_range_rejection(code: u16, body: &str) -> bool {
    match code {
        400 | 416 => true,
        403 => body.contains("SignatureDoesNotMatch"),
        _ => false,
    }
}

AccessDenied and InvalidAccessKeyId keep propagating as errors, so a genuine credential problem does not buy a full-object scan per request, and the existing 16 MiB object cap still bounds the fallback — which is the cost you flagged. Test covers both directions.

That said, I agree the fallback is the wrong place for the real fix. It is a correctness net for arbitrary S3-compatible backends, not a fix for this one: serving a range by fetching the whole object is the wrong shape even when capped. Since the pin is a block-owned fork, the one-line exemption of Command::GetObjectRange can land there directly without waiting on durch/rust-s3#470, and then this path stops being exercised on R2 at all. I have not opened that PR — it is a different repo and not mine to route.

I have no R2 bucket either, so the SignatureDoesNotMatch body on this exact call is inferred from your capture plus the header assembly above, not observed. If anyone here can hit a real R2 bucket, the warn!(s3_status, s3_body, key, start, end) in #4079 prints it directly and settles it.

Thanks for the durch/rust-s3#473/#474 note on Region::host as well — that one is not in play for this issue but is a real trap for anyone pointing buzz-media at Supabase Storage.

fix(media): treat R2 403 SignatureDoesNotMatch as a range rejection

Confirmed on a real Cloudflare R2 bucket (self-hosted relay, ghcr.io/block/buzz:main pulled 2026-10-03, storage = R2).

Relay, same blob (3,941,902-byte mp4):

GET /media/<sha>.mp4 (no Range) → 200
GET /media/<sha>.mp4 with Range: bytes=0-1023 → 500 {"error":"internal error"} (images too). On Android the in-app video player shows "failed to load" because it always sends Range.

Directly against the R2 bucket with boto3 (SigV4), same key:

ranged GET, normal signing → 206, Content-Range: bytes 0-1023/3941902
ranged GET with Content-Length: 0 and Content-Type: text/plain added before signing (what GetObjectRange sends per the analysis above) → 403 SignatureDoesNotMatch

So the SignatureDoesNotMatch inference holds on R2: exempting Command::GetObjectRange from the content-length/content-type arm in the pinned rust-s3 fork should fix it at the source, and #4079's 403 gate would catch it in the meantime.

Confirming this on another R2-backed relay (ghcr.io/block/buzz:sha-12670bd), with the storage error captured and a one-line fix verified.

Error from R2 (same bucket, key and credentials as the relay, rust-s3 =0.37.2, features tokio-rustls-tls, fail-on-err, tags, path style):

get_object(key)                    -> 200
get_object_range(key, 0, Some(99)) -> HTTP 403 SignatureDoesNotMatch

R2's canonical request in the error body has content-length: empty, while the client signed content-length:0. In 0.37.2 GetObjectRange falls into the default arm of the header block in request_trait.rs and gets Content-Length: 0 and Content-Type: text/plain, both signed; GetObject does not. R2 does not see a Content-Length value on a GET. With curl and SigV4, a ranged GET is 206, and adding a signed Content-Length: 0 to it makes it 403; Content-Type alone is fine.

Fix: rust-s3 0.38.0 (2026-10-04) no longer adds those two headers for GET, HEAD or DELETE. The same test program with only the dependency changed to =0.38.0 returns 206 for both get_object_range(key, 0, Some(99)) and get_object_range(key, 0, None) against the same R2 object. So bumping crates/buzz-media/Cargo.toml from 0.37 to 0.38 should close this without a fallback path. I have not built the workspace with 0.38, so the aws-creds fork pin in the root Cargo.toml may need a look.

Why nobody could find it in the logs: the error is logged by buzz_media ("media storage error"), and the default log filter is buzz_relay=info, so the line is dropped unless RUST_LOG also names buzz_media. Either widening the default filter or logging the S3 status in buzz_relay's range branch would have shown the 403.

Garage accepts the extra signed header (the same relay image on Garage answers 206), and presumably MinIO does too, which would be why this passes in tests and fails only on R2.

I have opened #8203 for this. It bumps rust-s3 from 0.37 to 0.38, which stops signing content-length/content-type on ranged GETs (the cause described in my earlier comment), and removes the aws-creds fork pin, since durch/rust-s3#449 is now released in aws-creds 0.40.0.

Verified: a new test that records the request on the wire fails on 0.37.2 and passes on 0.38.0; the media e2e suites, including the 206 and 416 range tests, pass against a local relay; workspace build, clippy and fmt pass. The pull request also handles two other 0.38 changes: whole-blob transfers keep no request deadline (0.38 would cut off downloads that take longer than 60 seconds), and media put/delete retry once on transient errors, as 0.37 did.

Not verified, and where I would appreciate a maintainer's help: the workspace build itself was not run against R2, and the EKS Pod Identity path needs checking on your cluster now that the fork pin is gone. The description lists what matches and what differs between the fork and aws-creds 0.40.0.

If you have time to review it, I would be grateful. Thank you.

Navigation Menu

Repository navigation

fix(media): Range requests to /media return 500 when storage is Cloudflare R2 #3786

Diagnosis so far — need one log line to pin it

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

fix(media): Range requests to /media return 500 when storage is Cloudflare R2 #3786

Description

Activity

iroiro147 commented on Jul 31, 2026

iroiro147 commented on Jul 31, 2026

Diagnosis so far — need one log line to pin it

iroiro147 commented on Aug 1, 2026

eastriverlee commented on Aug 14, 2026

iroiro147 commented on Aug 17, 2026

muphy commented on Oct 5, 2026

justbry commented on Oct 8, 2026

justbry commented on Oct 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions