Repository navigation
SeaweedFS S3 endpoint (TLS) drops large streaming GET responses, breaking CDI http-import of large images #3782
Description
What happened
When a VMDisk (or any CDI DataVolume with source.http) points at a large object served from a provisioned Bucket's S3 endpoint, the CDI importer fails partway through the download:
Unable to write file from dataReader: stream error: stream ID 3; INTERNAL_ERROR; received from peer
failed to pull image: ... Unable to transfer source data to scratch space
The importer retries and CrashLoops; the import never completes.
Root cause
The SeaweedFS S3 gateway (port 8333, TLS) aborts large streaming GET responses after ~25-40 MB (non-deterministic). Reproduced directly against the ClusterIP from inside the cluster (no ingress involved), same client, comparing layers:
| Source | Full streaming GET of an ~840 MB object |
|---|---|
seaweedfs-filer:8888 (plain HTTP, file API) |
full object, stable |
seaweedfs-s3:8333 (HTTPS, S3 API) |
aborts at ~25-40 MB, OpenSSL SSL_read: unexpected eof while reading |
seaweedfs-s3:8333 over plain HTTP |
HTTP 400 (endpoint is TLS-only) |
Notes:
- Range requests against the S3 endpoint work fine (a 50 MB
RangeGET returns 206 correctly) — only a long single streaming response breaks, so the stored object is intact. - The Bucket
BucketInfoadvertises anhttps://S3 endpoint, so consumers hit the failing TLS path by default (possibly related: fix(bucket): endpoint helper unconditionally forces https:// — original scheme lost for provisioned buckets #3317).
Impact
Any CDI http-import of a large image (VM disk images, etc.) from a Bucket's S3 URL fails. Small objects (< ~25 MB) import fine, which masks the problem until real images are used.
Environment
- Cozystack v1.6.1-rc.1
- seaweedfs chart 4.31.0 (
weed version 4.31) - CDI v1.64.0
Steps to reproduce
- Create a
Bucket; upload a large (~800 MB) object. - From any in-cluster pod:
curl -k https://seaweedfs-s3.<ns>.svc:8333/<bucket>/<object> -o /dev/null-> aborts with SSL unexpected EOF after tens of MB. - Same object via filer:
curl http://seaweedfs-filer.<ns>.svc:8888/buckets/<bucket>/<object> -o /dev/null-> completes.
Workaround
Serve large objects through the filer HTTP API (8888) instead of the S3 endpoint (8333). Range-based clients are unaffected.
Activity
scooby87 commentedon Oct 8, 2026
Root cause found, and it is not TLS.
The S3 gateway is OOM-killed in the middle of the response. Its pod runs with a 512Mi memory limit (the
t1.smallpreset inpackages/extra/seaweedfs), and serving one large object makes it buffer far more than that:ReaderCache.MaybeCachecounts only the downloads it newly starts toward that window, andChunkReadAt.ReadAtcalls it after every read. The S3 handler reads 256 KiB at a time out of 4 MiB chunks, so each read pushes the window four chunks further, until the ReaderCache limit of 256 downloaders. One full GET of an 840 MiB object left about 830 MiB of chunk buffers live in the gateway's heap.unexpected eofover HTTP/1.1 orINTERNAL_ERRORover HTTP/2. The read-ahead cannot run past the end of the requested bytes, so small objects and moderate Range requests stay under the limit, which is why they work. The filer endpoint streams through a different code path, which is why it works too. TLS has nothing to do with it.Reproduced on a test cluster with an isolated SeaweedFS and the production S3 limits (512Mi, 1 CPU), an 840 MiB object uploaded through S3 multipart:
GOMEMLIMIT=400MiB-readerCacheSizeMB=256Upgrading SeaweedFS does not fix it: the defect is on upstream master as of today.
MaybeCacheso the window is the next N chunks whether or not they are already downloading, plus a regression test, is proposed in filer: keep sequential prefetch within prefetchCount chunks seaweedfs/seaweedfs#11674.Until that lands, the workaround from the description still holds: fetch large objects through the filer HTTP API, or use Range requests.
fix(seaweedfs): build SeaweedFS in-tree with a bounded S3 read-ahead (#…
scooby87 commentedon Oct 14, 2026
Fixed by #4819 (in-tree SeaweedFS build carrying the upstream read-ahead fix, seaweedfs/seaweedfs#11674). Ships in the next release after v1.7.0-rc.1.
[Backport release-1.7] fix(seaweedfs): build SeaweedFS in-tree with a…