Skip to content
This repository was archived by the owner on Feb 13, 2026. It is now read-only.
This repository was archived by the owner on Feb 13, 2026. It is now read-only.

Don't concatenate multi-part upload during completion #2334

@ramondeklein

Description

@ramondeklein

The FS version concatenates all the parts of a multi-part upload when the multi-upload is completed. Multi-part uploads are often used for huge files, so this takes a lot of time and also spikes the IOPS for a while. In some situations (like ours), it would be much better to use the XL method to keep the parts separate and combine them during download.

Of course, the parts cannot be accessed the regular way (via the filesystem) anymore. But most systems that use S3 only use S3 to access them (you can't do otherwise in AWS), so this wouldn't be a big problem.

Concatenating files might be the sensible default option, but when performance and I/O is critical, then keeping the files separate might be the better option. Now, we reverted using the XL method, solely for this feature, but this costs us a lot of additional disk space.

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions