Skip to content

refactor: change uploadAllParts to limited concurrency#1442

Merged
prakashsvmx merged 2 commits intominio:masterfrom
inway:sequential-parts-upload
Jan 23, 2026
Merged

refactor: change uploadAllParts to limited concurrency#1442
prakashsvmx merged 2 commits intominio:masterfrom
inway:sequential-parts-upload

Conversation

@SPodjasek
Copy link
Copy Markdown
Contributor

@SPodjasek SPodjasek commented Jan 11, 2026

This pull request refactors the logic for uploading parts in the composeObject method to use sequential processing of uploads.

Changes

  • Changed the uploadAllParts function to upload parts in chunks using a for...of loop and await, instead of running all uploads in parallel with Promise.all.
  • This change limits simultaneous number of requests, which may help with rate-limiting.

@prakashsvmx
Copy link
Copy Markdown
Member

Using multipart upload provides the following advantages:

Improved throughput – You can upload parts in parallel to improve throughput.

Quick recovery from any network issues – Smaller part size minimizes the impact of restarting a failed upload due to a network error.

Pause and resume object uploads – You can upload object parts over time. After you initiate a multipart upload, there is no expiry; you must explicitly complete or stop the multipart upload.

Begin an upload before you know the final object size – You can upload an object as you create it.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

So please revisit your requirement/chnages.

@SPodjasek
Copy link
Copy Markdown
Contributor Author

Ok, but at the same time I personally consider 1k+ req/s a bit of overkill.

Based on what others do (ex. boto3 has default concurrency setting of 10)

https://github.com/boto/boto3/blob/93978170acb80cf7b6ff85c3a9b657e2022bb9fb/boto3/s3/transfer.py#L260-L261

I've implemented similar limit here...

@SPodjasek SPodjasek changed the title refactor: change uploadAllParts to use for-of loop refactor: change uploadAllParts to limited concurrency Jan 12, 2026
@SPodjasek
Copy link
Copy Markdown
Contributor Author

@prakashsvmx I've just tried to run this 1k parts compose test against play.min.io using this little test inway/minio-compose-object with BUCKET_NAME set to minio-compose-object (I've already populated this bucket with parts) so feel free to test it on your own.

After setting minioClient.traceOn(); on Minio client you can see that it hits 429 status pretty fast...

@prakashsvmx prakashsvmx merged commit ac1b5a9 into minio:master Jan 23, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Current implementation of composeObject can lead to DoS

2 participants