Port Add stream upload (multi-part upload) #2147
Conversation
* feat: Implement multi-part upload Co-authored-by: Raphael Taylor-Davies <[email protected]> * chore: simplify local file implementation * chore: Remove pin-project * feat: make cleanup_upload() top-level * docs: Add some docs for upload * chore: fix linting issue * fix: rename to put_multipart * feat: Implement multi-part upload for GCP * fix: Get GCS test to pass * chore: remove more upload language * fix: Add guard to test so we don't run with fake gcs server * chore: small tweaks * fix: apply suggestions from code review Co-authored-by: Raphael Taylor-Davies <[email protected]> * feat: switch to quick-xml * feat: remove throttle implementation of multipart * fix: rename from cleanup to abort * feat: enforce upload not readable until shutdown * fix: ensure we close files before moving them * chore: fix lint issue Co-authored-by: Raphael Taylor-Davies <[email protected]> Co-authored-by: Raphael Taylor-Davies <[email protected]>
alamb
force-pushed
the
alamb/port-ulti-part-upload
branch
from
8d9aff7 to
5875b13
Compare
Codecov Report
@@ Coverage Diff @@
## master #2147 +/- ##
==========================================
- Coverage 82.86% 82.42% -0.44%
==========================================
Files 237 238 +1
Lines 61429 61963 +534
==========================================
+ Hits 50902 51074 +172
- Misses 10527 10889 +362
Help us with your feedback. Take ten seconds to tell us how you rate us. |
|
I am hopeful once the action checks run on this repo this PR will be ready for review |
|
@wjones127 and @tustvold this is now ready for review |
wjones127
left a comment
wjones127
left a comment
There was a problem hiding this comment.
Thank you for moving this over!
|
Benchmark runs are scheduled for baseline = aeb2776 and contender = 19fd885. 19fd885 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
* feat: Add stream upload (multi-part upload) (#20) * feat: Implement multi-part upload Co-authored-by: Raphael Taylor-Davies <[email protected]> * chore: simplify local file implementation * chore: Remove pin-project * feat: make cleanup_upload() top-level * docs: Add some docs for upload * chore: fix linting issue * fix: rename to put_multipart * feat: Implement multi-part upload for GCP * fix: Get GCS test to pass * chore: remove more upload language * fix: Add guard to test so we don't run with fake gcs server * chore: small tweaks * fix: apply suggestions from code review Co-authored-by: Raphael Taylor-Davies <[email protected]> * feat: switch to quick-xml * feat: remove throttle implementation of multipart * fix: rename from cleanup to abort * feat: enforce upload not readable until shutdown * fix: ensure we close files before moving them * chore: fix lint issue Co-authored-by: Raphael Taylor-Davies <[email protected]> Co-authored-by: Raphael Taylor-Davies <[email protected]> * fmt * RAT multipart * Fix build * fix: merge issue Co-authored-by: Will Jones <[email protected]> Co-authored-by: Raphael Taylor-Davies <[email protected]> Co-authored-by: Raphael Taylor-Davies <[email protected]>
4 participants
Draft until I have ported over the integration tests for the object_store crate: #2148
Which issue does this PR close?
Re #2030
Rationale for this change
Multipart upload support was added to the object store in influxdata/object_store_rs#20
by @wjones127 and @tustvold after the snapshot that was donated to the ASF in #2081.
What changes are included in this PR?
Cherry-pick code from influxdata/object_store_rs#20 to this repo
Specifically, what I did was
Are there any user-facing changes?
mutli-part uploads!