object-store: remove S3ConditionalPut::ETagPutIfNotExists#6802
object-store: remove S3ConditionalPut::ETagPutIfNotExists#6802tustvold merged 4 commits intoapache:mainfrom
Conversation
As of today [0] S3 now supports the If-Match for in-place conditional
writes. This commit adjusts the existing support for
S3ConditionalPut::Etag mode for compatibility with real S3's particular
semantics, which vary slightly from MinIO and R2. Specifically:
* Real S3 can occasionally return 409 Conflict when concurrent
If-Match requests are in progress. These requests need to be
retried.
* Real S3 returns 404 Not Found instead of 412 Precondition Failed
when issuing an If-Match request against an object that does not
exist.
Fix #6799.
[0]: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/
benesch
force-pushed
the
object-store-s3-put-if-match-cleanup
branch
from
977cd2f to
6a69ad1
Compare
Now that real S3 supports `If-Match`, we no longer need this special conditional put mode for real S3.
benesch
force-pushed
the
object-store-s3-put-if-match-cleanup
branch
from
6a69ad1 to
ac2b05c
Compare
|
Quick update on the LocalStack side: I've got the green light to release this feature during this week, and will work on it either tonight or tomorrow. The feature will be available in the |
|
That's amazing, thank you for working on this! |
Quite the opposite—thanks for turning the implementation around so quickly! |
|
As written in apache/arrow-rs-object-store#33, the feature is now available in the I've given a try to this branch with the flag enabled and it worked for me locally, might be worth trying it out. Thanks a lot for the tests, they're always great to run locally! 🙏 |
|
Yes, just bumped to the Give me a holler when 4.0.3 is released—I'll try to keep an eye on it myself—so that I can update this PR to use it. And then we should be ready to merge. |
|
Hello @benesch! The Can't wait to see this feature merged! 🚀 thanks a lot for the collaboration on this. Have a great week-end 😄 |
* Support real S3's If-Match semantics
As of today [0] S3 now supports the If-Match for in-place conditional
writes. This commit adjusts the existing support for
S3ConditionalPut::Etag mode for compatibility with real S3's particular
semantics, which vary slightly from MinIO and R2. Specifically:
* Real S3 can occasionally return 409 Conflict when concurrent
If-Match requests are in progress. These requests need to be
retried.
* Real S3 returns 404 Not Found instead of 412 Precondition Failed
when issuing an If-Match request against an object that does not
exist.
Fix #6799.
[0]: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/
* object-store: remove S3ConditionalPut::ETagPutIfNotExists
Now that real S3 supports `If-Match`, we no longer need this special
conditional put mode for real S3.
* [XXX put in real release version] Upgrade localstack
* Update .github/workflows/object_store.yml
---------
Co-authored-by: Raphael Taylor-Davies <[email protected]>
3 participants
Now that real S3 supports
If-Match, we no longer need this special conditional put mode for real S3.Considerations:
If-Match(which doesn't yet exist). @bentsku mentioned he'd try to add support to localstack later this week. See Support S3 Put IfMatch arrow-rs-object-store#33.Which issue does this PR close?
Cleanup related to apache/arrow-rs-object-store#33.
Are there any user-facing changes?
The removal of
S3ConditionalPut::ETagPutIfNotExistswould be user facing, but this variant has not yet made it into a release.cc @tustvold