Repository navigation
S3 virtual-style addressing for non AWS endpoints #722
Description
Any consideration to allowing DNS style bucked addresses for non s3 endpoints?
Expected Behavior
Expect virtual style addressing to work for non-s3 endpoints as well.
Current Behavior
Currently DNS style addressing is only enabled if the endpoint starts with s3. (
There are s3 compatible providers that support only DNS style addressing, which makes this SDK harder to use with them. (https://www.alibabacloud.com/help/doc-detail/64919.htm)
Possible Solution
Remove the restriction on DNS style addresses only working for the AWS s3 endpoints.
Context
Trying to use this client on Alicloud OSS.
Your Environment
<N/A>
Reacted by Zakhar Petrov
Activity
ebabani commentedon Sep 30, 2018
The change is fairly straight forward to make and I can send a PR, but just want to know what the direction is regarding this. At the moment if feels like for s3 compatible providers this sdk leans towards forcing the path style.
millems commentedon Oct 1, 2018
It's tough for us to test to make sure we're compatible with all of them, but we're definitely not against it. We'd gladly accept a PR that fixes the issue.
[-]S3: Support virtual style addressing for non AWS endpoints?[/-][+]Support S3 virtual-style addressing for non AWS endpoints[/+][-]Support S3 virtual-style addressing for non AWS endpoints[/-][+]S3 virtual-style addressing for non AWS endpoints[/+]Merge pull request #722 from aws/staging/0a806d85-a4bb-4690-8b9c-01b0…
blackstorm commentedon May 27, 2020
i decide back to sdk v1. Sad!
yannrouillard commentedon Oct 16, 2020
Currently experiencing this issue as well.
@millems I saw you were not keen to merge the PR #763, is there some other way to make it work that you would accept like creating a new option to force virtual-style addressing?
millems commentedon Oct 16, 2020
It's going to be difficult to figure out how to support this, with new features like S3 access points adding even more complexity to the URL building process, and the S3 specification being implemented differently by different providers.
Is there a specific non-AWS runtime we can focus in on supporting, so that we don't just have it be "all non-AWS S3 implementations"?
yannrouillard commentedon Oct 20, 2020
Hi @millems,
I am not sure I am legimitate to say which non-AWS runtime should be better to focus on.
You could target Google / Alicloud implementations to facilite migrations from them to AWS but I would find weird that you have to follow Google / Alicloud modifications to stay compatible (and pay some complexities price on the way).
(In our case it is with Alicloud implementation that we have an issue with using this library)
Supporting an alternate implementation like Minio sounds better as it is an open-source project and can be used by people to use S3-like interfaces on-premise in hybrid configuration (or for testing purpose).
In the end anyway, I don't see how you can maintain compatibility with existing behaviour without introducing a new option or a new configuration.
I saw you were refactoring to separate between
accessPointEndpointResolverandbucketEndointResolver, maybe you could allow to set a custom resolver so that library users have always a way to customize / override behavior like this without incurring too high maintainance on your side.In all cases, it would be nice to document this behaviour (_virtualAccesStyle can't be enabled for endpoints that do not begin by
s3) in the official documentation to avoid the surprise effect. Other SDK don't have this behaviour AFAIK so it took us some time to find the cause.yasminetalby commentedon Nov 14, 2022
Hello all,
This feature-request has been implemented as part of the 2.18.x version release.
As per the release Changelog, the new behavior is as follow:
When including an endpoint override, the SDK will use virtual bucket addressing as long as the bucket is a valid DNS label. This is different from the previous behavior where this only happens if the endpoint override begins with s3 : see changelog
As a side-note, if your bucket is not a valid DNS label , the behavior would be to not use virtual bucket addressing.
Thank you very much for your feature-request submission and feedback!
Best,
Yasmine
github-actions commentedon Nov 14, 2022
Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.