Repository navigation
Commit a04c8e1
committed
[fix](lance) Stop rewriting namespace-vended storage options
The alias table this series grew was the wrong shape, and it broke every non-S3
backend. It canonicalized vended keys onto the AWS spellings without knowing
which provider a dataset uses, but the accepted spellings differ per provider:
// object_store-0.13.2 src/azure/builder.rs:448
"azure_storage_endpoint" | "azure_endpoint" | "endpoint" => Ok(Self::Endpoint)
// lance-io/src/object_store/providers/oss.rs:71-73, :87
("endpoint", &["oss_endpoint"]), ("access_key_id", &["oss_access_key_id"]), ...
"OSS endpoint is required..."
So a namespace describing `az://container/table` with `endpoint=...`, or
`oss://bucket/table` with `endpoint=...` and `access_key_id=...`, had exactly
the options it needs renamed into AWS keys those providers do not read: the
Azure endpoint was silently lost, and OSS failed outright. That is the opposite
of what this series set out to do, which was to stop dropping vended options.
The table could not be made correct without a provider, and it was tracking two
libraries at two versions each - object_store plus OpenDAL, 0.57 on the FE and
0.56 on the BE. Resolving equivalent spellings belongs to Lance, which knows the
provider; Doris only has to carry the options across the FE/BE boundary intact.
mergeVended is now an exact-key overlay: a vended entry replaces the catalog's
entry under the same key, and everything else arrives as the namespace wrote it.
Gone with the table are the ambiguous-`token` special case, the
`aws_endpoint_url_s3` precedence handling, the endpoint double-write, allow_http
retraction, and the protected-key filter - the last of which was never a
boundary anyway, since the namespace already decides the dataset URI.
The catalog's own static properties still map to the canonical `aws_*` names.
That path is S3-only by construction, so the spelling is provider-correct there,
and it is what `with_env_s3` checks before folding in the environment.
Options that cannot cross the boundary now fail instead of being dropped. A NUL
truncates a key at lance-c's `CStr::from_ptr` while the FE keeps reading the
whole thing, so the FE rejects it and the BE returns InvalidArgument; dropping
it silently only moved the divergence to an FE that predates the check. Empty
values are carried rather than second-guessed - a C string expresses them fine.
Verified end to end against the docker Lance REST fixture: both the prefixed and
the unprefixed credential-vending tables scan correctly with no static
credentials configured, which is the case that motivated the series - and it now
works with no alias normalization at all.
Claude-Session: https://claude.ai/code/session_01PfVSEUJsfWwKy7WEuBaFCG1 parent e850bff commit a04c8e1
4 files changed
Lines changed: 140 additions & 483 deletions
File tree
- be/src/format_v2/table
- fe/fe-core/src
- main/java/org/apache/doris/datasource/lance
- test/java/org/apache/doris/datasource/lance
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | | - | |
32 | 31 | | |
33 | 32 | | |
34 | 33 | | |
| |||
274 | 273 | | |
275 | 274 | | |
276 | 275 | | |
277 | | - | |
| 276 | + | |
| 277 | + | |
278 | 278 | | |
279 | 279 | | |
280 | 280 | | |
| |||
386 | 386 | | |
387 | 387 | | |
388 | 388 | | |
389 | | - | |
| 389 | + | |
| 390 | + | |
390 | 391 | | |
391 | 392 | | |
392 | 393 | | |
| |||
898 | 899 | | |
899 | 900 | | |
900 | 901 | | |
901 | | - | |
902 | | - | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
903 | 905 | | |
904 | | - | |
| 906 | + | |
905 | 907 | | |
906 | | - | |
907 | | - | |
| 908 | + | |
908 | 909 | | |
909 | | - | |
910 | | - | |
911 | | - | |
912 | | - | |
913 | | - | |
914 | | - | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
915 | 914 | | |
916 | | - | |
917 | | - | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
918 | 918 | | |
919 | | - | |
920 | | - | |
| 919 | + | |
| 920 | + | |
921 | 921 | | |
922 | | - | |
| 922 | + | |
923 | 923 | | |
924 | 924 | | |
925 | | - | |
| 925 | + | |
926 | 926 | | |
927 | | - | |
928 | | - | |
929 | | - | |
930 | | - | |
931 | | - | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
932 | 930 | | |
933 | 931 | | |
934 | 932 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
87 | | - | |
88 | | - | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
89 | 90 | | |
90 | 91 | | |
91 | 92 | | |
| |||
0 commit comments