Repository navigation
Commit 88f5053
committed
[fix](lance) Resolve vended option spellings per storage provider
Passing vended options through untouched fixed the non-S3 breakage, but left a
case it should not have: a catalog with static `s3.access_key` whose namespace
also vends credentials, spelled the way real servers spell them.
aws_access_key_id = <catalog> # from Doris storage properties
access_key_id = <vended> # from the namespace, verbatim
Both parse to `AmazonS3ConfigKey::AccessKeyId`, and `as_s3_options()` collects
them into a HashMap, so the winner is whichever the iteration yields last -
decided separately for the access key and the secret, and separately in the FE
and in the BE. Two of the four combinations mix one side's key with the other's
secret, which is a SignatureDoesNotMatch rather than an obvious failure.
Renaming vended keys onto the S3 spellings is what broke Azure and OSS, so the
rename now happens only where the provider is known to accept it. Lance routes a
dataset to a provider by URL scheme; LanceStorageProvider does the same, and
each implementation owns both directions of its own vocabulary: Doris properties
in, and vended spellings normalized onto what it emits.
- LanceS3StorageProvider, for the `s3` and `s3+ddb` schemes lance-io registers
for its AWS provider. It resolves only the options Doris itself contributes,
since those are the only ones a vended option can collide with. `token`
resolves here too - it is a bearer token to object_store's Azure parser, and
only knowing the provider makes it decidable.
- LancePassThroughStorageProvider, for everything else. Both halves inert:
Doris cannot spell static credentials for those, and their vended options
are the namespace's to name.
`aws_endpoint_url_s3` is deliberately left alone. object_store parses it into a
config key of its own and prefers it over the generic endpoint, so a vended one
already wins; folding it in would replace a defined precedence with map order,
which is the mistake an earlier commit in this series made.
A namespace that vends two spellings of one option with different values is now
rejected rather than resolved by iteration order.
Verified end to end against the docker Lance REST fixture: the prefixed and
unprefixed credential-vending tables both scan with no static credentials, and
the FE build covers the call site, which now passes the dataset URL down.
Claude-Session: https://claude.ai/code/session_01PfVSEUJsfWwKy7WEuBaFCG1 parent a04c8e1 commit 88f5053
6 files changed
Lines changed: 430 additions & 119 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris/datasource/lance
- test/java/org/apache/doris/datasource/lance
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
284 | 284 | | |
285 | 285 | | |
286 | 286 | | |
287 | | - | |
288 | | - | |
289 | | - | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
290 | 291 | | |
291 | 292 | | |
292 | 293 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
41 | 33 | | |
42 | 34 | | |
43 | 35 | | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | 36 | | |
64 | 37 | | |
65 | 38 | | |
66 | | - | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
67 | 45 | | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
| 46 | + | |
85 | 47 | | |
86 | 48 | | |
87 | 49 | | |
88 | | - | |
| 50 | + | |
| 51 | + | |
89 | 52 | | |
90 | | - | |
91 | | - | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
92 | 57 | | |
93 | | - | |
94 | | - | |
95 | | - | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
96 | 62 | | |
97 | 63 | | |
98 | 64 | | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
| 65 | + | |
| 66 | + | |
103 | 67 | | |
104 | 68 | | |
105 | 69 | | |
| |||
122 | 86 | | |
123 | 87 | | |
124 | 88 | | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
130 | | - | |
131 | 89 | | |
0 commit comments