Skip to content

Commit 88f5053

Browse files
committed
[fix](lance) Resolve vended option spellings per storage provider
Passing vended options through untouched fixed the non-S3 breakage, but left a case it should not have: a catalog with static `s3.access_key` whose namespace also vends credentials, spelled the way real servers spell them. aws_access_key_id = <catalog> # from Doris storage properties access_key_id = <vended> # from the namespace, verbatim Both parse to `AmazonS3ConfigKey::AccessKeyId`, and `as_s3_options()` collects them into a HashMap, so the winner is whichever the iteration yields last - decided separately for the access key and the secret, and separately in the FE and in the BE. Two of the four combinations mix one side's key with the other's secret, which is a SignatureDoesNotMatch rather than an obvious failure. Renaming vended keys onto the S3 spellings is what broke Azure and OSS, so the rename now happens only where the provider is known to accept it. Lance routes a dataset to a provider by URL scheme; LanceStorageProvider does the same, and each implementation owns both directions of its own vocabulary: Doris properties in, and vended spellings normalized onto what it emits. - LanceS3StorageProvider, for the `s3` and `s3+ddb` schemes lance-io registers for its AWS provider. It resolves only the options Doris itself contributes, since those are the only ones a vended option can collide with. `token` resolves here too - it is a bearer token to object_store's Azure parser, and only knowing the provider makes it decidable. - LancePassThroughStorageProvider, for everything else. Both halves inert: Doris cannot spell static credentials for those, and their vended options are the namespace's to name. `aws_endpoint_url_s3` is deliberately left alone. object_store parses it into a config key of its own and prefers it over the generic endpoint, so a vended one already wins; folding it in would replace a defined precedence with map order, which is the mistake an earlier commit in this series made. A namespace that vends two spellings of one option with different values is now rejected rather than resolved by iteration order. Verified end to end against the docker Lance REST fixture: the prefixed and unprefixed credential-vending tables both scan with no static credentials, and the FE build covers the call site, which now passes the dataset URL down. Claude-Session: https://claude.ai/code/session_01PfVSEUJsfWwKy7WEuBaFCG
1 parent a04c8e1 commit 88f5053

6 files changed

Lines changed: 430 additions & 119 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalCatalog.java‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -284,9 +284,10 @@ public LanceTableMetadata loadTableMetadata(String dbName, String tableName,
284284
}
285285

286286
// One option map serves both readers: the FE opens the dataset through the Lance Java SDK
287-
// and the BE through lance-c, so neither can end up with credentials the other lacks.
288-
Map<String, String> storageOptions = LanceStorageOptions.mergeVended(
289-
lanceStorageOptions, table.getStorageOptions());
287+
// and the BE through lance-c, so neither can end up with credentials the other lacks. The
288+
// dataset URL picks the option vocabulary, the same way Lance picks a provider from it.
289+
Map<String, String> storageOptions = LanceStorageOptions.forDataset(datasetUri,
290+
catalogProperty.getBackendStorageProperties(), table.getStorageOptions());
290291
try {
291292
if (tableSnapshot.isPresent()) {
292293
TableSnapshot snapshot = tableSnapshot.get();
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.datasource.lance;
19+
20+
import java.util.Collections;
21+
import java.util.HashMap;
22+
import java.util.Map;
23+
24+
/**
25+
* Every provider Doris has no vocabulary of its own for - Azure, GCS, OSS, Tencent COS, local
26+
* files, and anything Lance adds later.
27+
*
28+
* <p>Both halves are deliberately inert. Doris cannot spell static credentials for these, so such
29+
* a dataset is reachable only through what its namespace vends, and those options are the
30+
* namespace's to name: Lance's OSS provider reads {@code access_key_id} and requires
31+
* {@code endpoint}, object_store's Azure parser reads {@code endpoint} and takes {@code token} as
32+
* a bearer token. Rewriting any of that onto the S3 spellings would leave the dataset unreachable,
33+
* which is exactly what this class exists to prevent.
34+
*/
35+
final class LancePassThroughStorageProvider implements LanceStorageProvider {
36+
37+
static final LancePassThroughStorageProvider INSTANCE = new LancePassThroughStorageProvider();
38+
39+
private LancePassThroughStorageProvider() {
40+
}
41+
42+
@Override
43+
public Map<String, String> fromDorisProperties(Map<String, String> backendProperties) {
44+
return Collections.emptyMap();
45+
}
46+
47+
@Override
48+
public Map<String, String> normalizeVended(Map<String, String> vendedOptions) {
49+
return vendedOptions == null ? new HashMap<>() : new HashMap<>(vendedOptions);
50+
}
51+
}
Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.datasource.lance;
19+
20+
import com.google.common.collect.ImmutableMap;
21+
22+
import java.util.HashMap;
23+
import java.util.Locale;
24+
import java.util.Map;
25+
26+
/**
27+
* S3-compatible storage, which Lance reaches through object_store's AWS provider.
28+
*
29+
* <p>The emitted spelling is the one object_store reports as canonical, because that is what
30+
* {@code StorageOptions::with_env_s3} looks for before pulling the same option out of the process
31+
* environment:
32+
*
33+
* <pre>
34+
* // lance-io/src/object_store/providers/aws.rs
35+
* if let Ok(config_key) = AmazonS3ConfigKey::from_str(&amp;key.to_ascii_lowercase())
36+
* &amp;&amp; !self.0.contains_key(config_key.as_ref()) // "aws_access_key_id"
37+
* </pre>
38+
*
39+
* <p>Any other accepted alias leaves that check unsatisfied, so a stray {@code AWS_ACCESS_KEY_ID}
40+
* in the FE or BE environment is inserted next to the configured value, and object_store's
41+
* {@code as_s3_options()} then folds both onto one config key and keeps whichever its HashMap
42+
* yields last - independently per option and per process. Lance's OpenDAL S3 backend accepts these
43+
* same spellings as serde aliases of its own field names.
44+
*/
45+
final class LanceS3StorageProvider implements LanceStorageProvider {
46+
47+
static final LanceS3StorageProvider INSTANCE = new LanceS3StorageProvider();
48+
49+
private static final String ENDPOINT = "aws_endpoint";
50+
private static final String VIRTUAL_HOSTED_STYLE = "aws_virtual_hosted_style_request";
51+
/**
52+
* Not prefixed: object_store carries this as a shared client option, and reports it as
53+
* canonical under this name. The spelling buys nothing against the environment here, though -
54+
* {@code StorageOptions::new} overwrites this key outright from {@code AWS_ALLOW_HTTP} before
55+
* {@code with_env_s3} runs.
56+
*/
57+
private static final String ALLOW_HTTP = "allow_http";
58+
59+
/** Doris backend property to Lance object-store option. */
60+
private static final Map<String, String> DORIS_KEYS = ImmutableMap.<String, String>builder()
61+
.put("AWS_ACCESS_KEY", "aws_access_key_id")
62+
.put("AWS_SECRET_KEY", "aws_secret_access_key")
63+
.put("AWS_TOKEN", "aws_session_token")
64+
.put("AWS_ENDPOINT", ENDPOINT)
65+
.put("AWS_REGION", "aws_region")
66+
.build();
67+
68+
/**
69+
* Every spelling object_store accepts for the options above, mapped onto the one emitted.
70+
*
71+
* <p>Confined to those options on purpose. They are the only ones this class contributes, so
72+
* they are the only ones a vended option can collide with; anything else a namespace sends is
73+
* between the namespace and Lance.
74+
*
75+
* <p>{@code token} is included because this provider only ever speaks for an S3 dataset, where
76+
* it is unambiguously the session token. It means a bearer token to object_store's Azure
77+
* parser, which is why it can only be resolved once the provider is known.
78+
*
79+
* <p>{@code aws_endpoint_url_s3} is deliberately absent. object_store parses it into a config
80+
* key of its own and prefers it over the generic endpoint, so a vended one already wins
81+
* without being rewritten, and folding it in would replace a defined precedence with map order.
82+
*/
83+
private static final Map<String, String> CANONICAL_BY_ALIAS = ImmutableMap.<String, String>builder()
84+
.put("access_key_id", "aws_access_key_id")
85+
.put("aws_access_key_id", "aws_access_key_id")
86+
.put("secret_access_key", "aws_secret_access_key")
87+
.put("aws_secret_access_key", "aws_secret_access_key")
88+
.put("session_token", "aws_session_token")
89+
.put("aws_session_token", "aws_session_token")
90+
.put("aws_token", "aws_session_token")
91+
.put("token", "aws_session_token")
92+
.put("endpoint", ENDPOINT)
93+
.put("endpoint_url", ENDPOINT)
94+
.put("aws_endpoint", ENDPOINT)
95+
.put("aws_endpoint_url", ENDPOINT)
96+
.put("region", "aws_region")
97+
.put("aws_region", "aws_region")
98+
.put("virtual_hosted_style_request", VIRTUAL_HOSTED_STYLE)
99+
.put("aws_virtual_hosted_style_request", VIRTUAL_HOSTED_STYLE)
100+
// OpenDAL's own field name, of which the two above are serde aliases. All three would
101+
// be the same field supplied more than once, which fails the operator build outright.
102+
.put("enable_virtual_host_style", VIRTUAL_HOSTED_STYLE)
103+
.put("allow_http", ALLOW_HTTP)
104+
.put("aws_allow_http", ALLOW_HTTP)
105+
.build();
106+
107+
private LanceS3StorageProvider() {
108+
}
109+
110+
@Override
111+
public Map<String, String> fromDorisProperties(Map<String, String> backendProperties) {
112+
Map<String, String> result = new HashMap<>();
113+
if (backendProperties == null) {
114+
return result;
115+
}
116+
DORIS_KEYS.forEach((dorisKey, lanceKey) -> putIfNotEmpty(result, lanceKey,
117+
backendProperties.get(dorisKey)));
118+
119+
String usePathStyle = backendProperties.get("use_path_style");
120+
if (usePathStyle != null && !usePathStyle.isEmpty()) {
121+
result.put(VIRTUAL_HOSTED_STYLE, String.valueOf(!Boolean.parseBoolean(usePathStyle)));
122+
}
123+
124+
// Lance refuses a plain-HTTP endpoint unless this is set, and Doris configures one for
125+
// MinIO. It describes the endpoint just mapped, so it is derived from the same properties.
126+
String endpoint = backendProperties.get("AWS_ENDPOINT");
127+
if (endpoint != null && endpoint.startsWith("http://")) {
128+
result.put(ALLOW_HTTP, "true");
129+
}
130+
return result;
131+
}
132+
133+
@Override
134+
public Map<String, String> normalizeVended(Map<String, String> vendedOptions) {
135+
Map<String, String> result = new HashMap<>();
136+
if (vendedOptions == null) {
137+
return result;
138+
}
139+
vendedOptions.forEach((key, value) -> {
140+
String canonical = CANONICAL_BY_ALIAS.getOrDefault(key.toLowerCase(Locale.ROOT), key);
141+
String previous = result.put(canonical, value);
142+
// Two spellings of one option, disagreeing. Picking one would be the coin toss this
143+
// whole class exists to remove, so say so instead.
144+
if (previous != null && !previous.equals(value)) {
145+
throw new IllegalArgumentException(
146+
"Lance namespace vended conflicting values for storage option '"
147+
+ canonical + "'");
148+
}
149+
});
150+
return result;
151+
}
152+
153+
private static void putIfNotEmpty(Map<String, String> target, String key, String value) {
154+
if (value != null && !value.isEmpty()) {
155+
target.put(key, value);
156+
}
157+
}
158+
}

‎fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceStorageOptions.java‎

Lines changed: 22 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -27,79 +27,43 @@
2727
* through lance-c, consume the map produced here, so neither can reach a dataset by a
2828
* configuration the other never saw.
2929
*
30-
* <p>Doris translates only its own static storage configuration. Options a namespace vends for a
31-
* table are overlaid key for key and otherwise left alone: the Lance Namespace specification
32-
* describes {@code storage_options} as configuration "passed directly to Lance", so the protocol
33-
* defines no key vocabulary of its own and a client cannot assume one.
34-
*
35-
* <p>In particular, vended keys are never renamed onto a canonical spelling. This class does not
36-
* know which provider a dataset uses, and the accepted spellings differ per provider: object_store's
37-
* Azure parser reads {@code endpoint} but not {@code aws_endpoint}, and Lance's OSS provider
38-
* requires {@code endpoint} and reads {@code access_key_id}. Canonicalizing onto the S3 names would
39-
* silently destroy a working configuration for every non-S3 backend, so a vended key reaches Lance
40-
* exactly as the namespace wrote it, and resolving equivalent spellings is left to Lance.
30+
* <p>The option vocabulary belongs to {@link LanceStorageProvider}, chosen from the dataset's URL
31+
* the same way Lance chooses one. This class only decides what happens when the catalog and the
32+
* namespace both describe a dataset: the namespace wins, because it just described the table.
4133
*/
4234
public final class LanceStorageOptions {
4335

44-
/**
45-
* Doris backend property to Lance object-store option.
46-
*
47-
* <p>Only S3-compatible storage is translated, which is all a Lance catalog's own properties
48-
* describe today. The spelling is the one object_store reports as canonical, which is what
49-
* {@code StorageOptions::with_env_s3} looks for before pulling the same option out of the
50-
* process environment, and which Lance's OpenDAL S3 backend accepts as a serde alias of its
51-
* own field names.
52-
*/
53-
private static final Map<String, String> S3_KEYS = new HashMap<>();
54-
55-
static {
56-
S3_KEYS.put("AWS_ACCESS_KEY", "aws_access_key_id");
57-
S3_KEYS.put("AWS_SECRET_KEY", "aws_secret_access_key");
58-
S3_KEYS.put("AWS_TOKEN", "aws_session_token");
59-
S3_KEYS.put("AWS_ENDPOINT", "aws_endpoint");
60-
S3_KEYS.put("AWS_REGION", "aws_region");
61-
}
62-
6336
private LanceStorageOptions() {
6437
}
6538

66-
/** Converts normalized Doris storage properties to Lance object-store options. */
39+
/**
40+
* Converts a catalog's own storage properties to Lance options.
41+
*
42+
* <p>Used where there is no dataset to route on - opening the namespace client itself, and the
43+
* {@code s3()} table-valued function. Both are S3-compatible storage by construction.
44+
*/
6745
public static Map<String, String> toLanceOptions(Map<String, String> backendProperties) {
68-
Map<String, String> result = new HashMap<>();
69-
S3_KEYS.forEach((dorisKey, lanceKey) -> putIfNotEmpty(result, lanceKey,
70-
backendProperties.get(dorisKey)));
71-
72-
String usePathStyle = backendProperties.get("use_path_style");
73-
if (usePathStyle != null && !usePathStyle.isEmpty()) {
74-
result.put("aws_virtual_hosted_style_request",
75-
String.valueOf(!Boolean.parseBoolean(usePathStyle)));
76-
}
77-
78-
// Lance refuses a plain-HTTP endpoint unless this is set, and Doris configures one for
79-
// MinIO. It describes the endpoint just mapped, so it is derived from the same properties.
80-
String endpoint = backendProperties.get("AWS_ENDPOINT");
81-
if (endpoint != null && endpoint.startsWith("http://")) {
82-
result.put("allow_http", "true");
83-
}
84-
return result;
46+
return LanceStorageProvider.forDorisCatalog().fromDorisProperties(backendProperties);
8547
}
8648

8749
/**
88-
* Overlays the options a namespace vended for one table onto the catalog's own.
50+
* Builds the options for one table, from the catalog's storage properties and whatever its
51+
* namespace vended for it.
8952
*
90-
* <p>A vended entry replaces the catalog's entry under the same key, and is otherwise added as
91-
* it arrives. Nothing is renamed, dropped, or interpreted - see the class comment.
53+
* <p>Both halves are put in the vocabulary of the provider Lance will route {@code datasetUri}
54+
* to, so a vended option lands on the same key as the catalog's and replaces it outright.
55+
* Without that they would reach Lance as two entries for one config key, and object_store
56+
* would keep whichever its HashMap yielded last - independently in the FE and in the BE.
9257
*/
93-
public static Map<String, String> mergeVended(Map<String, String> lanceOptions,
94-
Map<String, String> vendedOptions) {
95-
Map<String, String> result = new HashMap<>(lanceOptions);
58+
public static Map<String, String> forDataset(String datasetUri,
59+
Map<String, String> backendProperties, Map<String, String> vendedOptions) {
60+
LanceStorageProvider provider = LanceStorageProvider.forDataset(datasetUri);
61+
Map<String, String> result = new HashMap<>(provider.fromDorisProperties(backendProperties));
9662
if (vendedOptions == null) {
9763
return result;
9864
}
99-
vendedOptions.forEach((key, value) -> {
100-
validateVendedOption(key, value);
101-
result.put(key, value);
102-
});
65+
vendedOptions.forEach(LanceStorageOptions::validateVendedOption);
66+
result.putAll(provider.normalizeVended(vendedOptions));
10367
return result;
10468
}
10569

@@ -122,10 +86,4 @@ private static void validateVendedOption(String key, String value) {
12286
+ "' with a NUL in its key or value, which cannot reach the backend");
12387
}
12488
}
125-
126-
private static void putIfNotEmpty(Map<String, String> target, String key, String value) {
127-
if (value != null && !value.isEmpty()) {
128-
target.put(key, value);
129-
}
130-
}
13189
}

0 commit comments

Comments
 (0)