Skip to content

Commit 5442385

Browse files
authored
[fix](lance) Pass namespace-vended storage options through to the BE (#66805)
### What problem does this PR solve? Fixes #66772 (problem 1). A Lance REST catalog discarded the `storage_options` a namespace vended for a table, so any scan relying on credential vending failed: ``` open Lance dataset failed: LanceError(IO): ... Failed to get AWS credentials: CredentialsNotLoaded("no providers in chain provided credentials") ``` The options were re-encoded twice on the way to the BE, through one five-entry S3-only table used in both directions: ``` namespace vends access_key_id -> FE LanceStorageOptions.forBackend aws_access_key_id -> AWS_ACCESS_KEY -> TFileScanRangeParams.properties -> BE kStorageKeys AWS_ACCESS_KEY -> aws_access_key_id -> lance-c ``` That table was written to *emit* one canonical spelling, which is correct outbound. Reading it backwards turns it into a parser that accepts only the spelling it happens to emit, so everything else was dropped — credentials under any other accepted alias, and every non-S3 provider's keys, which left the catalog unable to use credential vending outside S3 at all. The failure was also split across the two halves, which made it hard to read: the FE passes the vended map to the Lance Java SDK untouched, so `SHOW TABLES` and `DESC` succeeded and only the scan failed. The Lance Namespace specification describes `storage_options` as configuration *"passed directly to Lance"*, so the protocol defines no key vocabulary of its own and a client cannot assume one. ### Release note Fixed a Lance REST catalog failing to scan with `CredentialsNotLoaded` when it relied on credentials vended by the namespace rather than static `s3.access_key` / `s3.secret_key`. ### What is changed and how it works? Stop re-encoding server-supplied options, and resolve the ones Doris does contribute per storage provider. **Transport.** `TFileScanRangeParams.lance_storage_options` (new, id 38) carries the options in Lance's own vocabulary. Set at ScanNode level — the same pattern as `paimon_options` — so credentials are not serialized once per fragment split. `properties` is deliberately not reused: it is copied wholesale into `io::FileSystemProperties` for the shared filesystem layer. The BE hands that map to lance-c as it arrives, so `kStorageKeys` is gone. The `s3()` TVF path populates the same field, including the schema-fetch RPC. **The option vocabulary belongs to a provider, not to this layer.** Lance routes a dataset to a provider by URL scheme, and the accepted spellings overlap without agreeing: ```rust // object_store-0.13.2 src/azure/builder.rs:448 — note: no aws_endpoint "azure_storage_endpoint" | "azure_endpoint" | "endpoint" => Ok(Self::Endpoint) // lance-io/src/object_store/providers/oss.rs:71-73, :87 ("endpoint", &["oss_endpoint"]), ("access_key_id", &["oss_access_key_id"]), ... "OSS endpoint is required. Please provide 'oss_endpoint' ..." ``` Rewriting a vended `endpoint` onto `aws_endpoint` therefore loses the Azure endpoint outright and makes OSS fail before it reads anything. `LanceStorageProvider` picks by scheme the way lance-io's own registry does, and each implementation owns both directions of its vocabulary: - **`LanceS3StorageProvider`** — the `s3` and `s3+ddb` schemes lance-io registers for its AWS provider. It converts the catalog's typed storage properties into the canonical `aws_*` spellings, and resolves vended spellings onto those same keys. `token` is decidable here too: it is a bearer token to object_store's Azure parser, and only the provider settles which it is. - **`LancePassThroughStorageProvider`** — everything else. Both halves inert, so such a dataset is reachable only through what its namespace vends, named as the namespace named it. **Why the two sides have to be resolved against each other.** A catalog with static `s3.access_key` whose namespace also vends credentials produces both spellings of one option: ``` aws_access_key_id = <catalog> access_key_id = <vended> ``` Both parse to `AmazonS3ConfigKey::AccessKeyId`, and `as_s3_options()` collects them into a `HashMap`, so the winner is whichever the iteration yields last — decided separately for the access key and the secret, and separately in the FE and in the BE. Two of the four combinations pair one side's key with the other's secret, which surfaces as `SignatureDoesNotMatch` rather than as a missing credential. Putting both sides in the provider's vocabulary first means one key, one value, and the namespace wins because it just described the table. The alias set covers only the options Doris itself contributes — those are the only ones a vended option can collide with. `aws_endpoint_url_s3` is deliberately left alone: object_store parses it into a config key of its own and prefers it over the generic endpoint, so a vended one already wins, and folding it in would replace a defined precedence with map order. **Doris's own configuration is read through its typed properties**, not through the flattened `AWS_*` backend map, which is only a re-encoding of them (`AbstractS3CompatibleProperties.doBuildS3Configuration`). That map also mixes in backend-only knobs Lance has no use for, and flattens every configured storage into one namespace where two S3-compatible ones overwrite each other. The typed list keeps them apart; the one to read is chosen the way Iceberg chooses it (`AbstractIcebergProperties.toFileIOProperties`) — prefer a concrete provider over the generic `S3Properties`. Selection filters by type rather than index, because `StorageProperties.createAll` prepends a default HDFS entry when no `fs.xx.support` flag is set. **What is rejected rather than resolved.** A NUL truncates a key at lance-c's `CStr::from_ptr` while the FE keeps reading the whole thing, so the FE rejects it and the BE returns `InvalidArgument`; dropping it silently would only move the divergence to an FE that predates the check. A namespace vending two spellings of one option with different values is rejected too — picking one would be the coin toss above. ### Not in scope **Credentials are static for the life of a scan.** `lance-c` opens datasets with a static option set, so `expires_at_millis` reaches the BE but is never acted on, and credentials that expire mid-scan are not re-vended. Renewal needs a channel of its own; this field is not one, and the thrift comment says so. **One map, two Lances.** The FE and the BE agree on the options, not on the library that reads them: both pin `object_store` 0.13.2, but the Java SDK (9.1.0-beta.3) carries OpenDAL 0.57 and `lance-c` 0.1.6 carries 0.56. An option only the newer one knows — `skip_signature`, say — takes effect on the FE and is ignored on the BE. This only reaches the OpenDAL backend, which a namespace has to opt into by vending `use_opendal=true`, and closing it means aligning the two upstream pins. **Non-S3 providers are unblocked, not enabled.** Their vended options now survive instead of being dropped or rewritten, but Doris contributes no static configuration for them — it models OSS and COS perfectly well, so writing that translation is possible, but it would mean committing to a vocabulary per provider with no backend here to test it against, which is how the rewriting bug above got in. A filesystem catalog's `warehouse` is still restricted to `file` and `s3`. **A catalog and a namespace that disagree.** If both name the same option in spellings the provider treats as distinct config keys, both survive and Lance decides. Only the spellings Doris itself emits are resolved. ### Rolling upgrade An FE upgraded ahead of the BEs no longer puts vended credentials into `TFileScanRangeParams.properties`, and an older BE reads only that. A REST catalog with no static credentials therefore cannot be scanned until the BEs are upgraded too. The Lance catalog is not in a release yet, so this only affects development clusters. ### How was this patch tested? **Unit tests.** `LanceStorageOptionsTest` is new — `LanceStorageOptions` had no test of its own, and the deleted `forBackend` had zero call sites in any test, which is why the original bug went unnoticed. Its fixtures are built with `StorageProperties.createAll` from user-facing properties, so they exercise Doris's real parsing and have to satisfy its rules. It covers the canonical mapping and anonymous access, vended credentials superseding the catalog's, the full endpoint alias class, `token` resolving differently on `s3://` than on `az://`, six non-S3 schemes arriving untouched, S3 credentials not leaking onto another provider, selection skipping the non-S3 entry Doris prepends, unknown options passing through, conflicting spellings and embedded NULs being rejected. `LanceThriftContractTest` gains round-trip coverage of the new field. **Regression.** The docker stub only ever vended `aws_`-prefixed keys, so the alias path had no coverage anywhere. It now serves a second table, `all_types_unprefixed`, backed by the same dataset but vending the unprefixed spelling, and `test_lance_rest_catalog` scans it with no static credentials configured. Both tables scan correctly on the current revision. **End to end**, against Apache Gravitino 1.3.0's `lance-rest` service, which vends the unprefixed spelling. This run validated the original transport fix, on the first revision of this PR; the docker suite above is what has been re-run since. Catalog with no `s3.access_key` / `s3.secret_key`: ```sql CREATE CATALOG lance_novend PROPERTIES ( "type" = "lance", "lance.catalog.type" = "rest", "lance.rest.uri" = "http://127.0.0.1:9101/lance", "lance.namespace.parent" = "lance_catalog" ); SELECT count(*), min(row_id), max(row_id), sum(row_id) FROM lance_novend.doris_probe.rest_probe; -- 1034 1 1034 535095 ``` against a 1034-row, 2-fragment dataset with an IVF_FLAT index. Predicate pushdown and vector search both return correct results on the same catalog, and a catalog with static credentials is unaffected. Before this change the same statement failed with `CredentialsNotLoaded`. https://claude.ai/code/session_01M3mYXBKShBonG6Lg3br4Ld
1 parent 05fafcc commit 5442385

25 files changed

Lines changed: 1012 additions & 172 deletions

‎be/src/format_v2/table/lance_reader.cpp‎

Lines changed: 32 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -261,7 +261,8 @@ Status LanceTableReader::fetch_schema(const TFileRangeDesc& range,
261261
return Status::InvalidArgument("Lance schema output must not be null");
262262
}
263263
const auto& params = range.table_format_params.lance_params;
264-
const auto storage_options = _storage_options(&scan_params);
264+
std::vector<std::string> storage_options;
265+
RETURN_IF_ERROR(_storage_options(&scan_params, &storage_options));
265266
std::vector<const char*> storage_option_ptrs;
266267
storage_option_ptrs.reserve(storage_options.size() + 1);
267268
for (const auto& option : storage_options) {
@@ -624,7 +625,8 @@ Status LanceTableReader::_validate_external_search_request() const {
624625
}
625626

626627
Status LanceTableReader::_ensure_dataset_open(const TFileRangeDesc& range) {
627-
const auto key = _dataset_key(range);
628+
DatasetKey key;
629+
RETURN_IF_ERROR(_dataset_key(range, &key));
628630
if (_dataset == nullptr) {
629631
RETURN_IF_ERROR(_open_dataset(key));
630632
_opened_dataset_key = key;
@@ -1025,47 +1027,39 @@ Status LanceTableReader::_fill_block_from_record_batch(
10251027
return Status::OK();
10261028
}
10271029

1028-
std::vector<std::string> LanceTableReader::_storage_options(
1029-
const TFileScanRangeParams* scan_params) {
1030-
if (scan_params == nullptr || !scan_params->__isset.properties) {
1031-
return {};
1032-
}
1033-
static constexpr std::array<std::pair<std::string_view, std::string_view>, 5> kStorageKeys = {
1034-
{{"AWS_ACCESS_KEY", "aws_access_key_id"},
1035-
{"AWS_SECRET_KEY", "aws_secret_access_key"},
1036-
{"AWS_TOKEN", "aws_session_token"},
1037-
{"AWS_ENDPOINT", "aws_endpoint"},
1038-
{"AWS_REGION", "aws_region"}}};
1039-
std::vector<std::string> options;
1040-
options.reserve(kStorageKeys.size() * 2);
1041-
for (const auto& [doris_key, lance_key] : kStorageKeys) {
1042-
const auto it = scan_params->properties.find(std::string(doris_key));
1043-
if (it != scan_params->properties.end() && !it->second.empty()) {
1044-
options.emplace_back(lance_key);
1045-
options.emplace_back(it->second);
1046-
}
1047-
}
1048-
const auto endpoint = scan_params->properties.find("AWS_ENDPOINT");
1049-
if (endpoint != scan_params->properties.end() && endpoint->second.rfind("http://", 0) == 0) {
1050-
options.emplace_back("allow_http");
1051-
options.emplace_back("true");
1030+
// The FE sends these already in Lance's own vocabulary, merged from the catalog properties and
1031+
// from whatever the namespace vended. Re-encoding them here would drop every option this list did
1032+
// not anticipate, so they are handed to lance-c as they arrive.
1033+
Status LanceTableReader::_storage_options(const TFileScanRangeParams* scan_params,
1034+
std::vector<std::string>* options) {
1035+
options->clear();
1036+
if (scan_params == nullptr || !scan_params->__isset.lance_storage_options) {
1037+
return Status::OK();
10521038
}
1053-
const auto path_style = scan_params->properties.find("use_path_style");
1054-
if (path_style != scan_params->properties.end() && !path_style->second.empty()) {
1055-
const bool use_path_style = path_style->second == "true" || path_style->second == "1";
1056-
options.emplace_back("aws_virtual_hosted_style_request");
1057-
options.emplace_back(use_path_style ? "false" : "true");
1039+
options->reserve(scan_params->lance_storage_options.size() * 2);
1040+
for (const auto& [key, value] : scan_params->lance_storage_options) {
1041+
// These become C strings below, so a NUL would truncate the option here while the FE went
1042+
// on using the whole thing, and the two halves would open the dataset with different
1043+
// configuration. The FE rejects these on both paths it builds options from - its own
1044+
// storage configuration and what a namespace vends - so this is the last line of defence,
1045+
// for an FE that predates those checks. Dropping one here instead of failing would just
1046+
// recreate the divergence it exists to prevent.
1047+
if (key.find('\0') != std::string::npos || value.find('\0') != std::string::npos) {
1048+
return Status::InvalidArgument(
1049+
"Lance storage option '{}' contains a NUL and cannot reach lance-c",
1050+
key.substr(0, key.find('\0')));
1051+
}
1052+
options->emplace_back(key);
1053+
options->emplace_back(value);
10581054
}
1059-
return options;
1055+
return Status::OK();
10601056
}
10611057

1062-
LanceTableReader::DatasetKey LanceTableReader::_dataset_key(const TFileRangeDesc& range) const {
1058+
Status LanceTableReader::_dataset_key(const TFileRangeDesc& range, DatasetKey* key) const {
10631059
const auto& params = range.table_format_params.lance_params;
1064-
return {
1065-
.uri = params.dataset_uri,
1066-
.version = params.version,
1067-
.storage_options = _storage_options(_scan_params),
1068-
};
1060+
key->uri = params.dataset_uri;
1061+
key->version = params.version;
1062+
return _storage_options(_scan_params, &key->storage_options);
10691063
}
10701064

10711065
Status LanceTableReader::_lance_error(std::string_view operation) {

‎be/src/format_v2/table/lance_reader.h‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -95,8 +95,9 @@ class LanceTableReader final : public TableReader {
9595
Block* block, size_t* rows);
9696
Status _append_global_row_ids(const std::shared_ptr<arrow::Array>& row_ids,
9797
MutableColumnPtr& output_column) const;
98-
static std::vector<std::string> _storage_options(const TFileScanRangeParams* scan_params);
99-
DatasetKey _dataset_key(const TFileRangeDesc& range) const;
98+
static Status _storage_options(const TFileScanRangeParams* scan_params,
99+
std::vector<std::string>* options);
100+
Status _dataset_key(const TFileRangeDesc& range, DatasetKey* key) const;
100101
static Status _lance_error(std::string_view operation);
101102

102103
LanceDataset* _dataset = nullptr;

‎be/test/format_v2/table/lance_reader_test.cpp‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -948,6 +948,31 @@ TEST(LanceTableReaderScanTest, ReadsLatestSnapshotWithoutFragmentIds) {
948948
EXPECT_TRUE(reader.close().ok());
949949
}
950950

951+
TEST(LanceTableReaderScanTest, RejectsStorageOptionWithEmbeddedNul) {
952+
const std::filesystem::path dataset_uri =
953+
"./be/test/format_v2/table/lance/data/all_types.lance";
954+
const Columns columns {projected_column("row_id", TYPE_BIGINT, false)};
955+
TQueryGlobals query_globals;
956+
RuntimeState state(query_globals);
957+
RuntimeProfile profile("lance_storage_option_embedded_nul");
958+
959+
// lance-c reads these as C strings, so a NUL truncates the option here while the FE goes on
960+
// using the whole thing, leaving the two halves opening the dataset with different
961+
// configuration. Dropping it instead of failing would only move that divergence.
962+
TFileScanRangeParams scan_params;
963+
scan_params.__set_lance_storage_options(
964+
{{std::string("aws_region\0ignored", 18), "us-east-1"}});
965+
966+
LanceTableReader reader;
967+
ASSERT_TRUE(init_reader(&reader, columns, &state, &profile, &scan_params).ok());
968+
969+
const auto status = prepare_range(&reader, make_latest_lance_range(dataset_uri));
970+
971+
EXPECT_FALSE(status.ok());
972+
EXPECT_NE(status.to_string().find("contains a NUL"), std::string::npos);
973+
EXPECT_TRUE(reader.close().ok());
974+
}
975+
951976
TEST(LanceTableReaderTypeTest, ReadsNumericTypesFromAllTypesFixture) {
952977
// The committed fixture contains four rows covering values, nulls, and boundary cases.
953978
const std::filesystem::path dataset_uri =

‎docker/thirdparties/docker-compose/iceberg/iceberg.yaml.tpl‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,10 @@ services:
118118
- ./scripts/lance_rest_server.py:/opt/lance-rest/server.py:ro
119119
environment:
120120
LANCE_REST_BEARER_TOKEN: doris-lance-rest-test-token
121-
LANCE_REST_TABLES_JSON: '{"all_types":"s3://warehouse/lance/all_types.lance"}'
121+
LANCE_REST_TABLES_JSON: '{"all_types":"s3://warehouse/lance/all_types.lance","all_types_unprefixed":"s3://warehouse/lance/all_types.lance"}'
122+
# all_types_unprefixed serves the same dataset but vends its credentials under the
123+
# unprefixed object-store spelling, which is what real namespace servers emit.
124+
LANCE_REST_UNPREFIXED_TABLES_JSON: '["all_types_unprefixed"]'
122125
LANCE_S3_ACCESS_KEY: admin
123126
LANCE_S3_SECRET_KEY: password
124127
LANCE_S3_REGION: us-east-1

‎docker/thirdparties/docker-compose/iceberg/scripts/lance_rest_server.py‎

Lines changed: 45 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,50 @@ def _load_tables() -> dict[tuple[str, ...], str]:
6565
TABLES = _load_tables()
6666

6767

68+
def _load_unprefixed_tables() -> set[tuple[str, ...]]:
69+
"""Tables whose vended credentials use the unprefixed object-store spelling.
70+
71+
A namespace may spell credentials with any alias Lance accepts, and real servers do use the
72+
unprefixed one, so at least one table has to exercise it.
73+
"""
74+
raw = os.environ.get("LANCE_REST_UNPREFIXED_TABLES_JSON", "[]")
75+
identifiers = json.loads(raw)
76+
if not isinstance(identifiers, list):
77+
raise ValueError("LANCE_REST_UNPREFIXED_TABLES_JSON must be a JSON array")
78+
return {
79+
tuple(part for part in identifier.split(DELIMITER) if part)
80+
for identifier in identifiers
81+
}
82+
83+
84+
UNPREFIXED_TABLES = _load_unprefixed_tables()
85+
86+
87+
def _storage_options(identifier: tuple[str, ...]) -> dict[str, str]:
88+
access_key = os.environ.get("LANCE_S3_ACCESS_KEY", "admin")
89+
secret_key = os.environ.get("LANCE_S3_SECRET_KEY", "password")
90+
region = os.environ.get("LANCE_S3_REGION", "us-east-1")
91+
if identifier in UNPREFIXED_TABLES:
92+
return {
93+
"access_key_id": access_key,
94+
"secret_access_key": secret_key,
95+
"region": region,
96+
"virtual_hosted_style_request": "false",
97+
# A key Doris assigns no meaning to, kept here so the pass-through stays covered.
98+
# The BE opens datasets with static options and never refreshes them, so this is
99+
# only carried, not acted on; the expiry is far enough out that it never matters.
100+
"expires_at_millis": os.environ.get(
101+
"LANCE_S3_EXPIRES_AT_MILLIS", "4102444800000"
102+
),
103+
}
104+
return {
105+
"aws_access_key_id": access_key,
106+
"aws_secret_access_key": secret_key,
107+
"aws_region": region,
108+
"aws_virtual_hosted_style_request": "false",
109+
}
110+
111+
68112
def _decode_identifier(identifier: str) -> tuple[str, ...]:
69113
identifier = unquote(identifier)
70114
if identifier == DELIMITER:
@@ -141,16 +185,7 @@ def do_POST(self) -> None:
141185
"namespace": list(identifier[:-1]),
142186
"location": table_uri,
143187
"table_uri": table_uri,
144-
"storage_options": {
145-
"aws_access_key_id": os.environ.get(
146-
"LANCE_S3_ACCESS_KEY", "admin"
147-
),
148-
"aws_secret_access_key": os.environ.get(
149-
"LANCE_S3_SECRET_KEY", "password"
150-
),
151-
"aws_region": os.environ.get("LANCE_S3_REGION", "us-east-1"),
152-
"aws_virtual_hosted_style_request": "false",
153-
},
188+
"storage_options": _storage_options(identifier),
154189
"managed_versioning": False,
155190
"is_only_declared": False,
156191
},

‎fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalCatalog.java‎

Lines changed: 27 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -86,8 +86,7 @@ public class LanceExternalCatalog extends ExternalCatalog {
8686
private transient List<String> parentNamespace = Collections.emptyList();
8787
private transient String catalogType;
8888
private transient String rootDatabase;
89-
private transient Map<String, String> javaStorageOptions = Collections.emptyMap();
90-
private transient Map<String, String> backendStorageOptions = Collections.emptyMap();
89+
private transient Map<String, String> namespaceStorageOptions = Collections.emptyMap();
9190
private transient Object namespaceLock = new Object();
9291

9392
public LanceExternalCatalog(long catalogId, String name, String resource, Map<String, String> props,
@@ -105,11 +104,12 @@ protected void initLocalObjectsImpl() {
105104
rootDatabase = properties.getRootDatabase();
106105
parentNamespace = LanceNamespaceName.parseParentNamespace(
107106
properties.getNamespaceParent(), properties.getNamespaceDelimiter());
108-
backendStorageOptions = catalogProperty.getBackendStorageProperties();
109-
javaStorageOptions = LanceStorageOptions.forJavaSdk(backendStorageOptions);
107+
namespaceStorageOptions = LanceStorageOptions.forUri(
108+
properties.getNamespaceStorageUri(),
109+
catalogProperty.getOrderedStoragePropertiesList());
110110

111111
allocator = new RootAllocator(ALLOCATOR_LIMIT);
112-
namespace = properties.createNamespace(allocator, javaStorageOptions);
112+
namespace = properties.createNamespace(allocator, namespaceStorageOptions);
113113
} catch (Exception e) {
114114
closeLanceObjects();
115115
throw new RuntimeException("Failed to initialize Lance catalog '" + getName()
@@ -127,8 +127,9 @@ public void checkWhenCreating() throws DdlException {
127127
}
128128

129129
AbstractLanceProperties properties = getLanceProperties();
130-
Map<String, String> storageOptions = LanceStorageOptions.forJavaSdk(
131-
catalogProperty.getBackendStorageProperties());
130+
Map<String, String> storageOptions = LanceStorageOptions.forUri(
131+
properties.getNamespaceStorageUri(),
132+
catalogProperty.getOrderedStoragePropertiesList());
132133
List<String> parent = LanceNamespaceName.parseParentNamespace(
133134
properties.getNamespaceParent(), properties.getNamespaceDelimiter());
134135
String type = properties.getLanceCatalogType();
@@ -316,16 +317,16 @@ private LanceTableMetadata loadTableMetadata(String dbName, String tableName,
316317
"Cannot parse Lance FOR TIME AS OF value '" + snapshot.getValue() + "'");
317318
}
318319
version = LanceSnapshotResolver.getVersionAtOrBefore(
319-
tableAccess.datasetUri, tableAccess.javaStorageOptions, timestamp, allocator);
320+
tableAccess.datasetUri, tableAccess.storageOptions, timestamp, allocator);
320321
}
321-
return LanceMetadataLoader.loadVersion(tableAccess.datasetUri, tableAccess.javaStorageOptions,
322-
tableAccess.backendStorageOptions, version, allocator);
322+
return LanceMetadataLoader.loadVersion(
323+
tableAccess.datasetUri, tableAccess.storageOptions, version, allocator);
323324
}
324325
return loadIndexSegments
325326
? LanceMetadataLoader.loadLatestWithIndexSegments(tableAccess.datasetUri,
326-
tableAccess.javaStorageOptions, tableAccess.backendStorageOptions, allocator)
327-
: LanceMetadataLoader.loadLatest(tableAccess.datasetUri, tableAccess.javaStorageOptions,
328-
tableAccess.backendStorageOptions, allocator);
327+
tableAccess.storageOptions, allocator)
328+
: LanceMetadataLoader.loadLatest(
329+
tableAccess.datasetUri, tableAccess.storageOptions, allocator);
329330
} catch (Exception e) {
330331
throw new RuntimeException("Failed to load Lance table metadata for " + dbName + "." + tableName
331332
+ ": " + sanitizedRootCauseMessage(e), safeCause(e));
@@ -340,7 +341,8 @@ public List<LanceLogicalIndex> loadTableIndexMetadata(
340341
try {
341342
makeSureInitialized();
342343
} catch (Exception e) {
343-
throw indexMetadataLoadFailure(dbName, tableName, e, null, javaStorageOptions);
344+
throw indexMetadataLoadFailure(
345+
dbName, tableName, e, null, namespaceStorageOptions);
344346
}
345347

346348
ResolvedTableAccess tableAccess = null;
@@ -351,7 +353,7 @@ public List<LanceLogicalIndex> loadTableIndexMetadata(
351353
// The deadline below covers the Dataset/JNI index metadata read itself.
352354
tableAccess = resolveTableAccess(dbName, tableName);
353355
String datasetUri = tableAccess.datasetUri;
354-
Map<String, String> storageOptions = tableAccess.javaStorageOptions;
356+
Map<String, String> storageOptions = tableAccess.storageOptions;
355357
return LanceMetadataReadExecutor.execute(() -> {
356358
// The caller may return on deadline while JNI is still running. A task-owned
357359
// allocator prevents catalog close from releasing native resources prematurely.
@@ -362,7 +364,7 @@ public List<LanceLogicalIndex> loadTableIndexMetadata(
362364
} catch (Exception e) {
363365
String datasetUri = tableAccess == null ? null : tableAccess.datasetUri;
364366
Map<String, String> runtimeStorageOptions = tableAccess == null
365-
? javaStorageOptions : tableAccess.javaStorageOptions;
367+
? namespaceStorageOptions : tableAccess.storageOptions;
366368
throw indexMetadataLoadFailure(
367369
dbName, tableName, e, datasetUri, runtimeStorageOptions);
368370
}
@@ -391,13 +393,12 @@ private ResolvedTableAccess resolveTableAccess(String dbName, String tableName)
391393
throw new RuntimeException("Lance namespace returned no table URI for " + dbName + "." + tableName);
392394
}
393395

394-
Map<String, String> tableJavaStorageOptions = new HashMap<>(javaStorageOptions);
395-
if (table.getStorageOptions() != null) {
396-
tableJavaStorageOptions.putAll(table.getStorageOptions());
397-
}
398-
Map<String, String> tableBackendStorageOptions = LanceStorageOptions.forBackend(
399-
backendStorageOptions, table.getStorageOptions());
400-
return new ResolvedTableAccess(datasetUri, tableJavaStorageOptions, tableBackendStorageOptions);
396+
// One option map serves both readers: the FE opens the dataset through the Lance Java SDK
397+
// and the BE through lance-c, so neither can end up with credentials the other lacks. The
398+
// dataset URL picks the option vocabulary, the same way Lance picks a provider from it.
399+
Map<String, String> storageOptions = LanceStorageOptions.forVendedTable(datasetUri,
400+
catalogProperty.getOrderedStoragePropertiesList(), table.getStorageOptions());
401+
return new ResolvedTableAccess(datasetUri, storageOptions);
401402
}
402403

403404
private DescribeTableResponse describeTable(String dbName, String tableName) {
@@ -431,11 +432,6 @@ private List<String> buildFullNamespace(List<String> relativeNamespace) {
431432
return result;
432433
}
433434

434-
public Map<String, String> getBackendStorageOptions() {
435-
makeSureInitialized();
436-
return backendStorageOptions;
437-
}
438-
439435
public String getLanceCatalogType() {
440436
makeSureInitialized();
441437
return catalogType;
@@ -538,14 +534,11 @@ private static String truncateUtf8(String value, int maxBytes) {
538534

539535
private static final class ResolvedTableAccess {
540536
private final String datasetUri;
541-
private final Map<String, String> javaStorageOptions;
542-
private final Map<String, String> backendStorageOptions;
537+
private final Map<String, String> storageOptions;
543538

544-
private ResolvedTableAccess(String datasetUri, Map<String, String> javaStorageOptions,
545-
Map<String, String> backendStorageOptions) {
539+
private ResolvedTableAccess(String datasetUri, Map<String, String> storageOptions) {
546540
this.datasetUri = datasetUri;
547-
this.javaStorageOptions = Collections.unmodifiableMap(new HashMap<>(javaStorageOptions));
548-
this.backendStorageOptions = Collections.unmodifiableMap(new HashMap<>(backendStorageOptions));
541+
this.storageOptions = Collections.unmodifiableMap(new HashMap<>(storageOptions));
549542
}
550543
}
551544
}

0 commit comments

Comments
 (0)