Skip to content

Commit a04c8e1

Browse files
committed
[fix](lance) Stop rewriting namespace-vended storage options
The alias table this series grew was the wrong shape, and it broke every non-S3 backend. It canonicalized vended keys onto the AWS spellings without knowing which provider a dataset uses, but the accepted spellings differ per provider: // object_store-0.13.2 src/azure/builder.rs:448 "azure_storage_endpoint" | "azure_endpoint" | "endpoint" => Ok(Self::Endpoint) // lance-io/src/object_store/providers/oss.rs:71-73, :87 ("endpoint", &["oss_endpoint"]), ("access_key_id", &["oss_access_key_id"]), ... "OSS endpoint is required..." So a namespace describing `az://container/table` with `endpoint=...`, or `oss://bucket/table` with `endpoint=...` and `access_key_id=...`, had exactly the options it needs renamed into AWS keys those providers do not read: the Azure endpoint was silently lost, and OSS failed outright. That is the opposite of what this series set out to do, which was to stop dropping vended options. The table could not be made correct without a provider, and it was tracking two libraries at two versions each - object_store plus OpenDAL, 0.57 on the FE and 0.56 on the BE. Resolving equivalent spellings belongs to Lance, which knows the provider; Doris only has to carry the options across the FE/BE boundary intact. mergeVended is now an exact-key overlay: a vended entry replaces the catalog's entry under the same key, and everything else arrives as the namespace wrote it. Gone with the table are the ambiguous-`token` special case, the `aws_endpoint_url_s3` precedence handling, the endpoint double-write, allow_http retraction, and the protected-key filter - the last of which was never a boundary anyway, since the namespace already decides the dataset URI. The catalog's own static properties still map to the canonical `aws_*` names. That path is S3-only by construction, so the spelling is provider-correct there, and it is what `with_env_s3` checks before folding in the environment. Options that cannot cross the boundary now fail instead of being dropped. A NUL truncates a key at lance-c's `CStr::from_ptr` while the FE keeps reading the whole thing, so the FE rejects it and the BE returns InvalidArgument; dropping it silently only moved the divergence to an FE that predates the check. Empty values are carried rather than second-guessed - a C string expresses them fine. Verified end to end against the docker Lance REST fixture: both the prefixed and the unprefixed credential-vending tables scan correctly with no static credentials configured, which is the case that motivated the series - and it now works with no alias normalization at all. Claude-Session: https://claude.ai/code/session_01PfVSEUJsfWwKy7WEuBaFCG
1 parent e850bff commit a04c8e1

4 files changed

Lines changed: 140 additions & 483 deletions

File tree

‎be/src/format_v2/table/lance_reader.cpp‎

Lines changed: 23 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,6 @@
2828
#include <limits>
2929
#include <memory>
3030

31-
#include "common/logging.h"
3231
#include "core/data_type/data_type_array.h"
3332
#include "core/data_type/data_type_factory.hpp"
3433
#include "core/data_type/data_type_map.h"
@@ -274,7 +273,8 @@ Status LanceTableReader::fetch_schema(const TFileRangeDesc& range,
274273
RETURN_IF_ERROR(_validate_range(range));
275274

276275
const auto& params = range.table_format_params.lance_params;
277-
const auto storage_options = _storage_options(&scan_params);
276+
std::vector<std::string> storage_options;
277+
RETURN_IF_ERROR(_storage_options(&scan_params, &storage_options));
278278
std::vector<const char*> storage_option_ptrs;
279279
storage_option_ptrs.reserve(storage_options.size() + 1);
280280
for (const auto& option : storage_options) {
@@ -386,7 +386,8 @@ Status LanceTableReader::prepare_split(const SplitReadOptions& options) {
386386
}
387387
}
388388

389-
const auto key = _dataset_key(options.current_range);
389+
DatasetKey key;
390+
RETURN_IF_ERROR(_dataset_key(options.current_range, &key));
390391
if (_dataset == nullptr) {
391392
RETURN_IF_ERROR(_open_dataset(key));
392393
_opened_dataset_key = key;
@@ -898,37 +899,34 @@ Status LanceTableReader::_fill_block_from_arrow(LanceBatch* batch, Block* block,
898899
// The FE sends these already in Lance's own vocabulary, merged from the catalog properties and
899900
// from whatever the namespace vended. Re-encoding them here would drop every option this list did
900901
// not anticipate, so they are handed to lance-c as they arrive.
901-
std::vector<std::string> LanceTableReader::_storage_options(
902-
const TFileScanRangeParams* scan_params) {
902+
Status LanceTableReader::_storage_options(const TFileScanRangeParams* scan_params,
903+
std::vector<std::string>* options) {
904+
options->clear();
903905
if (scan_params == nullptr || !scan_params->__isset.lance_storage_options) {
904-
return {};
906+
return Status::OK();
905907
}
906-
std::vector<std::string> options;
907-
options.reserve(scan_params->lance_storage_options.size() * 2);
908+
options->reserve(scan_params->lance_storage_options.size() * 2);
908909
for (const auto& [key, value] : scan_params->lance_storage_options) {
909-
if (value.empty()) {
910-
continue;
911-
}
912-
// These become C strings below, so an embedded NUL would silently truncate the option and
913-
// leave lance-c reading a different key than the FE resolved. The FE drops those already;
914-
// this guards against one that predates the check.
910+
// These become C strings below, so a NUL would truncate the option here while the FE went
911+
// on using the whole thing, and the two halves would open the dataset with different
912+
// configuration. The FE rejects these already; dropping one here instead of failing would
913+
// just recreate that divergence against an FE that predates the check.
915914
if (key.find('\0') != std::string::npos || value.find('\0') != std::string::npos) {
916-
LOG(WARNING) << "Ignoring Lance storage option whose key or value contains a NUL";
917-
continue;
915+
return Status::InvalidArgument(
916+
"Lance storage option '{}' contains a NUL and cannot reach lance-c",
917+
key.substr(0, key.find('\0')));
918918
}
919-
options.emplace_back(key);
920-
options.emplace_back(value);
919+
options->emplace_back(key);
920+
options->emplace_back(value);
921921
}
922-
return options;
922+
return Status::OK();
923923
}
924924

925-
LanceTableReader::DatasetKey LanceTableReader::_dataset_key(const TFileRangeDesc& range) const {
925+
Status LanceTableReader::_dataset_key(const TFileRangeDesc& range, DatasetKey* key) const {
926926
const auto& params = range.table_format_params.lance_params;
927-
return {
928-
.uri = params.dataset_uri,
929-
.version = params.version,
930-
.storage_options = _storage_options(_scan_params),
931-
};
927+
key->uri = params.dataset_uri;
928+
key->version = params.version;
929+
return _storage_options(_scan_params, &key->storage_options);
932930
}
933931

934932
Status LanceTableReader::_lance_error(std::string_view operation) {

‎be/src/format_v2/table/lance_reader.h‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -84,8 +84,9 @@ class LanceTableReader final : public TableReader {
8484
void _close_scanner();
8585
void _close_dataset();
8686
Status _fill_block_from_arrow(LanceBatch* batch, Block* block, size_t* rows);
87-
static std::vector<std::string> _storage_options(const TFileScanRangeParams* scan_params);
88-
DatasetKey _dataset_key(const TFileRangeDesc& range) const;
87+
static Status _storage_options(const TFileScanRangeParams* scan_params,
88+
std::vector<std::string>* options);
89+
Status _dataset_key(const TFileRangeDesc& range, DatasetKey* key) const;
8990
static Status _lance_error(std::string_view operation);
9091

9192
LanceDataset* _dataset = nullptr;

0 commit comments

Comments
 (0)