You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5442385
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](lance) Pass namespace-vended storage options through to the BE (#66805)
### What problem does this PR solve?
Fixes#66772 (problem 1).
A Lance REST catalog discarded the `storage_options` a namespace vended
for a table, so any scan relying on credential vending failed:
```
open Lance dataset failed: LanceError(IO): ...
Failed to get AWS credentials: CredentialsNotLoaded("no providers in chain provided credentials")
```
The options were re-encoded twice on the way to the BE, through one
five-entry S3-only table used in both directions:
```
namespace vends access_key_id
-> FE LanceStorageOptions.forBackend aws_access_key_id -> AWS_ACCESS_KEY
-> TFileScanRangeParams.properties
-> BE kStorageKeys AWS_ACCESS_KEY -> aws_access_key_id
-> lance-c
```
That table was written to *emit* one canonical spelling, which is
correct outbound. Reading it backwards turns it into a parser that
accepts only the spelling it happens to emit, so everything else was
dropped — credentials under any other accepted alias, and every non-S3
provider's keys, which left the catalog unable to use credential vending
outside S3 at all.
The failure was also split across the two halves, which made it hard to
read: the FE passes the vended map to the Lance Java SDK untouched, so
`SHOW TABLES` and `DESC` succeeded and only the scan failed.
The Lance Namespace specification describes `storage_options` as
configuration *"passed directly to Lance"*, so the protocol defines no
key vocabulary of its own and a client cannot assume one.
### Release note
Fixed a Lance REST catalog failing to scan with `CredentialsNotLoaded`
when it relied on credentials vended by the namespace rather than static
`s3.access_key` / `s3.secret_key`.
### What is changed and how it works?
Stop re-encoding server-supplied options, and resolve the ones Doris
does contribute per storage provider.
**Transport.** `TFileScanRangeParams.lance_storage_options` (new, id 38)
carries the options in Lance's own vocabulary. Set at ScanNode level —
the same pattern as `paimon_options` — so credentials are not serialized
once per fragment split. `properties` is deliberately not reused: it is
copied wholesale into `io::FileSystemProperties` for the shared
filesystem layer. The BE hands that map to lance-c as it arrives, so
`kStorageKeys` is gone. The `s3()` TVF path populates the same field,
including the schema-fetch RPC.
**The option vocabulary belongs to a provider, not to this layer.**
Lance routes a dataset to a provider by URL scheme, and the accepted
spellings overlap without agreeing:
```rust
// object_store-0.13.2 src/azure/builder.rs:448 — note: no aws_endpoint
"azure_storage_endpoint" | "azure_endpoint" | "endpoint" => Ok(Self::Endpoint)
// lance-io/src/object_store/providers/oss.rs:71-73, :87
("endpoint", &["oss_endpoint"]), ("access_key_id", &["oss_access_key_id"]), ...
"OSS endpoint is required. Please provide 'oss_endpoint' ..."
```
Rewriting a vended `endpoint` onto `aws_endpoint` therefore loses the
Azure endpoint outright and makes OSS fail before it reads anything.
`LanceStorageProvider` picks by scheme the way lance-io's own registry
does, and each implementation owns both directions of its vocabulary:
- **`LanceS3StorageProvider`** — the `s3` and `s3+ddb` schemes lance-io
registers for its AWS provider. It converts the catalog's typed storage
properties into the canonical `aws_*` spellings, and resolves vended
spellings onto those same keys. `token` is decidable here too: it is a
bearer token to object_store's Azure parser, and only the provider
settles which it is.
- **`LancePassThroughStorageProvider`** — everything else. Both halves
inert, so such a dataset is reachable only through what its namespace
vends, named as the namespace named it.
**Why the two sides have to be resolved against each other.** A catalog
with static `s3.access_key` whose namespace also vends credentials
produces both spellings of one option:
```
aws_access_key_id = <catalog> access_key_id = <vended>
```
Both parse to `AmazonS3ConfigKey::AccessKeyId`, and `as_s3_options()`
collects them into a `HashMap`, so the winner is whichever the iteration
yields last — decided separately for the access key and the secret, and
separately in the FE and in the BE. Two of the four combinations pair
one side's key with the other's secret, which surfaces as
`SignatureDoesNotMatch` rather than as a missing credential. Putting
both sides in the provider's vocabulary first means one key, one value,
and the namespace wins because it just described the table.
The alias set covers only the options Doris itself contributes — those
are the only ones a vended option can collide with.
`aws_endpoint_url_s3` is deliberately left alone: object_store parses it
into a config key of its own and prefers it over the generic endpoint,
so a vended one already wins, and folding it in would replace a defined
precedence with map order.
**Doris's own configuration is read through its typed properties**, not
through the flattened `AWS_*` backend map, which is only a re-encoding
of them (`AbstractS3CompatibleProperties.doBuildS3Configuration`). That
map also mixes in backend-only knobs Lance has no use for, and flattens
every configured storage into one namespace where two S3-compatible ones
overwrite each other. The typed list keeps them apart; the one to read
is chosen the way Iceberg chooses it
(`AbstractIcebergProperties.toFileIOProperties`) — prefer a concrete
provider over the generic `S3Properties`. Selection filters by type
rather than index, because `StorageProperties.createAll` prepends a
default HDFS entry when no `fs.xx.support` flag is set.
**What is rejected rather than resolved.** A NUL truncates a key at
lance-c's `CStr::from_ptr` while the FE keeps reading the whole thing,
so the FE rejects it and the BE returns `InvalidArgument`; dropping it
silently would only move the divergence to an FE that predates the
check. A namespace vending two spellings of one option with different
values is rejected too — picking one would be the coin toss above.
### Not in scope
**Credentials are static for the life of a scan.** `lance-c` opens
datasets with a static option set, so `expires_at_millis` reaches the BE
but is never acted on, and credentials that expire mid-scan are not
re-vended. Renewal needs a channel of its own; this field is not one,
and the thrift comment says so.
**One map, two Lances.** The FE and the BE agree on the options, not on
the library that reads them: both pin `object_store` 0.13.2, but the
Java SDK (9.1.0-beta.3) carries OpenDAL 0.57 and `lance-c` 0.1.6 carries
0.56. An option only the newer one knows — `skip_signature`, say — takes
effect on the FE and is ignored on the BE. This only reaches the OpenDAL
backend, which a namespace has to opt into by vending
`use_opendal=true`, and closing it means aligning the two upstream pins.
**Non-S3 providers are unblocked, not enabled.** Their vended options
now survive instead of being dropped or rewritten, but Doris contributes
no static configuration for them — it models OSS and COS perfectly well,
so writing that translation is possible, but it would mean committing to
a vocabulary per provider with no backend here to test it against, which
is how the rewriting bug above got in. A filesystem catalog's
`warehouse` is still restricted to `file` and `s3`.
**A catalog and a namespace that disagree.** If both name the same
option in spellings the provider treats as distinct config keys, both
survive and Lance decides. Only the spellings Doris itself emits are
resolved.
### Rolling upgrade
An FE upgraded ahead of the BEs no longer puts vended credentials into
`TFileScanRangeParams.properties`, and an older BE reads only that. A
REST catalog with no static credentials therefore cannot be scanned
until the BEs are upgraded too. The Lance catalog is not in a release
yet, so this only affects development clusters.
### How was this patch tested?
**Unit tests.** `LanceStorageOptionsTest` is new — `LanceStorageOptions`
had no test of its own, and the deleted `forBackend` had zero call sites
in any test, which is why the original bug went unnoticed. Its fixtures
are built with `StorageProperties.createAll` from user-facing
properties, so they exercise Doris's real parsing and have to satisfy
its rules. It covers the canonical mapping and anonymous access, vended
credentials superseding the catalog's, the full endpoint alias class,
`token` resolving differently on `s3://` than on `az://`, six non-S3
schemes arriving untouched, S3 credentials not leaking onto another
provider, selection skipping the non-S3 entry Doris prepends, unknown
options passing through, conflicting spellings and embedded NULs being
rejected. `LanceThriftContractTest` gains round-trip coverage of the new
field.
**Regression.** The docker stub only ever vended `aws_`-prefixed keys,
so the alias path had no coverage anywhere. It now serves a second
table, `all_types_unprefixed`, backed by the same dataset but vending
the unprefixed spelling, and `test_lance_rest_catalog` scans it with no
static credentials configured. Both tables scan correctly on the current
revision.
**End to end**, against Apache Gravitino 1.3.0's `lance-rest` service,
which vends the unprefixed spelling. This run validated the original
transport fix, on the first revision of this PR; the docker suite above
is what has been re-run since. Catalog with no `s3.access_key` /
`s3.secret_key`:
```sql
CREATE CATALOG lance_novend PROPERTIES (
"type" = "lance",
"lance.catalog.type" = "rest",
"lance.rest.uri" = "http://127.0.0.1:9101/lance",
"lance.namespace.parent" = "lance_catalog"
);
SELECT count(*), min(row_id), max(row_id), sum(row_id) FROM lance_novend.doris_probe.rest_probe;
-- 1034 1 1034 535095
```
against a 1034-row, 2-fragment dataset with an IVF_FLAT index. Predicate
pushdown and vector search both return correct results on the same
catalog, and a catalog with static credentials is unaffected. Before
this change the same statement failed with `CredentialsNotLoaded`.
https://claude.ai/code/session_01M3mYXBKShBonG6Lg3br4Ld
0 commit comments