[Bug](lance) REST catalog against Gravitino: namespace-vended credentials are dropped, and the root database is not listable #66772

Activity

1. Doris-Breakwater commented on Aug 14, 2026

   

   Doris-Breakwater

   on Aug 14, 2026

   Collaborator

   More actions

   Breakwater-GitHub-Analysis-Slot: slot_13a9e214821c

   This content is generated by AI for reference only.

   Initial assessment: both reported behaviors are supported by the code at branch-4.1 commit 0e53b31f586. Problem 1 is a confirmed Doris credential-conversion defect and blocks REST credential vending. Problem 2 is a confirmed Gravitino interoperability defect, although its fix needs to preserve REST servers that legitimately allow tables directly under the configured parent namespace. This is not evidence of data corruption or a protocol-version mismatch.

   The issue is currently open with no labels, assignee, milestone, or prior comments. It is unusually complete: the version, server version/configuration, REST response, error, reproduction, positive control, and workarounds are sufficient to start a fix. I did not run the full Doris/Gravitino deployment locally; the conclusions below come from the exact Doris source, the public Gravitino v1.3.0 source, and the reporter's controlled reproduction.

   Verified facts

   1. Vended credentials are lost only on the FE-to-BE conversion path.

      • LanceExternalCatalog.describeTable() requests credentials for REST catalogs.
      • loadTableMetadata() merges DescribeTableResponse.storage_options verbatim into the Java SDK options, explaining why FE metadata reads work.
      • The same method calls LanceStorageOptions.forBackend(), which recognizes only aws_access_key_id, aws_secret_access_key, aws_session_token, aws_endpoint, aws_region, and aws_virtual_hosted_style_request.
      • Gravitino v1.3.0's LancePropertiesUtils deliberately strips lance.storage. and therefore returns the configured unprefixed keys such as access_key_id, secret_access_key, endpoint, and region.
      • Unmatched vended values are absent from LanceTableMetadata; LanceScanNode.getLocationProperties() then sends only that reduced map to the BE. The BE lance_reader.cpp reconstructs Lance options only from normalized AWS_*/use_path_style properties, so an empty converted map reaches the credential provider. This exactly matches the reported FE-success/BE-failure split and the key-spelling positive control.

   2. The unusable root database and failing connectivity check follow directly from the current namespace mapping.

      • listDatabaseNames() always publishes rootDatabase; that name decodes to an empty relative namespace and therefore maps to the configured parent itself.
      • listTableNamesFromRemote() lists tables against that parent. With lance.namespace.parent = lance_catalog, Doris sends a one-level identifier.
      • Gravitino v1.3.0 accepts a one-level identifier for listNamespaces() (catalog -> schemas) but requires exactly two levels for listTables() (catalog + schema), producing the reported error.
      • checkWhenCreating() calls listTables(parent) before listNamespaces(parent), so test_connection=true fails before performing the valid namespace probe.
      • The in-repo REST fixture does not expose this incompatibility: it permits tables at the root and vends only aws_*-prefixed credentials.

   What remains a design decision

   • The parent namespace is table-listable on the existing fixture and may be table-listable on other Lance Namespace implementations, so globally removing the root database would be a compatibility regression. Doris should either detect parent table-list capability and omit only an unsupported root, or introduce an explicit mapping policy. Whichever approach is chosen, SHOW DATABASES should not advertise an entry that immediately fails SHOW TABLES.
   • The connectivity test should first use the operation valid at the configured parent (listNamespaces(parent)). If table-endpoint validation is retained, it should target a discovered child namespace when one exists rather than requiring tables to be legal directly under the parent.

   Recommended next steps

   1. Accept this as a branch-4.1 Lance REST catalog bug and add the repository's bug/component labels; the Lance FE code is not present on master at the reported/current revisions.
   2. Normalize both accepted S3 key families in one place before building BE properties. Preserve deterministic precedence when both spellings are returned, keep vended values ahead of static catalog credentials, and cover the full temporary-credential set (including the session token) plus endpoint/region/path-style aliases supported by the pinned Lance object-store layer.
   3. Add focused unit tests for LanceStorageOptions.forBackend() covering unprefixed keys, prefixed keys, conflicts, empty values, static fallback, and temporary credentials.
   4. Extend the REST regression fixture with an unprefixed vending response and prove a scan succeeds with no static access/secret key. Add a Gravitino-shaped namespace case where listNamespaces([catalog]) succeeds, listTables([catalog]) is invalid, and listTables([catalog, schema]) succeeds; verify both catalog creation with test_connection=true and that every displayed database is listable. Keep a root-table-listable case to prevent compatibility regressions.
   5. Document that lance.rest.uri may include the service base path (for example /lance). This is a documentation follow-up, not a blocker for the two code fixes.

   No additional reporter logs or profile are required to begin implementation. The reporter has offered to submit a PR; maintainers mainly need to choose the root-database compatibility policy above and review the alias precedence/test matrix.
2. Doris-Breakwater commented on Aug 14, 2026

   

   Doris-Breakwater

   on Aug 14, 2026

   Collaborator

   More actions

   Breakwater-GitHub-Analysis-Slot: slot_13a9e214821c

   Initial triage

   Judgment: valid, high-confidence branch-4.1 Lance REST integration bugs. There are two independent FE-side compatibility defects: credential alias normalization drops Gravitino-vended options on the BE path, and Doris assumes that the configured parent namespace is itself table-listable. Problem 1 blocks scans that rely on credential vending; Problem 2 creates an unusable advertised database and rejects an otherwise reachable catalog when test_connection=true. The issue currently has no labels, assignee, milestone, or earlier comments, and the reporter is willing to submit a PR.

   I checked the exact reported Doris commit (0e53b31f58674a4716f12cdab4c46f3cb607ed05) and the Apache Gravitino 1.3.0 tag (40fdf6ab96ac87b47e6d3e14e7c4dc0d815e68f0). I did not rerun the full MinIO/Gravitino fixture locally; the runtime A/B evidence below is reporter-supplied, while the failure mechanisms are source-confirmed.

   Verified facts

   1. Vended credentials are lost only on the FE-to-BE conversion path

   • Doris explicitly requests credentials in describeTable. It merges the returned storage_options unchanged into the Java SDK options, but separately calls LanceStorageOptions.forBackend(...) for the scan path (lines 288-313). This explains why FE metadata loading can succeed while the BE scan fails.
   • forBackend recognizes only aws_access_key_id, aws_secret_access_key, aws_session_token, aws_endpoint, aws_region, and aws_virtual_hosted_style_request. With no static credentials, Gravitino's unprefixed access_key_id, secret_access_key, endpoint, and region therefore contribute no BE properties.
   • The BE does not receive the raw vended map. LanceScanNode.getLocationProperties() returns the converted map, and LanceTableReader::_storage_options reads only Doris's normalized AWS_*/use_path_style keys. There is no later opportunity for the unprefixed values to recover.
   • Gravitino intentionally strips lance.storage. from catalog/table properties in LancePropertiesUtils, and its table-description path returns that resolved map as storage_options (source). Its tests explicitly expect access_key_id, endpoint, and region without aws_ prefixes.
   • The reporter's positive control—same Doris catalog and physical dataset, changing only the vended key spelling to aws_*, after which all 1024 rows scan—matches this exact branch in the code. This is not explained by the Java namespace library version difference.

   2. The root database and connectivity probe use an invalid Gravitino namespace depth

   • listDatabaseNames unconditionally adds rootDatabase. The root name maps back to an empty relative namespace, so listTableNamesFromRemote calls ListTables on the configured parent itself.
   • For lance.namespace.parent=lance_catalog, that request has one level. Gravitino's native backend accepts a one-level identifier for ListNamespaces and returns its schemas (source), but requires exactly two levels for ListTables (source). The reported error is therefore deterministic for this mapping.
   • checkWhenCreating calls ListTables(parent) before ListNamespaces(parent), so test_connection=true fails even though the namespace endpoint and the schema-level table endpoints are reachable.

   3. Existing tests model only the successful spellings and a permissive root

   • The regression stub vends only aws_* keys (source), while the scan regression deliberately has no static access/secret key. Thus the test exercises vending but not Gravitino's spelling.
   • The same stub allows ListTables at any namespace depth and places its default table directly at the empty/root namespace (source, list implementation). It therefore cannot reveal the fixed-depth Gravitino behavior.
   • There is no focused FE unit test for LanceStorageOptions.forBackend; current coverage tests only the static Doris-to-Java direction.

   Missing information / remaining decision

   No additional version, log, profile, or reproduction information is required to accept and prioritize these two defects. The supplied versions, REST response, SQL reproduction, BE error, workaround, and spelling-only positive control are sufficient. A maintainer-side end-to-end run is still useful validation, but it is not a reporter blocker.

   One implementation contract needs an explicit maintainer decision: generic Lance namespaces and the existing filesystem/stub behavior can have tables directly under the configured parent, while Gravitino's catalog/schema model cannot. Removing the root database unconditionally would regress the former; continuing to add it unconditionally leaves a phantom database for the latter. The fix should preserve both models rather than infer support solely from namespace depth.

   Recommended next steps

   1. Normalize vended aliases centrally. Extend forBackend (or a shared canonicalizer) to accept the unprefixed keys actually emitted by Gravitino as well as the existing aws_* forms, preserve the current precedence of vended values over static catalog defaults, and define deterministic precedence if both aliases are present. Cover endpoint/HTTP and addressing-style behavior, not only access/secret keys. Do not log credential values.
   2. Do not require the configured parent to be table-listable for connectivity. Probe ListNamespaces(parent) first; if a child namespace exists, ListTables can be tested against that table-capable child. An empty catalog should still pass a namespace connectivity check. Do not catch and suppress arbitrary table-list failures, since that would hide authentication and service errors.
   3. Make root exposure capability-aware or explicitly configurable. Include the root database only when tables directly under the configured parent are supported, or add a documented way to suppress it for catalog/schema backends. Keep the existing root behavior for filesystem/permissive namespaces. Whichever approach is chosen, every database returned by SHOW DATABASES must be listable.
   4. Add interoperability coverage. Add a unit test for forBackend using Gravitino's unprefixed map and alias precedence; add a regression-stub mode that vends unprefixed keys with no static credentials; and add a fixed-depth mode where ListNamespaces([catalog]) succeeds but ListTables([catalog]) fails while ListTables([catalog,schema]) succeeds. Retain a root-table case to prevent regressions in the generic model.
   5. Add a REST URI base-path case (for example /lance) to tests/docs, since the current URI validation already permits it but the root-mounted stub does not exercise it.

   Suggested triage: add the repository's bug and Lance/external-catalog ownership labels, and let the willing reporter prepare the PR once the root-namespace contract is agreed. The code is currently branch-4.1-specific, so the target branch should follow the maintainers' branch policy.
3. FANNG1 commented on Aug 15, 2026

   

   FANNG1

   on Aug 15, 2026

   Last edited by FANNG1

   ContributorAuthor

   More actions

   Correction to the fix guidance above

   Digging further into how Lance actually consumes storage_options, the suggestion in the issue body — teach LanceStorageOptions.forBackend to accept the unprefixed aliases — is necessary but not sufficient, and one of its premises was wrong. Everything below is verified against the exact Lance revision that lance-c v0.1.6 pins (f69324596), i.e. what the BE on branch-4.1 actually compiles, not a newer release.

   1. The outbound spelling is wrong too, so an inbound-only fix still breaks

   The BE emits aws_-prefixed option names (kStorageKeys in be/src/format_v2/table/lance_reader.cpp). Those silently stop working the moment Lance takes its OpenDAL path:

   // rust/lance-io/src/object_store/providers/aws.rs  —  build_opendal_s3_store
   let mut config_map: HashMap<String, String> = storage_options.0.clone();
   config_map.insert("bucket".to_string(), bucket);
   let operator = Operator::from_iter::<S3>(config_map)

   There is no alias normalization on this path — the map goes straight into OpenDAL's S3Config, whose field names are unprefixed, and OpenDAL's deserializer ignores unknown keys without error.

   	object_store path (default)	OpenDAL path (use_opendal=true)
   access_key_id	works	works
   aws_access_key_id	works	silently dropped

   To be precise about the current blast radius: use_opendal is itself read from the storage options (aws.rs:139-143), and the same five-key filter drops it, so this path is not reachable through Doris today — the prefixed spelling is a latent hazard rather than a live bug. It becomes live exactly when the filter is opened up, which is what any real fix to this issue does. That is why it belongs in the same change: widening the inbound aliases while continuing to emit the prefixed form outbound would make use_opendal=true reachable and silently credential-less at the same moment.

   The unprefixed short names are the only spelling both paths accept, so forJavaSdk and the BE's kStorageKeys are the place to converge.

   2. Both spellings have first-party sources — the real principle is that the consumer must be permissive

   The issue body implied Gravitino's unprefixed spelling is the more standard one. That is not accurate, and the correct argument is stronger.

   Lance's own credential vendor emits the prefixed form:

   // rust/lance-namespace-impls/src/credentials/aws.rs
   storage_options.insert("aws_access_key_id".to_string(), access_key_id);
   storage_options.insert("aws_session_token".to_string(), session_token);

   while Lance's own consumer accepts both, prefixed first:

   // rust/lance-io/src/object_store/dynamic_credentials.rs:172-186
   .get("aws_access_key_id").or_else(|| credentials.0.get("access_key_id"))
   .get("aws_session_token")
       .or_else(|| credentials.0.get("session_token"))
       .or_else(|| credentials.0.get("token"))

   So both spellings legitimately occur in the wild from first-party sources, and Lance's answer is: vendors emit one, consumers accept all of them. Doris is a consumer that accepts one. That is the defect, and it holds regardless of which namespace server is on the other end — this is not a Gravitino-specific quirk.

   Worth noting the spec is explicit that this field is opaque: "Configuration options to be used to access storage… These will be passed directly to Lance to initialize storage access." The namespace protocol deliberately does not define the key vocabulary, so a client cannot assume one.

   3. The bigger loss: the filter also discards Lance's credential auto-refresh (corrected)

   Corrected 2026-08-16 — the claim originally in this section was wrong. I wrote that
   expires_at_millis drives a credential refresh that the five-key filter silently opts Doris out
   of. It does not, on the path the BE actually uses.

   Lance does have that machinery (LanceNamespaceStorageOptionsProvider / StorageOptionsAccessor
   in rust/lance-io/src/object_store/storage_options.rs), but it is engaged by installing a
   provider through DatasetBuilder::with_storage_options_provider. lance-c only ever calls
   with_storage_options:

   // lance-c v0.1.6, src/dataset.rs
   builder = builder.with_storage_options(opts);

   which builds a static accessor with no provider (rust/lance/src/dataset/builder.rs:305-317), so
   nothing re-vends. as_s3_options then drops expires_at_millis outright, since it parses as
   neither an S3 nor a client config key.

   So: passing the key through is forward-looking, not load-bearing, and this is not an additional
   loss caused by the filter. The underlying limitation is real but belongs elsewhere — short-lived
   vended credentials do expire mid-scan on the BE, and fixing that needs lance-c to expose the
   provider API, not a change to this filter.

   The rest of this comment stands; items 1, 2 and 4 are unaffected.

   4. The filter also locks the Lance catalog to S3

   S3_KEYS (FE) and kStorageKeys (BE) are both S3-only, five entries each. Lance vends Azure and GCP credentials as well (azure_storage_sas_token, google_storage_token — credentials/azure.rs, credentials/gcp.rs). Against a namespace serving Lance tables on Azure or GCS, 100% of the vended credentials are dropped — not a spelling mismatch, simply no entry in the table. As written, the Lance catalog cannot support a non-S3 backend through credential vending at all.

   5. Revised suggestion

   Rather than widening the lookup table, stop translating server-supplied options at all:

   • FE — pass DescribeTable's storage_options through untranslated.
   • BE — hand them to lance-c as-is and drop the second translation. Note today's round trip is access_key_id → AWS_ACCESS_KEY → aws_access_key_id: Lance vocabulary out, Lance vocabulary back in, with a lossy filter in between and no information gained.
   • Static catalog properties (s3.access_key and friends) keep their conversion — they genuinely are in Doris vocabulary — but should emit the unprefixed spelling.

   On the safety concern that presumably motivated an allowlist in the first place: Lance's own model for exactly this problem is a denylist of location-controlling keys, not an allowlist of credential keys.

   // rust/lance-io/src/object_store/dynamic_opendal.rs
   protected_keys: Vec<&'static str>,   // stripped from server-supplied options

   That defends the thing actually worth defending — a namespace server redirecting where data is read from — while scaling to other clouds and to keys added later, which an allowlist cannot.

   If a smaller change is preferred for now, widening forBackend's aliases still fixes the immediate breakage and is worth doing on its own; it just leaves items 1, 3 and 4 open.

   Testing note

   The coverage gap noted in the issue body is a little wider than described: lance_rest_server.py vends only aws_-prefixed keys, so the unprefixed alias path is not exercised anywhere today.
4. FANNG1

   added a commit that references this issue on Aug 24, 2026

   [fix](lance) Resolve branch-4.1 conflicts for storage options

   ...

   This issue will close once commit 3b4e8f2 is merged into the 'master' branch.

   3b4e8f2
5. yiguolei

   added a commit that references this issue on Aug 25, 2026

   [fix](lance) Pass namespace-vended storage options through to the BE (#…

   ...

   This issue will close once commit 5442385 is merged into the 'master' branch.

   5442385