You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6d547c0
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](fe) Keep sibling capabilities on iceberg-on-HMS tables pinned at a snapshot
### What problem does this PR solve?
Issue Number: None
Related PR: #69299, #69488
Problem Summary:
An Iceberg table stored in HMS can be read through a dedicated Iceberg
catalog or through an HMS catalog ('type'='hms'), whose embedded Iceberg
connector builds the table's schema. fe-core resolves the table-scoped
capabilities (Top-N lazy materialization, nested column pruning, field-id
access paths, storage predicate pruning, nested column DDL, write-only
defaults, auto analyze) from the catalog's connector or from the per-table
set in the table's schema, never from the embedded connector. So the Hive
connector copies the ones Iceberg declares into the per-table set of every
schema it delegates, but it did that only in getTableSchema(session,
handle), not in getTableSchema(session, handle, snapshot).
fe-core uses the second one for explicit time travel and, since #69299,
for every statement that reads an Iceberg table: the latest-snapshot pin
retains its schema, and the table's capabilities are then read from that
schema. On an iceberg-on-HMS table such a statement therefore saw none of
the inherited capabilities. Reproduced on a local cluster with a
standalone Hive 3.1.2 metastore, comparing the same statements on twin
format-version 3 tables, one through each catalog:
| Statement | Iceberg catalog | HMS catalog before | HMS catalog after |
|---|---|---|---|
| EXPLAIN ... ORDER BY id LIMIT 1 (latest, and FOR VERSION AS OF) | VMaterializeNode | no lazy materialization | VMaterializeNode |
| EXPLAIN SELECT struct_element(s, 'a2') | all access paths: [s.a2] | no pruning | all access paths: [s.a2] |
| UPDATE t SET c = 100 WHERE d = DEFAULT(d) | updates the rows with d = 42 | updates nothing | updates the rows with d = 42 |
| MERGE ... ON t.id = s.id AND t.d = DEFAULT(t.d) | updates row 0 | updates nothing | updates row 0 |
| DELETE FROM t WHERE d = DEFAULT(d) | error: cannot be used in a DELETE | deletes nothing | same error |
| SELECT DEFAULT(d) FROM t | error: must reference the written table | NULL | same error |
The snapshot overload now builds the delegated schema through the same
helper as the latest-schema overload, which resolves the owning sibling
once, asks it for the schema (at the pin, for the snapshot overload) and
reflects the inherited capabilities onto it. A hudi-on-HMS table is
unchanged: the Hudi connector declares no capabilities.
### Release note
Fix iceberg-on-HMS tables read through an HMS catalog losing Iceberg
optimizations (Top-N lazy materialization, nested column pruning) and
write-default resolution of DEFAULT(column) in UPDATE / MERGE conditions.
### Check List (For Author)
- Test: Unit Test / Regression test / Manual test
- HiveConnectorMetadataSiblingDelegationTest: the per-table capability
cases now check both getTableSchema overloads; a new case checks that
the snapshot overload forwards the pin unchanged and resolves the
owner once. All 49 fe-connector-hive test classes pass (502 tests).
Three mutations (no reflection in the snapshot overload, asking the
sibling for its latest schema, resolving the owner twice) each fail
only their target cases.
- New regression test_iceberg_on_hms_gateway_pinned_capabilities (hive3
docker). Run locally against a standalone Hive 3.1.2 metastore with a
file:// warehouse (catalog properties adapted): fails on the HMS
catalog's first Top-N check without this change, passes with it.
- Behavior changed: Yes. Statements on iceberg-on-HMS tables read through
an HMS catalog plan with the same capabilities as through an Iceberg
catalog.
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java
+24-7Lines changed: 24 additions & 7 deletions
Original file line number
Diff line number
Diff line change
@@ -534,12 +534,8 @@ public ConnectorTableSchema getTableSchema(
534
534
// delegated schema as a per-table marker so it survives delegation (mirrors Trino table-redirection,
535
535
// where the redirected-to connector's capabilities govern the table). Only the subset fe-core
536
536
// actually resolves per-table is reflected; a catalog-wide capability (view / show-create / mvcc)
537
-
// would be discarded by the reader, so emitting it would only record an unhonoured intent. Resolve
538
-
// the owner ONCE (getMetadata is not free) and reuse it for the schema build and the capability read.
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java
0 commit comments