Skip to content

Commit 6d547c0

Browse files
morningmanclaude
andcommitted
[fix](fe) Keep sibling capabilities on iceberg-on-HMS tables pinned at a snapshot
### What problem does this PR solve? Issue Number: None Related PR: #69299, #69488 Problem Summary: An Iceberg table stored in HMS can be read through a dedicated Iceberg catalog or through an HMS catalog ('type'='hms'), whose embedded Iceberg connector builds the table's schema. fe-core resolves the table-scoped capabilities (Top-N lazy materialization, nested column pruning, field-id access paths, storage predicate pruning, nested column DDL, write-only defaults, auto analyze) from the catalog's connector or from the per-table set in the table's schema, never from the embedded connector. So the Hive connector copies the ones Iceberg declares into the per-table set of every schema it delegates, but it did that only in getTableSchema(session, handle), not in getTableSchema(session, handle, snapshot). fe-core uses the second one for explicit time travel and, since #69299, for every statement that reads an Iceberg table: the latest-snapshot pin retains its schema, and the table's capabilities are then read from that schema. On an iceberg-on-HMS table such a statement therefore saw none of the inherited capabilities. Reproduced on a local cluster with a standalone Hive 3.1.2 metastore, comparing the same statements on twin format-version 3 tables, one through each catalog: | Statement | Iceberg catalog | HMS catalog before | HMS catalog after | |---|---|---|---| | EXPLAIN ... ORDER BY id LIMIT 1 (latest, and FOR VERSION AS OF) | VMaterializeNode | no lazy materialization | VMaterializeNode | | EXPLAIN SELECT struct_element(s, 'a2') | all access paths: [s.a2] | no pruning | all access paths: [s.a2] | | UPDATE t SET c = 100 WHERE d = DEFAULT(d) | updates the rows with d = 42 | updates nothing | updates the rows with d = 42 | | MERGE ... ON t.id = s.id AND t.d = DEFAULT(t.d) | updates row 0 | updates nothing | updates row 0 | | DELETE FROM t WHERE d = DEFAULT(d) | error: cannot be used in a DELETE | deletes nothing | same error | | SELECT DEFAULT(d) FROM t | error: must reference the written table | NULL | same error | The snapshot overload now builds the delegated schema through the same helper as the latest-schema overload, which resolves the owning sibling once, asks it for the schema (at the pin, for the snapshot overload) and reflects the inherited capabilities onto it. A hudi-on-HMS table is unchanged: the Hudi connector declares no capabilities. ### Release note Fix iceberg-on-HMS tables read through an HMS catalog losing Iceberg optimizations (Top-N lazy materialization, nested column pruning) and write-default resolution of DEFAULT(column) in UPDATE / MERGE conditions. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - HiveConnectorMetadataSiblingDelegationTest: the per-table capability cases now check both getTableSchema overloads; a new case checks that the snapshot overload forwards the pin unchanged and resolves the owner once. All 49 fe-connector-hive test classes pass (502 tests). Three mutations (no reflection in the snapshot overload, asking the sibling for its latest schema, resolving the owner twice) each fail only their target cases. - New regression test_iceberg_on_hms_gateway_pinned_capabilities (hive3 docker). Run locally against a standalone Hive 3.1.2 metastore with a file:// warehouse (catalog properties adapted): fails on the HMS catalog's first Top-N check without this change, passes with it. - Behavior changed: Yes. Statements on iceberg-on-HMS tables read through an HMS catalog plan with the same capabilities as through an Iceberg catalog. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 4bc7f42 commit 6d547c0

4 files changed

Lines changed: 269 additions & 37 deletions

File tree

‎fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java‎

Lines changed: 24 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -534,12 +534,8 @@ public ConnectorTableSchema getTableSchema(
534534
// delegated schema as a per-table marker so it survives delegation (mirrors Trino table-redirection,
535535
// where the redirected-to connector's capabilities govern the table). Only the subset fe-core
536536
// actually resolves per-table is reflected; a catalog-wide capability (view / show-create / mvcc)
537-
// would be discarded by the reader, so emitting it would only record an unhonoured intent. Resolve
538-
// the owner ONCE (getMetadata is not free) and reuse it for the schema build and the capability read.
539-
SiblingOwner owner = siblingOwnerResolver.apply(handle);
540-
ConnectorTableSchema siblingSchema = memoizedSiblingMetadata(session, owner.connector(), owner.label())
541-
.getTableSchema(session, handle);
542-
return reflectSiblingCapabilities(owner.connector(), siblingSchema);
537+
// would be discarded by the reader, so emitting it would only record an unhonoured intent.
538+
return delegatedTableSchema(session, handle, sibling -> sibling.getTableSchema(session, handle));
543539
}
544540
HiveTableHandle hiveHandle = (HiveTableHandle) handle;
545541
String dbName = hiveHandle.getDbName();
@@ -626,6 +622,24 @@ public ConnectorTableSchema getTableSchema(
626622
ConnectorCapability.SUPPORTS_NESTED_COLUMN_SCHEMA_CHANGE,
627623
ConnectorCapability.SUPPORTS_WRITE_ONLY_DEFAULTS));
628624

625+
/**
626+
* A delegated (iceberg/hudi-on-HMS) table's schema as {@code buildSchema} asks its owning sibling for it,
627+
* with the capabilities the table inherits from that sibling reflected onto it
628+
* ({@link #reflectSiblingCapabilities}). Both {@code getTableSchema} overloads go through here because
629+
* fe-core reads a table's per-table capabilities from whichever of the two schemas the statement uses: a
630+
* statement that pins the table (explicit time travel, and every read of an iceberg table, whose
631+
* latest-snapshot pin retains its schema) uses the snapshot overload's, so the two must carry the same
632+
* capabilities. The owner is resolved ONCE (getMetadata is not free) and reused for the schema build and the
633+
* capability read.
634+
*/
635+
private ConnectorTableSchema delegatedTableSchema(ConnectorSession session, ConnectorTableHandle handle,
636+
Function<ConnectorMetadata, ConnectorTableSchema> buildSchema) {
637+
SiblingOwner owner = siblingOwnerResolver.apply(handle);
638+
ConnectorTableSchema siblingSchema = buildSchema.apply(
639+
memoizedSiblingMetadata(session, owner.connector(), owner.label()));
640+
return reflectSiblingCapabilities(owner.connector(), siblingSchema);
641+
}
642+
629643
/**
630644
* Reflects the {@link #SIBLING_INHERITABLE_CAPABILITIES} subset of the owning sibling connector's
631645
* connector-wide capability set onto a delegated (iceberg/hudi-on-HMS) table's schema as per-table
@@ -1490,7 +1504,10 @@ private static long lastDdlMillis(Map<String, String> parameters) {
14901504
public ConnectorTableSchema getTableSchema(ConnectorSession session, ConnectorTableHandle handle,
14911505
ConnectorMvccSnapshot snapshot) {
14921506
if (!(handle instanceof HiveTableHandle)) {
1493-
return siblingMetadata(session, handle).getTableSchema(session, handle, snapshot);
1507+
// The schema a pinned statement reads its per-table capabilities from, so it inherits exactly what
1508+
// the latest-schema overload above does.
1509+
return delegatedTableSchema(session, handle,
1510+
sibling -> sibling.getTableSchema(session, handle, snapshot));
14941511
}
14951512
// Hive has no schema-at-snapshot; the SPI default ignores the snapshot and returns the latest schema.
14961513
return getTableSchema(session, handle);

‎fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java‎

Lines changed: 88 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@
7070
import java.util.OptionalLong;
7171
import java.util.Set;
7272
import java.util.TreeSet;
73+
import java.util.concurrent.atomic.AtomicInteger;
7374
import java.util.function.BooleanSupplier;
7475
import java.util.function.Supplier;
7576
import java.util.regex.Matcher;
@@ -223,6 +224,9 @@ private static Path metadataSource() {
223224
elsewhere.put("icebergSiblingMetadata", "the iceberg by-type resolver, driven by HiveConnectorSiblingTest");
224225
elsewhere.put("hudiSiblingMetadata", "the hudi by-type resolver, the other half of the same pair");
225226
elsewhere.put("memoizedSiblingMetadata", "the per-session memo behind the two resolvers above");
227+
elsewhere.put("delegatedTableSchema",
228+
"the body both getTableSchema overloads share, driven through each of them by the"
229+
+ " capability-reflection cases here");
226230
elsewhere.put("getTableHandle",
227231
"the pivot that PRODUCES a foreign handle rather than consuming one, so it cannot be driven"
228232
+ " with the foreignHandle these cases use; covered by HiveConnectorThreeWayRoutingTest");
@@ -512,14 +516,33 @@ public ConnectorTransaction beginTransaction(ConnectorSession session) {
512516
Assertions.assertTrue(siblingMetadata.calls.isEmpty(), "the sibling must not be forwarded a hive handle");
513517
}
514518

519+
/** The pin of a statement that reads an iceberg table: its latest snapshot, with the schema retained. */
520+
private static final ConnectorMvccSnapshot PINNED_SNAPSHOT =
521+
ConnectorMvccSnapshot.builder().snapshotId(7L).schemaId(3L).retainSchema(true).build();
522+
523+
/**
524+
* The schema each getTableSchema overload returns for the foreign handle, keyed by the overload. fe-core reads
525+
* a table's per-table capabilities from the snapshot overload's schema in every statement that pins the table
526+
* (explicit time travel, and every read of an iceberg table, whose latest-snapshot pin retains its schema) and
527+
* from the latest-schema overload's otherwise, so what a delegated schema inherits must hold for both.
528+
*/
529+
private Map<String, ConnectorTableSchema> schemasFromBothOverloads(HiveConnectorMetadata md) {
530+
Map<String, ConnectorTableSchema> schemas = new LinkedHashMap<>();
531+
schemas.put("getTableSchema(session, handle)", md.getTableSchema(session, foreignHandle));
532+
schemas.put("getTableSchema(session, handle, snapshot)",
533+
md.getTableSchema(session, foreignHandle, PINNED_SNAPSHOT));
534+
return schemas;
535+
}
536+
515537
@Test
516538
public void foreignHandleSchemaReflectsSiblingScanCapabilitiesAsPerTableMarker() {
517539
// Option C: fe-core's PluginDrivenExternalTable.hasCapability reads only the CATALOG (hive) connector,
518540
// never the embedded sibling — so the hive gateway must reflect the sibling's connector-wide scan
519541
// capabilities onto the delegated schema as a per-table marker, or an iceberg-on-HMS table silently loses
520542
// auto-analyze / Top-N lazy / nested-column prune / storage predicate pruning (all declared connector-wide).
521543
// MUTATION: dropping the reflection -> the returned schema carries no marker -> the embedded table drops the
522-
// capabilities post-flip -> red here.
544+
// capabilities post-flip -> red here. Dropping it from the snapshot overload alone is red too: every
545+
// statement reading an iceberg-on-HMS table would then plan without them.
523546
Set<ConnectorCapability> siblingCaps = EnumSet.of(
524547
ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE,
525548
ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE,
@@ -532,24 +555,56 @@ public void foreignHandleSchemaReflectsSiblingScanCapabilitiesAsPerTableMarker()
532555
handle -> new SiblingOwner(new CapabilityDeclaringSiblingConnector(siblingCaps),
533556
SiblingOwner.ICEBERG_LABEL));
534557

535-
ConnectorTableSchema schema = md.getTableSchema(session, foreignHandle);
536-
Set<ConnectorCapability> reflected = schema.getTableCapabilities();
537-
Assertions.assertTrue(reflected.contains(ConnectorCapability.SUPPORTS_WRITE_ONLY_DEFAULTS),
538-
"an iceberg-on-HMS table must inherit write-only defaults, or DEFAULT(column) on it answers "
539-
+ "NULL from the read schema");
540-
Assertions.assertTrue(reflected.contains(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE),
541-
"auto-analyze must survive the delegation as a per-table capability");
542-
Assertions.assertTrue(reflected.contains(ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE),
543-
"Top-N lazy must survive the delegation as a per-table capability");
544-
Assertions.assertTrue(reflected.contains(ConnectorCapability.SUPPORTS_NESTED_COLUMN_PRUNE),
545-
"nested-column prune must survive the delegation as a per-table capability");
546-
Assertions.assertTrue(reflected.contains(ConnectorCapability.SUPPORTS_FIELD_ID_ACCESS_PATH),
547-
"an iceberg-on-HMS table must inherit the field-id access path capability, or its nested "
548-
+ "access paths stay name-based while BE matches them by iceberg field id");
549-
Assertions.assertTrue(reflected.contains(ConnectorCapability.SUPPORTS_STORAGE_PREDICATE_PRUNING),
550-
"storage predicate pruning must survive the delegation as a per-table capability");
551-
Assertions.assertEquals("sibling-generation", schema.getWriteMetadataIdentity(),
552-
"capability reflection must not discard the sibling's write-generation fence");
558+
for (Map.Entry<String, ConnectorTableSchema> entry : schemasFromBothOverloads(md).entrySet()) {
559+
String overload = entry.getKey();
560+
ConnectorTableSchema schema = entry.getValue();
561+
Set<ConnectorCapability> reflected = schema.getTableCapabilities();
562+
Assertions.assertTrue(reflected.contains(ConnectorCapability.SUPPORTS_WRITE_ONLY_DEFAULTS),
563+
overload + ": an iceberg-on-HMS table must inherit write-only defaults, or DEFAULT(column) on "
564+
+ "it answers NULL from the read schema");
565+
Assertions.assertTrue(reflected.contains(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE),
566+
overload + ": auto-analyze must survive the delegation as a per-table capability");
567+
Assertions.assertTrue(reflected.contains(ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE),
568+
overload + ": Top-N lazy must survive the delegation as a per-table capability");
569+
Assertions.assertTrue(reflected.contains(ConnectorCapability.SUPPORTS_NESTED_COLUMN_PRUNE),
570+
overload + ": nested-column prune must survive the delegation as a per-table capability");
571+
Assertions.assertTrue(reflected.contains(ConnectorCapability.SUPPORTS_FIELD_ID_ACCESS_PATH),
572+
overload + ": an iceberg-on-HMS table must inherit the field-id access path capability, or "
573+
+ "its nested access paths stay name-based while BE matches them by iceberg field id");
574+
Assertions.assertTrue(reflected.contains(ConnectorCapability.SUPPORTS_STORAGE_PREDICATE_PRUNING),
575+
overload + ": storage predicate pruning must survive the delegation as a per-table capability");
576+
Assertions.assertEquals("sibling-generation", schema.getWriteMetadataIdentity(),
577+
overload + ": capability reflection must not discard the sibling's write-generation fence");
578+
}
579+
}
580+
581+
@Test
582+
public void snapshotOverloadForwardsThePinAndResolvesTheOwnerOnce() {
583+
// The snapshot overload must hand the sibling the statement's own pin (a schema at another snapshot would
584+
// describe other columns) and, like the latest-schema overload, resolve the owning sibling once for both
585+
// the schema build and the capability read.
586+
// MUTATION: asking the sibling for its latest schema instead -> the pin never reaches it -> red; resolving
587+
// the owner a second time for the capability read -> red on the count.
588+
CapabilityDeclaringSiblingConnector sibling = new CapabilityDeclaringSiblingConnector(
589+
EnumSet.of(ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE));
590+
AtomicInteger resolutions = new AtomicInteger();
591+
HiveConnectorMetadata md = new HiveConnectorMetadata(null, HiveTestProperties.minimal(), new FakeConnectorContext(),
592+
SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED,
593+
handle -> {
594+
resolutions.incrementAndGet();
595+
return new SiblingOwner(sibling, SiblingOwner.ICEBERG_LABEL);
596+
});
597+
598+
ConnectorTableSchema schema = md.getTableSchema(session, foreignHandle, PINNED_SNAPSHOT);
599+
600+
Assertions.assertEquals(Collections.singletonList("getTableSchemaAtSnapshot"), sibling.metadata.calls,
601+
"the snapshot overload must ask the sibling for its schema at the pin, not its latest schema");
602+
Assertions.assertSame(PINNED_SNAPSHOT, sibling.metadata.lastSchemaSnapshot,
603+
"the sibling must receive the statement's pin unchanged");
604+
Assertions.assertEquals(EnumSet.of(ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE),
605+
schema.getTableCapabilities(), "the schema at the pin inherits the sibling's capability");
606+
Assertions.assertEquals(1, resolutions.get(),
607+
"the owning sibling is resolved once for the schema build and the capability read");
553608
}
554609

555610
@Test
@@ -570,20 +625,20 @@ public void foreignHandleSchemaReflectsOnlyThePerTableResolvedCapabilitySubset()
570625
handle -> new SiblingOwner(new CapabilityDeclaringSiblingConnector(siblingCaps),
571626
SiblingOwner.ICEBERG_LABEL));
572627

573-
ConnectorTableSchema schema = md.getTableSchema(session, foreignHandle);
574-
Assertions.assertEquals(EnumSet.of(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE),
575-
schema.getTableCapabilities(),
576-
"only the per-table-resolved subset may be reflected onto a delegated table");
628+
schemasFromBothOverloads(md).forEach((overload, schema) -> Assertions.assertEquals(
629+
EnumSet.of(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE), schema.getTableCapabilities(),
630+
overload + ": only the per-table-resolved subset may be reflected onto a delegated table"));
577631

578632
// A NON-empty sibling declaring none of the subset must inherit nothing.
579633
HiveConnectorMetadata noneInheritable = new HiveConnectorMetadata(null, HiveTestProperties.minimal(),
580634
new FakeConnectorContext(), SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED,
581635
handle -> new SiblingOwner(new CapabilityDeclaringSiblingConnector(
582636
EnumSet.of(ConnectorCapability.SUPPORTS_VIEW, ConnectorCapability.SUPPORTS_MVCC_SNAPSHOT)),
583637
SiblingOwner.ICEBERG_LABEL));
584-
Assertions.assertTrue(noneInheritable.getTableSchema(session, foreignHandle)
585-
.getTableCapabilities().isEmpty(),
586-
"a sibling declaring only catalog-wide capabilities must contribute no per-table capability");
638+
schemasFromBothOverloads(noneInheritable).forEach((overload, schema) -> Assertions.assertTrue(
639+
schema.getTableCapabilities().isEmpty(),
640+
overload + ": a sibling declaring only catalog-wide capabilities must contribute no per-table "
641+
+ "capability"));
587642
}
588643

589644
@Test
@@ -594,9 +649,9 @@ public void foreignHandleSchemaUnchangedWhenSiblingDeclaresNoInheritableCapabili
594649
// foreignHandleSchemaReflectsOnlyThePerTableResolvedCapabilitySubset above.
595650
// MUTATION: dropping the isEmpty() early-return and rebuilding the schema unconditionally -> red here.
596651
HiveConnectorMetadata md = withSibling(); // RecordingSiblingConnector declares no capabilities
597-
ConnectorTableSchema schema = md.getTableSchema(session, foreignHandle);
598-
Assertions.assertTrue(schema.getTableCapabilities().isEmpty(),
599-
"nothing inherited when the sibling declares no capabilities");
652+
schemasFromBothOverloads(md).forEach((overload, schema) -> Assertions.assertTrue(
653+
schema.getTableCapabilities().isEmpty(),
654+
overload + ": nothing inherited when the sibling declares no capabilities"));
600655
}
601656

602657
// ============== per-statement sibling-metadata funnel (HMS heterogeneous gateway) ==============
@@ -719,6 +774,7 @@ public ConnectorMetadata getMetadata(ConnectorSession session) {
719774
/** A sibling {@link Connector} declaring a fixed capability set; its metadata returns a marker-less schema. */
720775
private static final class CapabilityDeclaringSiblingConnector implements Connector {
721776
private final Set<ConnectorCapability> caps;
777+
private final RecordingSiblingMetadata metadata = new RecordingSiblingMetadata();
722778

723779
CapabilityDeclaringSiblingConnector(Set<ConnectorCapability> caps) {
724780
this.caps = caps;
@@ -731,7 +787,7 @@ public Set<ConnectorCapability> getCapabilities() {
731787

732788
@Override
733789
public ConnectorMetadata getMetadata(ConnectorSession session) {
734-
return new RecordingSiblingMetadata();
790+
return metadata;
735791
}
736792
}
737793

@@ -772,6 +828,7 @@ private static final class RecordingSiblingMetadata implements ConnectorMetadata
772828
"deleteUsingMetadata"));
773829

774830
final List<String> calls = new ArrayList<>();
831+
ConnectorMvccSnapshot lastSchemaSnapshot;
775832
ConnectorExpression lastDeletePredicate;
776833
String lastBoundWriteMetadataIdentity;
777834
BooleanSupplier lastCancellation;
@@ -789,6 +846,7 @@ public ConnectorTableSchema getTableSchema(ConnectorSession session, ConnectorTa
789846
public ConnectorTableSchema getTableSchema(ConnectorSession session, ConnectorTableHandle handle,
790847
ConnectorMvccSnapshot snapshot) {
791848
calls.add("getTableSchemaAtSnapshot");
849+
lastSchemaSnapshot = snapshot;
792850
return new ConnectorTableSchema("sibling", Collections.emptyList(), "iceberg", Collections.emptyMap(),
793851
Collections.emptySet(), "sibling-generation");
794852
}
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !iceberg_renamed_nested --
3+
0 1 x
4+
1 2 y
5+
2 3 z
6+
7+
-- !iceberg_renamed_nested_topn --
8+
0 {"a2":1, "b":"x"}
9+
1 {"a2":2, "b":"y"}
10+
11+
-- !iceberg_update_where_default --
12+
0 100 42
13+
1 100 42
14+
2 2 7
15+
16+
-- !iceberg_merge_on_default --
17+
0 300 42
18+
1 100 42
19+
2 2 7
20+
21+
-- !gateway_renamed_nested --
22+
0 1 x
23+
1 2 y
24+
2 3 z
25+
26+
-- !gateway_renamed_nested_topn --
27+
0 {"a2":1, "b":"x"}
28+
1 {"a2":2, "b":"y"}
29+
30+
-- !gateway_update_where_default --
31+
0 100 42
32+
1 100 42
33+
2 2 7
34+
35+
-- !gateway_merge_on_default --
36+
0 300 42
37+
1 100 42
38+
2 2 7
39+

0 commit comments

Comments
 (0)