Skip to content

Commit a45d9bf

Browse files
committed
[improvement](fe) Version Iceberg partition views and SQL cache metadata
### What problem does this PR solve? Issue Number: None Problem Summary: Ordinary Iceberg scans and SQL result-cache validation materialize and copy the full Doris partition view even when native file pruning or a pinned metadata version is sufficient. Cache immutable engine partition views and their pruning index by connector metadata generation, defer conversion for native planning, and obtain SQL cache versions through a source-neutral version provider. Record distinct current, historical and branch/tag references when a statement reads several versions of one table, and revalidate them through the existing snapshot resolver. Reuse statement-retained table handles and combine relation-name discovery through the connector SPI to avoid repeated remote existence and listing requests. Preserve MV/partition metadata consumers, refresh invalidation and frozen-reader schema consistency. ### Release note Iceberg ordinary scans use native file pruning by default. The session variable enable_iceberg_internal_partition_pruning enables Doris internal partition pruning when needed. SQL result caching supports independently versioned Iceberg snapshot and branch/tag references without enumerating partitions. ### Check List (For Author) - Test: Unit Test - run-fe-ut.sh: 325 targeted metadata, MVCC, partition view, SQL cache and dependent test cases passed - run-fe-ut.sh: all 85 connector SPI tests passed separately - build.sh --fe -j4 passed, including Checkstyle and connector architecture gates - Behavior changed: Yes, defer ordinary Iceberg partition conversion by default and support version-aware external SQL caching - Does this need documentation: Yes, updated fe/fe-connector/README.md ### Performance validation Measured independently against unoptimized master `e0bf4874070`, with identical Linux AArch64 Release BE, data/footer prewarming, and session settings. Numbers are complete SQL medians, in milliseconds. The baseline column shows two separate baseline runs. | Scenario | Baseline runs | This PR | |---|---:|---:| | 120,000 partitions, warm LIMIT 10 | 389.38–463.29 | 9.69 | | 120,000 partitions, warm EXPLAIN | 188.19–189.07 | 3.45 | | 120,000 partitions, warm LIMIT 10 with internal pruning enabled | 333.18–391.80 | 9.76 | | 1,000 manifests / 10,000 files, cold LIMIT 10 | 925.44–933.77 | 26.50 | The large-partition protocol enables SQL Cache validation but varies an always-true predicate to avoid result-cache hits. Ordinary LIMIT cases have 40 measured samples; EXPLAIN has 20. Baseline EXPLAIN retention is bounded by using equivalent SELECT warmup, identically for all variants. Function-heavy partition predicates remain approximately 5.5 seconds in this fixture and are not claimed as a benefit. ### Additional validation - Live SQL Cache checks: 170 assertions for current reads, snapshots, timestamps, branch/tag movement, multiple versions of one table, schema evolution/rename, snapshot expiry and ref removal/recreation. - The final combined tree passed a clean FE build, 997 scoped unit tests without skips, and the Hive streaming / Iceberg scheduled-split / table-handle-reuse regression suites.
1 parent e0bf487 commit a45d9bf

61 files changed

Lines changed: 2937 additions & 222 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎fe/fe-connector/README.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -311,6 +311,42 @@ String, convert inside `of()`, log a warning) explicitly.
311311

312312
`fe-connector-adbc` is the reference implementation of both classes.
313313

314+
## Snapshot Metadata and Partition Views
315+
316+
The engine must pin a table read before binding columns or building scan ranges.
317+
Derived metadata caches must describe that same read, including the selected
318+
snapshot/ref, schema and partition-spec generation. Never obtain a newer table
319+
independently to build one part of an already pinned read.
320+
321+
Connectors can supply `getPartitionViewVersion` as a cheap generation key for
322+
an immutable partition view and its engine pruning index. Derive this key from
323+
the pinned metadata; do not enumerate or hash every partition to calculate it.
324+
A metadata refresh invalidates the corresponding engine cache, while in-flight
325+
readers retain their existing view. Query-specific conversions must not mutate
326+
objects shared by the connector cache.
327+
328+
A connector declaring `supportsNativePartitionPlanning` can defer the engine
329+
partition view until a consumer actually requires internal partition semantics.
330+
For Iceberg, `enable_iceberg_internal_partition_pruning` defaults to `false`:
331+
ordinary scans use native manifest/file pruning. Setting it to `true` enables
332+
Doris partition pruning. MV planning and partition metadata rules still request
333+
the full view; deferred metadata must never be treated as an empty table.
334+
335+
`getSqlCacheVersion` supplies a lightweight result-cache version for a pinned
336+
read without loading its partitions. The version must distinguish table
337+
identity and all metadata that can change the query result or bound schema.
338+
The engine records each table reference separately, so reading the same table
339+
through current, historical and branch/tag references does not collapse their
340+
cache dependencies. Cache validation resolves those original references again:
341+
a moved or removed ref, changed schema, expired snapshot or recreated table
342+
must not reuse an incompatible cached answer.
343+
344+
Within a statement, reuse the frozen table and its resource lease through
345+
`ConnectorStatementScope`. Its read-only lookup must not load metadata on a
346+
miss. Relation existence and name listing can reuse this metadata through the
347+
SPI without source-specific checks in fe-core. This does not extend cache TTLs
348+
or authorize cross-user reuse of metadata.
349+
314350
## Testing and Verification
315351

316352
- **Unit tests** live in each module; build/test recipes are in `AGENTS.md`.

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@
5353
import java.io.IOException;
5454
import java.util.ArrayList;
5555
import java.util.Collections;
56+
import java.util.LinkedHashSet;
5657
import java.util.List;
5758
import java.util.Locale;
5859
import java.util.Map;
@@ -94,6 +95,13 @@ public interface IcebergCatalogOps {
9495
/** Lists the view names in {@code dbName}; empty when the catalog is not a (view-enabled) ViewCatalog. */
9596
List<String> listViewNames(String dbName);
9697

98+
/** Lists relations without requiring callers to subtract and then re-add view names. */
99+
default List<String> listRelationNames(String dbName) {
100+
LinkedHashSet<String> names = new LinkedHashSet<>(listTableNames(dbName));
101+
names.addAll(listViewNames(dbName));
102+
return new ArrayList<>(names);
103+
}
104+
97105
/** Returns {@code true} iff {@code dbName.tableName} exists. */
98106
boolean tableExists(String dbName, String tableName);
99107

@@ -392,6 +400,15 @@ public List<String> listViewNames(String dbName) {
392400
.collect(Collectors.toList());
393401
}
394402

403+
@Override
404+
public List<String> listRelationNames(String dbName) {
405+
LinkedHashSet<String> names = catalog.listTables(toNamespace(dbName)).stream()
406+
.map(TableIdentifier::name)
407+
.collect(Collectors.toCollection(LinkedHashSet::new));
408+
names.addAll(listViewNames(dbName));
409+
return new ArrayList<>(names);
410+
}
411+
395412
@Override
396413
public boolean tableExists(String dbName, String tableName) {
397414
return catalog.tableExists(toTableIdentifier(dbName, tableName));

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java‎

Lines changed: 126 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
import org.apache.doris.connector.spi.ConnectorMetadata;
2626
import org.apache.doris.connector.spi.ConnectorPartitionInfo;
2727
import org.apache.doris.connector.spi.ConnectorSession;
28+
import org.apache.doris.connector.spi.ConnectorStatementScope;
2829
import org.apache.doris.connector.spi.ConnectorStorageContext;
2930
import org.apache.doris.connector.spi.ConnectorTableSchema;
3031
import org.apache.doris.connector.spi.ConnectorTableStatistics;
@@ -45,6 +46,7 @@
4546
import org.apache.doris.connector.spi.handle.WriteOperation;
4647
import org.apache.doris.connector.spi.mvcc.ConnectorMvccPartitionView;
4748
import org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot;
49+
import org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion;
4850
import org.apache.doris.connector.spi.mvcc.ConnectorTimeTravelSpec;
4951
import org.apache.doris.connector.spi.pushdown.ConnectorExpression;
5052
import org.apache.doris.thrift.THiveTable;
@@ -64,6 +66,7 @@
6466
import org.apache.iceberg.SnapshotRef;
6567
import org.apache.iceberg.SortOrder;
6668
import org.apache.iceberg.Table;
69+
import org.apache.iceberg.TableMetadata;
6770
import org.apache.iceberg.TableProperties;
6871
import org.apache.iceberg.catalog.TableIdentifier;
6972
import org.apache.iceberg.exceptions.NoSuchNamespaceException;
@@ -311,6 +314,17 @@ public List<String> listTableNames(ConnectorSession session, String dbName) {
311314
}
312315
}
313316

317+
@Override
318+
public List<String> listRelationNames(ConnectorSession session, String dbName) {
319+
try {
320+
return executeAuthenticated(() -> catalogOps.listRelationNames(dbName));
321+
} catch (RuntimeException e) {
322+
throw e;
323+
} catch (Exception e) {
324+
throw new RuntimeException("Failed to list relation names, error message is: " + e.getMessage(), e);
325+
}
326+
}
327+
314328
@Override
315329
public List<String> listViewNames(ConnectorSession session, String dbName) {
316330
// Mirror legacy IcebergMetadataOps.listViewNames: wrap in the auth context; a RuntimeException
@@ -424,26 +438,56 @@ private String loadTableComment(ConnectorSession session, String dbName, String
424438
@Override
425439
public Optional<ConnectorTableHandle> getTableHandle(
426440
ConnectorSession session, String dbName, String tableName) {
441+
IcebergTableHandle handle = new IcebergTableHandle(dbName, tableName);
442+
if (IcebergStatementScope.hasReadTable(session, dbName, tableName)
443+
|| (tableCache != null && tableCache.contains(TableIdentifier.of(dbName, tableName)))) {
444+
return Optional.of(handle);
445+
}
446+
// Offline callers cannot retain a cache lease. Pure existence checks use tableExists instead.
447+
if (session == null || session.getStatementScope() == ConnectorStatementScope.NONE) {
448+
return tableExists(session, dbName, tableName) ? Optional.of(handle) : Optional.empty();
449+
}
450+
try {
451+
// Loading proves existence and pins the same metadata used by schema/snapshot/scan planning.
452+
// Borrow through the existing generation-aware cache so concurrent cold readers share one load.
453+
executeAuthenticated(() -> resolveTableForRead(session, handle));
454+
return Optional.of(handle);
455+
} catch (Exception e) {
456+
if (ExceptionUtils.getThrowableList(e).stream().anyMatch(NoSuchTableException.class::isInstance)) {
457+
return Optional.empty();
458+
}
459+
throw IcebergExceptionUtils.wrapTableLoadFailure(handle, e,
460+
"Failed to load table, error message is:");
461+
}
462+
}
463+
464+
@Override
465+
public boolean tableExists(
466+
ConnectorSession session, String dbName, String tableName) {
467+
// Reuse the statement's frozen read before consulting the shared cache: REFRESH must not
468+
// make an in-flight reader re-check a different table generation. A shared cache is absent
469+
// for credential-dependent catalogs; their statement-local proof never crosses identities.
470+
if (IcebergStatementScope.hasReadTable(session, dbName, tableName)
471+
|| (tableCache != null && tableCache.contains(TableIdentifier.of(dbName, tableName)))) {
472+
return true;
473+
}
427474
// Mirror legacy IcebergMetadataOps.tableExist: wrap the remote existence check in the auth context
428-
// (the handle build below is pure — no remote call).
475+
// without loading and pinning a table merely to answer a DDL existence check.
429476
boolean exists;
430477
try {
431478
exists = executeAuthenticated(() -> catalogOps.tableExists(dbName, tableName));
432479
} catch (Exception e) {
433480
// Preserve Optional.empty even when an auth/catalog layer wraps NoSuchTableException.
434481
if (ExceptionUtils.getThrowableList(e).stream()
435482
.anyMatch(NoSuchTableException.class::isInstance)) {
436-
return Optional.empty();
483+
return false;
437484
}
438485
// Existence checks in several catalogs load metadata internally, so normalize them at the shared
439486
// handle boundary just like explicit table loads.
440487
throw IcebergExceptionUtils.wrapTableLoadFailure(new IcebergTableHandle(dbName, tableName), e,
441488
"Failed to check table exist, error message is:");
442489
}
443-
if (!exists) {
444-
return Optional.empty();
445-
}
446-
return Optional.of(new IcebergTableHandle(dbName, tableName));
490+
return exists;
447491
}
448492

449493
@Override
@@ -1998,6 +2042,32 @@ public boolean supportsCastPredicatePushdown(ConnectorSession session) {
19982042
return true;
19992043
}
20002044

2045+
@Override
2046+
public Optional<ConnectorSqlCacheVersion> getSqlCacheVersion(
2047+
ConnectorSession session, ConnectorTableHandle handle) {
2048+
IcebergTableHandle iceHandle = (IcebergTableHandle) handle;
2049+
try {
2050+
return executeAuthenticated(() -> {
2051+
Table table = resolveTableForRead(session, iceHandle);
2052+
String version = metadataVersion(table, iceHandle).orElseThrow(
2053+
() -> new UnsupportedOperationException("Iceberg SQL cache requires published table metadata"));
2054+
long updateMillis = iceHandle.getSnapshotId() < 0 ? 0
2055+
: table.snapshot(iceHandle.getSnapshotId()).timestampMillis();
2056+
return Optional.of(new ConnectorSqlCacheVersion(version, updateMillis));
2057+
});
2058+
} catch (Exception e) {
2059+
throw IcebergExceptionUtils.wrapTableLoadFailure(iceHandle, e,
2060+
"Failed to get iceberg SQL cache version, error message is:");
2061+
}
2062+
}
2063+
2064+
@Override
2065+
public boolean supportsNativePartitionPlanning(ConnectorSession session, ConnectorTableHandle handle) {
2066+
return !((IcebergTableHandle) handle).isSystemTable()
2067+
&& (session == null || !Boolean.parseBoolean(session.getSessionProperties()
2068+
.getOrDefault("enable_iceberg_internal_partition_pruning", "false")));
2069+
}
2070+
20012071
// ========== B-2: partition enumeration (MTMV RANGE view + SHOW PARTITIONS) ==========
20022072

20032073
/**
@@ -2102,6 +2172,39 @@ public List<String> listPartitionNames(ConnectorSession session, ConnectorTableH
21022172
}
21032173
}
21042174

2175+
private static Optional<String> metadataVersion(Table table, IcebergTableHandle handle) {
2176+
// Custom catalog tables and unpublished metadata cannot supply a stable metadata generation.
2177+
if (!(table instanceof BaseTable)) {
2178+
return Optional.empty();
2179+
}
2180+
TableMetadata metadata = ((BaseTable) table).operations().current();
2181+
String location = metadata.metadataFileLocation();
2182+
if (location == null) {
2183+
return Optional.empty();
2184+
}
2185+
return Optional.of(metadata.uuid() + ":" + handle.getSnapshotId() + ":"
2186+
+ handle.getSchemaId() + ":" + location);
2187+
}
2188+
2189+
@Override
2190+
public Optional<String> getPartitionViewVersion(ConnectorSession session, ConnectorTableHandle handle) {
2191+
IcebergTableHandle iceHandle = (IcebergTableHandle) handle;
2192+
if (!iceHandle.hasSnapshotPin() || session == null
2193+
|| session.getStatementScope() == ConnectorStatementScope.NONE) {
2194+
// Without a statement-frozen table, metadata could change between version lookup and listing.
2195+
return Optional.empty();
2196+
}
2197+
try {
2198+
return executeAuthenticated(() -> {
2199+
Table table = resolveTableForRead(session, iceHandle);
2200+
return metadataVersion(table, iceHandle);
2201+
});
2202+
} catch (Exception e) {
2203+
throw IcebergExceptionUtils.wrapTableLoadFailure(iceHandle, e,
2204+
"Failed to resolve iceberg partition view version, error message is:");
2205+
}
2206+
}
2207+
21052208
/**
21062209
* The physical iceberg partitions of {@code handle} with per-partition value maps, so the generic
21072210
* {@code PluginDrivenExternalTable.getNameToPartitionItems} can populate {@code selectedPartitionNum}
@@ -2170,7 +2273,7 @@ private List<ConnectorPartitionInfo> listPartitionsUncached(
21702273
}
21712274

21722275
private List<ConnectorPartitionInfo> listPartitions(Table table, IcebergTableHandle iceHandle) {
2173-
return IcebergPartitionUtils.listPartitions(table,
2276+
return IcebergPartitionUtils.listPartitions(table, iceHandle.getSnapshotId(),
21742277
TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()), partitionCache);
21752278
}
21762279

@@ -2182,7 +2285,8 @@ private List<ConnectorPartitionInfo> listPartitions(Table table, IcebergTableHan
21822285
* snapshot id (or {@code -1} for an empty table — iceberg DOES support MVCC, so it still pins, mirroring
21832286
* paimon), and the LATEST schema id (NOT {@code currentSnapshot().schemaId()} — a schema-only change without
21842287
* a new snapshot advances the schema while the snapshot's id lags; legacy reads {@code table.schema()
2185-
* .schemaId()}). T08 serves this through the per-catalog {@link IcebergLatestSnapshotCache}: within the TTL
2288+
* .schemaId()}). An already frozen statement read takes precedence over the shared pin. Otherwise T08
2289+
* serves this through the per-catalog {@link IcebergLatestSnapshotCache}: within the TTL
21862290
* a HIT returns the cached pin without re-loading the table (saves the load I/O and keeps the snapshot
21872291
* STABLE across queries — the legacy with-cache catalog). The cached value carries BOTH ids atomically so a
21882292
* schema-only ALTER between two queries cannot skew snapshotId vs schemaId. A disabled cache
@@ -2198,11 +2302,21 @@ public Optional<ConnectorMvccSnapshot> beginQuerySnapshot(
21982302
// per-user loadTable authorization) reads live per-user every call, so authorization runs every time (no
21992303
// stale-authz window); mirrors resolveTableForRead's tableCache null-fallback. A disabled cache (the table
22002304
// entry turned off) is a non-null cache that reads live internally.
2201-
IcebergLatestSnapshotCache.CachedSnapshot pin = latestSnapshotCache != null
2202-
? latestSnapshotCache.getOrLoad(id, () -> loadLatestSnapshotPin(session, iceHandle))
2203-
: loadLatestSnapshotPin(session, iceHandle);
2305+
IcebergLatestSnapshotCache.CachedSnapshot pin;
2306+
if (IcebergStatementScope.hasReadTable(session, iceHandle.getDbName(), iceHandle.getTableName())) {
2307+
// Handle/schema resolution may have frozen an older generation before another query's REFRESH.
2308+
// A newer shared pin must not point outside that frozen metadata, nor may this reader publish its
2309+
// old pin over the refreshed cache entry.
2310+
pin = latestSnapshotPin(loadTable(session, iceHandle));
2311+
} else {
2312+
pin = latestSnapshotCache != null
2313+
? latestSnapshotCache.getOrLoad(id, () -> loadLatestSnapshotPin(session, iceHandle))
2314+
: loadLatestSnapshotPin(session, iceHandle);
2315+
}
2316+
// SQL result-cache identity and column binding must describe the same read generation. The engine
2317+
// must not combine this snapshot/schema pin with an independently expiring name-keyed schema cache.
22042318
ConnectorMvccSnapshot.Builder snapshot = ConnectorMvccSnapshot.builder()
2205-
.snapshotId(pin.snapshotId).schemaId(pin.schemaId);
2319+
.snapshotId(pin.snapshotId).schemaId(pin.schemaId).retainSchema(true);
22062320
if (pin.snapshotId < 0) {
22072321
snapshot.property(EMPTY_PARTITION_STYLE_PROPERTY, pin.emptyPartitionStyle.name());
22082322
}

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -777,10 +777,15 @@ static List<ConnectorPartitionInfo> listPartitions(Table table) {
777777
/** Cache-aware overload (PERF-02): see {@link #buildMvccPartitionView(Table, long, TableIdentifier,
778778
* IcebergPartitionCache)}. Keyed by the table's CURRENT snapshot id. */
779779
static List<ConnectorPartitionInfo> listPartitions(Table table, TableIdentifier id, IcebergPartitionCache cache) {
780+
return listPartitions(table, -1L, id, cache);
781+
}
782+
783+
static List<ConnectorPartitionInfo> listPartitions(Table table, long pinnedSnapshotId,
784+
TableIdentifier id, IcebergPartitionCache cache) {
780785
if (table.spec().isUnpartitioned()) {
781786
return Collections.emptyList();
782787
}
783-
Snapshot current = table.currentSnapshot();
788+
Snapshot current = pinnedSnapshotId >= 0 ? table.snapshot(pinnedSnapshotId) : table.currentSnapshot();
784789
if (current == null) {
785790
return Collections.emptyList();
786791
}

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,11 @@ final class IcebergStatementScope {
7474

7575
private IcebergStatementScope() {}
7676

77+
/** A frozen read or its live lease proves existence even if REFRESH retires the catalog cache. */
78+
static boolean hasReadTable(ConnectorSession session, String dbName, String tableName) {
79+
return ConnectorStatementScopes.getIfPresent(session, TABLE_NAMESPACE, dbName, tableName) != null;
80+
}
81+
7782
/**
7883
* Loads and freezes the iceberg {@link Table} for {@code db.tbl} once per statement. The frozen operations
7984
* prevent another statement's write refresh from changing the metadata generation after slots are bound.

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,11 @@ boolean isEnabled() {
113113
return entry.isEnabled();
114114
}
115115

116+
/** Only a retained, unexpired table proves existence; never loads or caches a boolean answer. */
117+
boolean contains(TableIdentifier identifier) {
118+
return entry.getIfPresent(identifier) != null;
119+
}
120+
116121
/**
117122
* Returns the cached table for {@code identifier} if present and unexpired, else runs {@code loader} (the
118123
* live remote {@code loadTable}), caches and returns it. When caching is disabled ({@link #isEnabled()} is

0 commit comments

Comments
 (0)