You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a45d9bf
Browse filesBrowse the repository at this point in the historyBrowse files
[improvement](fe) Version Iceberg partition views and SQL cache metadata
### What problem does this PR solve?
Issue Number: None
Problem Summary: Ordinary Iceberg scans and SQL result-cache validation materialize and copy the full Doris partition view even when native file pruning or a pinned metadata version is sufficient. Cache immutable engine partition views and their pruning index by connector metadata generation, defer conversion for native planning, and obtain SQL cache versions through a source-neutral version provider. Record distinct current, historical and branch/tag references when a statement reads several versions of one table, and revalidate them through the existing snapshot resolver. Reuse statement-retained table handles and combine relation-name discovery through the connector SPI to avoid repeated remote existence and listing requests. Preserve MV/partition metadata consumers, refresh invalidation and frozen-reader schema consistency.
### Release note
Iceberg ordinary scans use native file pruning by default. The session variable enable_iceberg_internal_partition_pruning enables Doris internal partition pruning when needed. SQL result caching supports independently versioned Iceberg snapshot and branch/tag references without enumerating partitions.
### Check List (For Author)
- Test: Unit Test
- run-fe-ut.sh: 325 targeted metadata, MVCC, partition view, SQL cache and dependent test cases passed
- run-fe-ut.sh: all 85 connector SPI tests passed separately
- build.sh --fe -j4 passed, including Checkstyle and connector architecture gates
- Behavior changed: Yes, defer ordinary Iceberg partition conversion by default and support version-aware external SQL caching
- Does this need documentation: Yes, updated fe/fe-connector/README.md
### Performance validation
Measured independently against unoptimized master `e0bf4874070`, with identical Linux AArch64 Release BE, data/footer prewarming, and session settings. Numbers are complete SQL medians, in milliseconds. The baseline column shows two separate baseline runs.
| Scenario | Baseline runs | This PR |
|---|---:|---:|
| 120,000 partitions, warm LIMIT 10 | 389.38–463.29 | 9.69 |
| 120,000 partitions, warm EXPLAIN | 188.19–189.07 | 3.45 |
| 120,000 partitions, warm LIMIT 10 with internal pruning enabled | 333.18–391.80 | 9.76 |
| 1,000 manifests / 10,000 files, cold LIMIT 10 | 925.44–933.77 | 26.50 |
The large-partition protocol enables SQL Cache validation but varies an always-true predicate to avoid result-cache hits. Ordinary LIMIT cases have 40 measured samples; EXPLAIN has 20. Baseline EXPLAIN retention is bounded by using equivalent SELECT warmup, identically for all variants. Function-heavy partition predicates remain approximately 5.5 seconds in this fixture and are not claimed as a benefit.
### Additional validation
- Live SQL Cache checks: 170 assertions for current reads, snapshots, timestamps, branch/tag movement, multiple versions of one table, schema evolution/rename, snapshot expiry and ref removal/recreation.
- The final combined tree passed a clean FE build, 997 scoped unit tests without skips, and the Hive streaming / Iceberg scheduled-split / table-handle-reuse regression suites.
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java
+17Lines changed: 17 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -53,6 +53,7 @@
53
53
importjava.io.IOException;
54
54
importjava.util.ArrayList;
55
55
importjava.util.Collections;
56
+
importjava.util.LinkedHashSet;
56
57
importjava.util.List;
57
58
importjava.util.Locale;
58
59
importjava.util.Map;
@@ -94,6 +95,13 @@ public interface IcebergCatalogOps {
94
95
/** Lists the view names in {@code dbName}; empty when the catalog is not a (view-enabled) ViewCatalog. */
95
96
List<String> listViewNames(StringdbName);
96
97
98
+
/** Lists relations without requiring callers to subtract and then re-add view names. */
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java
+5Lines changed: 5 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -74,6 +74,11 @@ final class IcebergStatementScope {
74
74
75
75
privateIcebergStatementScope() {}
76
76
77
+
/** A frozen read or its live lease proves existence even if REFRESH retires the catalog cache. */
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java
+5Lines changed: 5 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -113,6 +113,11 @@ boolean isEnabled() {
113
113
returnentry.isEnabled();
114
114
}
115
115
116
+
/** Only a retained, unexpired table proves existence; never loads or caches a boolean answer. */
117
+
booleancontains(TableIdentifieridentifier) {
118
+
returnentry.getIfPresent(identifier) != null;
119
+
}
120
+
116
121
/**
117
122
* Returns the cached table for {@code identifier} if present and unexpired, else runs {@code loader} (the
118
123
* live remote {@code loadTable}), caches and returns it. When caching is disabled ({@link #isEnabled()} is
0 commit comments