You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b44b384
Browse filesBrowse the repository at this point in the historyBrowse files
[improvement](fe) Version Iceberg partition views and SQL cache metadata
### What problem does this PR solve?
Issue Number: None
Problem Summary: Ordinary Iceberg scans and SQL result-cache validation materialize and copy the full Doris partition view even when native file pruning or a pinned metadata version is sufficient. Cache immutable engine partition views and their pruning index by connector metadata generation, defer conversion for native planning, and obtain SQL cache versions through a source-neutral version provider. Record distinct current, historical and branch/tag references when a statement reads several versions of one table, and revalidate them through the existing snapshot resolver. Reuse statement-retained table handles and combine relation-name discovery through the connector SPI to avoid repeated remote existence and listing requests. Preserve MV/partition metadata consumers, refresh invalidation and frozen-reader schema consistency.
Review repairs: retain heterogeneous display/freshness partition maps without an invalid sorted index, carry entry-specific scalable byte estimates through catalog cache definitions, and align partition-metadata regression settings and the Hive fixture with native planning. Rebase includes upstream source-nullability and the existing Paimon CHAR regression repair; connector API is 20.0. Final rebased validation: 210 scoped unit tests, an overlapping 21-test weighted-cache/runtime pass, and build.sh --fe -j4 with Checkstyle passed. Historical performance and live-SQL validation below predate these review repairs.
Review follow-up: keep Iceberg tableExists authoritative for DDL and use that live preflight before DROP TABLE. Cached/pinned handles remain available to read planning. Added externally dropped-table cases for IF EXISTS and strict DROP, plus cached/pinned read isolation. Keep the partition-rule Env accessor independent of the scheduler import removal so the FE PRs compile together. 96 scoped tests and build.sh --fe with Checkstyle passed.
Review follow-up: use live tableExists for both CREATE TABLE existence probes, including IF NOT EXISTS race recovery. Added stale cached-handle tests for both CREATE modes and failed-create propagation. All 90 DDL routing tests, full FE build and Checkstyle passed.
### Release note
Iceberg ordinary scans use native file pruning by default. The session variable enable_iceberg_internal_partition_pruning enables Doris internal partition pruning when needed. SQL result caching supports independently versioned Iceberg snapshot and branch/tag references without enumerating partitions.
### Check List (For Author)
- Test: Unit Test
- run-fe-ut.sh: 325 targeted metadata, MVCC, partition view, SQL cache and dependent test cases passed
- run-fe-ut.sh: all 85 connector SPI tests passed separately
- build.sh --fe -j4 passed, including Checkstyle and connector architecture gates
- Behavior changed: Yes, defer ordinary Iceberg partition conversion by default and support version-aware external SQL caching
- Does this need documentation: Yes, updated fe/fe-connector/README.md
### Performance validation
Measured independently against unoptimized master `e0bf4874070`, with identical Linux AArch64 Release BE, data/footer prewarming, and session settings. Numbers are complete SQL medians, in milliseconds. The baseline column shows two separate baseline runs.
| Scenario | Baseline runs | This PR |
|---|---:|---:|
| 120,000 partitions, warm LIMIT 10 | 389.38–463.29 | 9.69 |
| 120,000 partitions, warm EXPLAIN | 188.19–189.07 | 3.45 |
| 120,000 partitions, warm LIMIT 10 with internal pruning enabled | 333.18–391.80 | 9.76 |
| 1,000 manifests / 10,000 files, cold LIMIT 10 | 925.44–933.77 | 26.50 |
The large-partition protocol enables SQL Cache validation but varies an always-true predicate to avoid result-cache hits. Ordinary LIMIT cases have 40 measured samples; EXPLAIN has 20. Baseline EXPLAIN retention is bounded by using equivalent SELECT warmup, identically for all variants. Function-heavy partition predicates remain approximately 5.5 seconds in this fixture and are not claimed as a benefit.
### Additional validation
- Live SQL Cache checks: 170 assertions for current reads, snapshots, timestamps, branch/tag movement, multiple versions of one table, schema evolution/rename, snapshot expiry and ref removal/recreation.
- The final combined tree passed a clean FE build, 997 scoped unit tests without skips, and the Hive streaming / Iceberg scheduled-split / table-handle-reuse regression suites.
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java
+17Lines changed: 17 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -53,6 +53,7 @@
53
53
importjava.io.IOException;
54
54
importjava.util.ArrayList;
55
55
importjava.util.Collections;
56
+
importjava.util.LinkedHashSet;
56
57
importjava.util.List;
57
58
importjava.util.Locale;
58
59
importjava.util.Map;
@@ -94,6 +95,13 @@ public interface IcebergCatalogOps {
94
95
/** Lists the view names in {@code dbName}; empty when the catalog is not a (view-enabled) ViewCatalog. */
95
96
List<String> listViewNames(StringdbName);
96
97
98
+
/** Lists relations without requiring callers to subtract and then re-add view names. */
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java
+5Lines changed: 5 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -74,6 +74,11 @@ final class IcebergStatementScope {
74
74
75
75
privateIcebergStatementScope() {}
76
76
77
+
/** A frozen read or its live lease proves existence even if REFRESH retires the catalog cache. */
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java
+5Lines changed: 5 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -113,6 +113,11 @@ boolean isEnabled() {
113
113
returnentry.isEnabled();
114
114
}
115
115
116
+
/** Only a retained, unexpired table proves existence; never loads or caches a boolean answer. */
117
+
booleancontains(TableIdentifieridentifier) {
118
+
returnentry.getIfPresent(identifier) != null;
119
+
}
120
+
116
121
/**
117
122
* Returns the cached table for {@code identifier} if present and unexpired, else runs {@code loader} (the
118
123
* live remote {@code loadTable}), caches and returns it. When caching is disabled ({@link #isEnabled()} is
0 commit comments