Repository navigation
branch-4.2: [improvement](datasource) Version Iceberg partition views and SQL cache metadata (#69299) - #69983
Conversation
…he metadata ### What problem does this PR solve? Issue Number: None Related PR: apache#69299 Problem Summary: Adapt the metadata projection and SQL result-cache changes to branch-4.2's native datasource APIs. Keep lightweight Iceberg snapshot metadata separate from a lazily loaded, generation-keyed and budgeted partition view. Validate each cached table reference using its original read selector and a version covering table identity, metadata, snapshot and schema, while retaining the existing Hive fallback and native Iceberg partition pruning. ### Release note Reduce unnecessary Iceberg partition metadata reads and allow opt-in SQL result caching to validate Iceberg read generations. ### Check List (For Author) - Test: Unit Test: 225 selected FE tests passed; full build.sh --fe package build and Checkstyle passed. External-service regression tests were not run locally. - Behavior changed: Yes, Iceberg partition views load on demand and opt-in external SQL caching validates Iceberg read generations. - Does this need documentation: No
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
### What problem does this PR solve? Issue Number: None Related PR: apache#69299 Problem Summary: The snapshot-id test passed a null partition-item map because it only exercised partition aliases. Iceberg partition projections now copy their maps before cache publication, so this invalid fixture fails before reaching its snapshot assertions. Supply an empty map and retain all snapshot-selection assertions. No production behavior changes. ### Release note None ### Check List (For Author) - Test: Unit Test: reproduced the original IcebergPartitionInfoTest failure, then passed its 2 tests and all 25 FrontendServiceImplTest tests; full build.sh --fe package build and Checkstyle passed. The remote authorization-test failure was not reproduced locally. - Behavior changed: No - Does this need documentation: No
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
There was a problem hiding this comment.
Reviewed all 17 changed files at head 4471ebb0a93b2cfb6a5d6375058a080e070c800f against the supplied merge-base diff. Two new P2 findings are inline: repeated full partition enumeration when shared caching is unavailable, and retention of obsolete complete partition views after normal table-generation replacement. No P0/P1 finding was verified and no existing inline findings require carrying forward. No additional user focus was supplied.
The review converged after 2 rounds: all three reviewers returned NO_NEW_VALUABLE_FINDINGS in the final round; every candidate and initial risk has an explicit, independently checked disposition. Static review coverage is complete. No builds, tests, benchmarks, source modifications, or submodule initialization were performed. The conclusions below describe inspected code and tests, not runtime validation.
- Goal and proof: version identity covers table UUID, metadata file, selected snapshot and schema without partition enumeration. Lazy partition views are separately cached. Mock tests inspect identities/selectors and shared loads; the two partition-cache regressions remain.
- Scope: changes stay within FE Iceberg metadata, SQL-cache integration, and three unit-test files; no unrelated modification found.
- Concurrency and locks: relation-local snapshots and one physical table-generation lease per statement preserve the binding-to-scan boundary. Existing cache admission/mutation fencing and table locking remain; no additional race, lock-order or deadlock finding verified.
- Lifecycle and initialization: captured frozen generations/authentication and statement/operation leases preserve resource ownership. The replacement-retention finding concerns redundant full partition maps, not a FileIO leak or stale data. No new static initialization dependency found.
- Configuration: the new partition entry inherits enable/TTL/capacity/max-weight controls and generic catalog reset handling. Disabled/rejected admission triggers the repeated-work finding.
- Compatibility: changes affect in-memory FE identities and cache keys; no persistent storage layout or backend function-symbol change.
- Thrift/protobuf: none changed, so release-branch field/enum comparison is not applicable.
- Parallel paths: native Iceberg, ordinary Hive, OLAP, mixed external scans, HMS-backed Iceberg and system-table exclusions were traced. Unsupported/versionless providers bypass caching; no additional issue verified.
- Special conditions: immutable original selectors, branch/tag restrictions, explicit snapshots, missing identities and empty-table sentinels were inspected through their actual resolvers. No additional stale-hit path found.
- Test coverage: added tests cover identity dimensions, defensive selector copies, independent references, sharing and explicit invalidation. Disabled/rejected multi-partition refresh and normal generation replacement are not covered; end-to-end cache hit/miss behavior was inspected only.
- Expected results: no regression result files changed; new unit-test assertions match their mocked paths but do not exercise the two findings.
- Observability: the new entry inherits cache load/admission statistics, and existing exception/logging boundaries remain; no extra observability defect verified.
- Transactions and persistence: no EditLog, replay, commit, visibility publication or failover path changed.
- Data writes and crashes: no data-write path changed. The writable projection's rewrite-task consumer does not access lazy partition information after commit; existing cleanup ownership was traced.
- FE/BE variables: no new transmitted variable; existing SQL-key/cache parameter messages are reused.
- Performance and memory: version extraction avoids partition reads, and partition payloads have a separate estimator. The two inline findings identify repeated enumeration and avoidable obsolete-generation retention; neither alleges an unbounded leak or measured runtime impact.
- Remaining/core invariants: no new BE memory/null/Status path, MoW delete bitmap, or internal data-version write semantics are touched. Provider/cache failures disable caching. Immutable-map consumers read or copy their inputs. No unresolved code-level question or unreviewed changed scope remains.
Review converged after 2 round(s).
| } | ||
|
|
||
| public IcebergPartitionInfo getPartitionInfo() { | ||
| return partitionInfoLoader == null ? partitionInfo : partitionInfoLoader.get(); |
There was a problem hiding this comment.
[P2] Avoid reloading the entire partition view per MV partition
With meta.cache.iceberg.partition.enable=false (or a partition view rejected by its weight limit), this invokes the full loader on every access. MTMVRefreshContext.loadSnapshots calls the default MTMVRelatedTableIf.getPartitionSnapshots, which loops over N names and reaches this method through IcebergExternalTable.getPartitionSnapshot each time. One refresh therefore performs N full PartitionsTable.planFiles() scans and map rebuilds for the same pinned snapshot; previously the snapshot retained one loaded view. Reuse the view across that bulk operation or in the refresh scope, while preserving shared-cache accounting, and cover repeated accesses when admission is disabled/rejected.
There was a problem hiding this comment.
Fixed in 350e47d. Native and HMS Iceberg now load the partition view once per bulk MV operation and derive all requested versions from it. The view remains local to that operation; disabled or weight-rejected entries are not retained in the shared snapshot. Added disabled/rejected-admission tests (three requested partitions, one full load), HMS dispatch coverage, and fallback/per-partition failure tests. All 197 related FE unit tests and the full FE build including Checkstyle pass locally.
| MetaCacheEntry<IcebergSnapshotEntryKey, IcebergPartitionInfo> partitions = | ||
| partitionEntry.getIfInitialized(nameMapping.getCtlId()); | ||
| if (partitions != null) { | ||
| partitions.invalidateIf(key -> key.getNameMapping().equals(nameMapping)); |
There was a problem hiding this comment.
[P2] Retire partition entries during table-generation replacement
This cleanup only runs on table removal. Normal asynchronous refresh calls retireTableGeneration, and REPLACED deliberately skips the removal listener, so old partition keys survive even though superseded snapshot projections are invalidated. Each consumed generation now leaves another full partition map cached; after its pinned queries finish, new latest readers use the new metadata-file key and explicit historical reads do not use this entry. The default strong count cache can retain these obsolete full views for 24 hours (up to 1000 entries), multiplying partition-metadata memory compared with the previous in-snapshot storage. Extend generation-aware retirement to partition entries as well, including preventing an old loader from republishing retired keys, while preserving same-generation reuse.
There was a problem hiding this comment.
Fixed in 350e47d. Table-generation replacement now invalidates partition entries from obsolete physical generations. Partition lookups also check the published table generation before and after loading, so a pinned old projection can read its own immutable view without republishing it after retirement. Same-generation and unrelated-table entries remain reusable. Added cached-replacement, replacement-during-load, and late-access tests. All 197 related FE unit tests and the full FE build including Checkstyle pass locally.
### What problem does this PR solve? Issue Number: None Related PR: apache#69299 Problem Summary: Moving Iceberg partition views to a separate cache caused an MV refresh to enumerate the complete view once per requested partition whenever caching was disabled or weight admission rejected the view. Table-generation replacement also left obsolete partition views behind, and a previously pinned projection could republish an old view after retirement. Load one view for the native/HMS Iceberg bulk MV operation, share the existing version fallback, and preserve per-partition failures. Retire obsolete physical-generation entries during replacement and prevent stale lazy loads from republishing them. Same-generation reuse and cache accounting remain intact. ### Release note Reduce repeated Iceberg metadata loading during materialized-view refresh and retire obsolete partition-cache views after table metadata refresh. ### Check List (For Author) - Test: Unit Test - Reproduced all four lifecycle/load-count regressions before the fix. - All 197 related FE tests pass, including native/HMS bulk reads, disabled/rejected cache admission, replacement and late loads, version fallback and per-partition errors. - Full FE packaging and Checkstyle pass through build.sh --fe -j4 (UI build disabled). - Behavior changed: Yes (reuse operation-local partition views and retire obsolete cached generations) - Does this need documentation: No
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
FE UT Coverage ReportIncrement line coverage |
There was a problem hiding this comment.
Static review of PR 69983 completed at head 350e47d. The review converged after 2 bounded rounds. The changed SQL-cache/MVCC selector flow, Iceberg generation retirement and accounting, bulk native/HMS MTMV snapshot flow, and Iceberg cache admission were traced end to end. One P1 finding is included below: lazy partition/schema loading can fail for retained projections after an Iceberg catalog reset retires the cache group, even though the retained table generation is still expected to serve pinned work. Existing P2 comments 4442498716 and 4442499089 were independently revalidated as fixed on this head and are not reposted. The supplied review focus has no additional points. Tests and builds were not run because this invocation is static-only; changed tests were inspected as review evidence.
Review converged after 2 round(s).
| + "." + dorisTable.getName(); | ||
| Optional<IcebergSnapshotEntryKey> key = IcebergSnapshotEntryKey.tryCreate(tableIdentity, retainedTable); | ||
| MetaCacheEntry<IcebergSnapshotEntryKey, IcebergPartitionInfo> partitions = | ||
| key.isPresent() ? partitionEntry.get(key.get().getNameMapping().getCtlId()) : null; |
There was a problem hiding this comment.
[P1] Do not require a live partition-cache catalog group while loading a retained Iceberg projection
loadSnapshotProjection now publishes an empty partition view and calls partitionEntry.get(catalogId) before installing the lazy loader. A statement or rewrite can still hold a retained table/writable lease while IcebergExternalCatalog.resetToUninitialized / removeCatalogByEngine retires the catalog group; AbstractExternalMetaCache.entry then throws before the captured loader is installed. If the projection was created before reset, its first partition read instead reaches IcebergUtils.loadPartitionInfo and the new schemaEntry.get(catalogId) lookup fails for the same reason. Pinned reads therefore fail even though the retained table generation is intended to remain usable. Capture reset-safe generation-bound partition/schema dependencies, or retain the loaded partition view for such projections, and skip current-group lookups for session-catalog/direct-load paths.
There was a problem hiding this comment.
Fixed in 1385a1e. Lazy partition loading now derives partition columns from the same retained table/schema under its captured authenticator and mapping options; it no longer resolves a current schema-cache/catalog group. Projection creation captures only an existing shared partition entry, while retired-group and session-catalog paths load directly. The existing admission and generation-retirement fences are preserved. Added three tests with real Iceberg day-partition metadata and a table-generation lease held across reset, covering reset before the first read, reset before projection creation, and a session-catalog path with no cache group. All 204 related FE tests and full FE packaging/Checkstyle pass.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69299 Problem Summary: Iceberg partition views are loaded lazily from a retained table generation. Catalog reset can remove its cache group before projection creation or the first partition read. Looking up current partition and schema entries then fails pinned work despite its live table lease. Capture only an existing shared partition entry when applicable and derive partition columns from the same retained table/schema under captured authentication. Session-catalog and retired-group paths load directly without resolving another catalog generation, while existing cache admission and retirement fences remain in place. ### Release note Keep retained Iceberg partition metadata usable after catalog cache reset. ### Check List (For Author) - Test: Unit Test: 204 related FE tests; full FE packaging and Checkstyle via build.sh --fe - Behavior changed: Yes; retained partition projections can finish lazy metadata loading after catalog reset - Does this need documentation: No
|
run buildall |
|
/review |
There was a problem hiding this comment.
Review converged after 2 rounds at the verified head 1385a1e. I found one new P2 performance regression: the lazy Iceberg partition supplier is not memoized for unadmitted/direct projections, so dictionary freshness checks can rerun PartitionsTable.planFiles() and rebuild the full partition maps on every check. The earlier MTMV bulk-refresh finding (comment 4442499089) was independently confirmed fixed and is a distinct consumer; the reset-safe partition finding (comment 4446153215) was also confirmed fixed. The SQL-cache/MVCC selector and cache-generation review found no additional issue. The supplied review focus file contains no extra focus points. This was a static review only; no builds or tests were run.
Review converged after 2 round(s).
| @@ -222,7 +233,7 @@ public Optional<Map<Integer, List<String>>> getNameMapping() { | |||
| } | |||
|
|
|||
| IcebergSnapshotCacheValue withoutRetainedTable() { | |||
There was a problem hiding this comment.
[P2] Memoize the lazy partition view for repeated dictionary checks
getPartitionInfo() now invokes partitionInfoLoader directly whenever the contextual partition entry is disabled, weight-rejected, absent, or retired. IcebergExternalTable.getNewestUpdateVersionOrTime() reaches this through getLatestSnapshotCacheValue(...).withoutRetainedTable(), and Dictionary.hasNewerSourceVersion() / checkBaseDataValid() call it on every freshness check. Each check therefore reruns PartitionsTable.planFiles() and rebuilds the full partition maps; before this lazy split, the snapshot projection retained one loaded view. The earlier MTMV bulk fix only reuses the view within one bulk operation. Keep a generation-scoped or local memoized view for this consumer (or compute the max update time from a lightweight cached value) while preserving the bounded partition-entry policy.
There was a problem hiding this comment.
Fixed in f80fa3b. Dictionary freshness now uses the existing scoped snapshot callback and memoizes only the exact maximum partition-update timestamp on that immutable projection. Full partition maps remain subject to their separate admission/weight policy; the fixed scalar and loaded flag are reserved in snapshot accounting. The snapshot entry still controls whether a projection is retained across checks, so no separate cache bypasses its policy. Disabled, weight-rejected and retired partition-entry tests reproduce three loader calls before the fix and one per projection afterward, including a new metadata generation and the zero result for an empty view. All 207 related FE tests and full FE packaging/Checkstyle pass.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69299 Problem Summary: Repeated Iceberg dictionary freshness checks detach the latest snapshot by eagerly loading its partition view. When the partition entry is disabled, rejected by its weight limit or retired, three checks of the same immutable generation invoke the complete view loader three times. Use the existing scoped snapshot callback and retain only the exact maximum partition-update timestamp in that generation. A synchronized lazy getter computes it once, including an empty view's zero result; a new metadata generation computes its own value. Reserve the fixed scalar/flag fields in snapshot accounting and keep complete partition maps subject to the existing admission policy. Focused tests reduce three loader invocations to one without retaining an unadmitted full view; this is a loader-count result, not a latency benchmark. ### Release note Avoid repeated partition metadata enumeration during Iceberg dictionary freshness checks. ### Check List (For Author) - Test: Unit Test: 207 related FE tests; full FE packaging and Checkstyle via build.sh --fe - Behavior changed: Yes; reuse the same freshness value within one immutable metadata generation without retaining full partition views - Does this need documentation: No
|
run buildall |
|
/review |
There was a problem hiding this comment.
Automated static review of PR 69983 at head f80fa3b, completed in two bounded rounds and converged.
The review covers the new selector-aware Iceberg SQL-cache dependencies, pinned latest/VERSION/TIME/branch/tag identities, CacheAnalyzer admission, native and HMS Iceberg MV bulk snapshot loading, lazy partition-entry admission and retirement, catalog reset and generation fencing, immutable partition maps, and cache accounting. The SQL-cache and MVCC paths preserve selector and generation identity; provider-absent HMS/system-table variants fail closed. The prior inline threads about partition-view reuse, reset-safe loading, generation retirement, and dictionary freshness were revalidated at this head and are fixed, so they are not reposted. No additional user-provided focus was supplied.
Critical checkpoints: the change is focused on cache/version and partition-metadata lifecycle behavior; external metadata is paired with retained generations and captured authenticators; reset/replacement fences and admission checks were rechecked; no protocol, persistence, write-transaction, or new configuration changes were introduced; native and HMS parallel paths were reviewed. Added tests were inspected but not executed, and no builds or runtime tests were run because this invocation is static-only.
One P2 issue is reported inline: the retained-table lazy partition loader now constructs the schema directly and bypasses the existing SchemaCacheValue.validateSchema() boundary, so case-insensitive duplicate Iceberg columns can enter MV/dictionary partition range or snapshot derivation even though normal schema access rejects them. Restore validation while preserving the reset-safe retained-table path.
Review converged after 2 round(s).
| try { | ||
| icebergPartitions = authenticator.execute(() -> loadIcebergPartition(table, snapshotId)); | ||
| // A retained projection may outlive its catalog cache group. Its columns and ranges | ||
| // must use the same frozen table/schema, without looking up a new catalog generation. |
There was a problem hiding this comment.
[P2] Preserve Iceberg schema validation when deriving lazy partition metadata
The new retained-table path calls buildTableSchemaCacheValue(table, parseSchema(...)) directly, while the previous path went through getIcebergSchemaCacheValue, whose loader calls SchemaCacheValue.validateSchema() and rejects case-insensitive duplicate columns. A table with fields such as Foo and foo can therefore enter MV/dictionary partition range or snapshot derivation through this lazy loader even though normal Iceberg schema access rejects the same table. Validate the parsed schema before deriving partition columns, while retaining the existing AnalysisException context, so this path fails closed consistently.
There was a problem hiding this comment.
Fixed in 5d3cb75. The retained-table partition loader now applies SchemaCacheValue.validateSchema() immediately after parsing and before deriving partition columns, following the existing retained schema-display pattern. It stays inside the captured authenticator and existing AnalysisException wrapper; no live catalog/schema-cache lookup is reintroduced. Two real HadoopTables metadata tests reproduced ambiguous dt/DT schemas being accepted before the fix, for both active and retired catalog projections. They now verify rejection, table/snapshot error context, and no new authenticator lookup. Existing reset/session success cases remain covered. All 173 related FE tests and full FE packaging/Checkstyle pass locally.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69299, apache#69983 Problem Summary: Reset-safe lazy partition metadata loading parses a retained Iceberg table directly instead of using the live schema cache. It therefore bypassed SchemaCacheValue.validateSchema(), allowing case-insensitive duplicate columns into partition-range and MV/dictionary snapshot derivation even though normal schema loading rejects them. Reuse the same pure validator before deriving partition columns, inside the captured authenticator and existing table/snapshot AnalysisException wrapper. Preserve the retained table/schema generation and avoid a live catalog/cache lookup after reset. ### Release note Reject case-insensitive duplicate Iceberg columns consistently when deriving lazy partition metadata for materialized views and dictionaries. ### Check List (For Author) - Test: Unit tests and packaging - Two real Iceberg metadata tests reproduced the missing rejection for active and retired catalog projections before repair. - All 173 related schema, snapshot, partition and cache FE tests passed, including existing reset/session success paths. - Full FE packaging and Checkstyle passed. - git diff --check passed; no local full P0 or external-service regression in this cycle. - Behavior changed: Yes. Lazy retained partition loads reject ambiguous schemas consistently and keep table/snapshot error context. Snapshot identity, cache admission and generation-retirement semantics remain unchanged. - Does this need documentation: No
|
run buildall |
|
/review |
There was a problem hiding this comment.
Reviewed the complete PR diff at the pinned head, including the new Iceberg lazy partition cache, native Iceberg SQL-cache versioning and MVCC selector keys, HMS Iceberg MTMV bulk dispatch, schema derivation, cache accounting, generation retirement, and all changed tests. The implementation keeps partition metadata bound to the retained table generation and authenticator, fences cache publication across replacement/reset races, validates retained schemas, and preserves the existing table-snapshot fallback and per-partition AnalysisException recording. Native Iceberg SQL-cache identities cover the physical table generation, selected snapshot and schema; HMS-backed Iceberg scans remain safely uncacheable as they were before this PR because they do not implement the provider used by the new analyzer path.
All five existing inline threads were independently rechecked at head 5d3cb75: the repeated partition-load, generation-retirement, retained-projection reset, dictionary scalar, and retained-schema-validation concerns are addressed by the current code and must not be reposted. No new verified finding remains, so there are no inline comments or still-applicable blocking comment IDs.
The review converged in round 1. This was a static-only review: the added tests and relevant call paths were inspected, but no builds or tests were executed.
Review converged after 1 round(s).
|
PR approved by anyone and no changes requested. |
|
PR approved by at least one committer and no changes requested. |
cherry pick from #69299