Repository navigation
Conversation
### What problem does this PR solve?
Issue Number: None
Problem Summary:
Iceberg full-schema dictionaries lowercased top-level fields while scan slots preserved their Iceberg names. Snapshot-pinned partition carriers also resolved names from the latest table schema instead of the query schema. Mixed-case, time-travel, Top-N, partition evolution and equality-delete scans could therefore send internally inconsistent FE/BE schemas.
Build every carrier from one authoritative scan schema and resolve partition keys by stable Iceberg field id. Preserve case only for top-level fields; nested struct names keep the existing Doris lowercase contract. Thread the same schema through batch, streaming, and count-pushdown range planning.
### Release note
Fix mixed-case and time-travel Iceberg reads by keeping scan-slot and partition-carrier names consistent.
### Check List (For Author)
- Test: Unit Test
- ICEBERG connector module: 1478 tests, 0 failures, 1 skipped live test
- Checkstyle and connector import gates passed
- Behavior changed: Yes. Mixed-case top-level dictionary names now match scan-slot names, and pinned partition carriers use the pinned schema generation.
- Does this need documentation: No
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
/review |
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
Add an external Iceberg regression suite for mixed-case full-schema and snapshot-pinned scans. The suite covers Top-N, nested structs, partition schema evolution, partition predicates, and count pushdown.
### Release note
None
### Check List (For Author)
- Test: Regression test
- test_iceberg_scan_schema_carrier_names
- Behavior changed: No
- Does this need documentation: No
|
/review |
|
run buildall |
There was a problem hiding this comment.
Reviewed all 10 changed paths at commit 4e60704. One new P2 finding concerns repeated full-schema searches during per-file partition planning. No P0/P1 finding and no existing blocking inline comments were identified.
Critical checkpoint conclusions:
- Goal and correctness: preserved root names now agree with Doris scan slots; partition keys and per-file values resolve against the same pinned schema. The added unit and regression cases cover mixed case, historical renames, nested fields, Top-N carriers, eager/streaming ranges and counts. Static FE-to-BE tracing supports the intended correction.
- Scope and reuse: focused Iceberg carrier changes; parameter/comment renames match existing case-preserved handles. P2 below identifies an available indexed lookup that should be reused.
- Concurrency: no new threads, locks or shared mutable state. Streaming captures an immutable Schema and retains its existing single-consumer contract.
- Lifecycle: iterator ownership, close/error paths, count retries and one-file scratch lifetime remain intact. No new static initialization or ownership dependency.
- Configuration: no configuration items added or changed.
- Compatibility: existing schema transport and symbols are retained. V1 exact root lookup agrees with preserved slots; V2 resolves roots case-insensitively/by field ID. Nested names remain lowercase.
- Thrift/protobuf: no definitions, IDs or required fields changed.
- Parallel paths: eager, streaming, manifest-count representative, file-task count fallback, Top-N, equality-delete and system-table paths were inspected; schema propagation is consistent.
- Conditions and errors: existing empty-pin, system-table, binary/fixed partition and count safety gates remain. Missing requested columns and malformed schema carriers continue to fail explicitly; nested partition sources are excluded consistently.
- Test coverage: new JUnit cases and nine ORC regression queries exercise the changed naming contract. The SQL does not explicitly assert Top-N materialization plan activation; equality-delete runtime coverage and wide-table planning performance are not added here. No concrete correctness failure was found in those paths.
- Expected results: all nine output labels align with queries; sorted rows, nested values, historical/current row sets, limits and counts of 2/3 match the inserted data. Output generation and author-reported executions were not independently reproduced.
- Observability: existing schema/decode/planning errors remain adequate for this local carrier change; no new distributed operation requires metrics or logging.
- Transactions and persistence: no EditLog, transaction, replay, durability or failover behavior is changed.
- Writes and crash cleanup: production changes affect read planning only; no new data-write atomicity or cleanup requirement.
- FE-to-BE variables: no new protocol variable; existing schema-name and path-key/value carriers were traced through both native reader generations.
- Performance: P2 below adds work proportional to schema width for every file on identity-partitioned scans. The operation count follows directly from the loops; runtime latency was not measured.
- Other concerns and user focus: no extra user focus was supplied. Initial casing/equality and pinned-partition correctness risks were dismissed with concrete caller/reader evidence; the lookup performance issue is the sole accepted finding.
Validation was static only, as required: no builds, tests, source edits or mixed-version cluster execution. All changed paths and candidate dispositions are covered; no unresolved code candidate remains. The independent confirmation round and final sweep determine the convergence result recorded with this delivery.
Review converged after 2 round(s).
|
|
||
| String columnName = table.schema().findColumnName(partitionField.sourceId()); | ||
| String columnName = | ||
| topLevelSourceColumnName(scanSchema, partitionField.sourceId()); |
There was a problem hiding this comment.
[P2] Keep identity source lookup indexed on the per-file path
computePerFileInvariants calls this method for every distinct data file on eager and streaming scans. topLevelSourceColumnName scans all of scanSchema.columns() for each identity field, replacing the cached map lookup in Schema.findColumnName. For a 1,000-column table whose identity partition source is last, planning 1,000,000 files now performs about one billion field-ID comparisons even for a narrow projection; the scratch cache only saves repeated byte slices of the same file. Use the cached top-level lookup scanSchema.asStruct().field(sourceId), or a scan-scoped ID-to-name map, to preserve pinned names and nested-source exclusion without multiplying planning work by schema width.
There was a problem hiding this comment.
Reviewed all ten changed files at 4e60704ac3b634d97c1d745af08663d6e42d8115. No new inline findings. The existing P2 identity-source lookup finding still applies: source-name lookup now scans the top-level schema for each identity field of each distinct data file. It is already covered and was not reposted. There are no existing P0/P1 findings to carry forward.
Review converged in 1 round: both complete-review subagents and the additional risk reviewer returned NO_NEW_VALUABLE_FINDINGS against the same empty new-comment set. All candidates and risk questions have evidence-based dispositions; no changed-path coverage gap or unresolved code candidate remains.
| Critical checkpoint | Conclusion |
|---|---|
| Goal and proof | The implementation aligns full/equality schema root names with case-preserved scan slots and resolves partition names by stable field ID in the selected scan schema. Added unit cases inspect root/nested casing and pinned eager/streaming carriers; the ORC regression covers rename, filters, Top-N-shaped SQL and counts. |
| Scope, clarity and reuse | Focused internal Iceberg changes with shared range builders; handle/range edits are comments, and helper parameter renames match the actual name contract. |
| Concurrency and locks | The streaming source retains a final immutable schema reference and uses its existing serial consumption and per-source scratch. No new shared mutation, locks, lock ordering or thread entry. |
| Lifecycle and static initialization | No new resource ownership or static initialization dependency. Existing iterator/task close and count try-with-resources paths remain intact. |
| Configuration | No configuration changes. |
| Compatibility | No shared SPI, function-symbol or storage-format changes. Corrected names use existing carriers; both legacy and current BE consumers were traced. |
| Thrift/protobuf IDs | No definition changes; branch field-ID comparison is inapplicable. |
| Parallel paths | Eager, streaming, manifest-count and file-count paths, including cache/SDK retries, consistently pass the scan schema. System tables remain separate; full pinned, Top-N and equality carriers preserve root names. |
| Conditions and errors | Stable-ID lookup intentionally excludes absent/dropped/nested root sources; existing nonidentity/binary handling and tuple typing remain unchanged. Nested dictionary names still lowercase; missing requested fields still fail explicitly. |
| Test coverage and negative cases | New tests cover the target carrier failures. Existing cases cover absent requested fields, nulls, spec evolution and equality drop/re-add histories. Builds and tests were not run under this static-review contract. |
| Expected results | All nine new output blocks agree with the seeded rows, ordering, limits, nested values, two-row pinned snapshot and three-row current table; query/result labels match. Generation provenance was not independently verified. |
| Observability | No new execution mode requiring additional metrics/logs; existing metadata error conversion, cache fallback warnings and scan metrics remain. |
| Transactions, persistence and failover | Read-side planning changes only; no EditLog, commit/replay or failover change. Snapshot/schema selection remains the read visibility fence. |
| Data writes and crash atomicity | No production data-write changes or new cleanup obligations. |
| FE/BE propagation | Dictionary root names, path partition keys and per-range keys use the same selected schema; values/null flags remain paired. Top-N phase two copies the full schema params and preserves names/categories. No new wire variable. |
| Performance | The existing P2 above is independently confirmed. No distinct additional regression was substantiated. |
| Other core invariants | No native allocation or MoW/delete-bitmap changes. Equality dependencies continue to resolve stable IDs, including historical hidden keys. |
| User focus | No additional user-provided focus was supplied. |
Validation limits: this was a static review, with no builds, tests or source modifications. Author-reported executions were not reproduced. The new regression does not force a particular scanner version or lazy-materialization plan, and mixed-case equality-delete application was traced through both BE implementations rather than executed end to end. These execution limits do not leave an unresolved code candidate.
Review converged after 1 round(s).
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
Per-file identity partition planning searched every top-level schema column for each partition field. Resolve the field-id through Iceberg's top-level struct index so planning no longer scales with schema width.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- IcebergPartitionUtilsTest
- IcebergScanPlanProviderTest
- IcebergSchemaUtilsTest
- Behavior changed: No
- Does this need documentation: No
|
run buildall |
|
/review |
There was a problem hiding this comment.
No new actionable findings in PR #69382 at 4450fac. All 10 changed paths and 52 diff hunks were reviewed, including their relevant upstream/downstream consumers. The review converged in 1 round: both general reviewers and the separate risk reviewer returned NO_NEW_VALUABLE_FINDINGS against the same empty inline set.
Existing P2 thread 4334582627 is fixed at this head: topLevelSourceColumnName now calls asStruct().field(sourceId), which uses a cached direct-child ID map in Iceberg 1.12.0. It is not reposted. No existing P0/P1 applies. No additional user focus was supplied.
Critical checkpoint conclusions:
- Goal: the inspected code aligns top-level dictionary names, identity partition keys and per-file values with the selected scan schema. Added unit and regression cases target mixed-case and historical reads.
- Scope and reuse: the changes are focused on Iceberg name/schema propagation, reuse existing selectors, and cover internal signature plumbing without broad API changes.
- Concurrency and locks: no new threads or locks. Statement-frozen metadata and an immutable captured Schema keep naming stable; streaming scratch remains source-local.
- Lifecycle and initialization: existing iterable/iterator cleanup is preserved. The new schema reference follows split-source lifetime; no new resource cycle or static initialization dependency.
- Configuration: none added or changed; dynamic update requirements do not apply.
- Compatibility: no public SPI, function symbol or storage-format change. Existing carriers retain stable field IDs and now match case-preserved slots.
- Thrift/protobuf: no definitions, field IDs or enum values changed.
- Parallel paths: eager, streaming, both count strategies and cache fallbacks pass the selected schema; pinned, ref, equality-delete, Top-N and system-table paths were traced.
- Special conditions: identity-only/top-level source restrictions match engine slot classification. Dropped IDs cannot bind reused names; nested fields remain file-decoded and nested dictionary names remain lowercase.
- Test coverage: new tests inspect full-schema/equality case, historical partition names, Top-N dictionaries and eager/streaming carriers. Existing negative/evolution/count tests were inspected for the affected contracts.
- Expected results: all nine regression sections match their queries; ordering is deterministic, and rows, limits and pinned/current counts agree with setup by inspection.
- Observability: existing metadata errors and scan metrics remain sufficient for this local carrier correction; no new workflow needs instrumentation.
- Transactions/persistence/failover: no transaction, EditLog, commit or replay behavior changed. Existing snapshot selection is preserved; native MoW/version-write invariants are outside the change.
- Writes and atomicity: no data-write or recovery protocol change. The reused static-partition helper retains equivalent direct-root lookup semantics.
- FE/BE propagation: existing schema dictionaries and partition key/value carriers agree with slot names in both reader generations. Top-N refetch retains the source schema parameters.
- Performance and memory: no schema copy or growing per-file cache is introduced. Indexed source lookup removes the existing schema-width multiplier while retaining bounded scratch state.
- Other risks: rename/drop/reuse, schema-only commits, empty pins, nested source collisions, historical equality IDs and row-lineage paths were rechecked; no unresolved static correctness candidate remains.
Validation limits: this was a static-only review as required; no builds or tests were run. The author's reported test success and generated-output provenance were not independently reproduced. No physical Top-N plan or equality-delete read was observed at runtime; the new regression does not itself create equality-delete data. These are runtime validation limits, with no uncovered assigned static scope.
Review converged after 1 round(s).
FE UT Coverage ReportIncrement line coverage `` 🎉 |
TPC-H: Total hot run time: 28824 ms |
TPC-DS: Total hot run time: 150949 ms |
ClickBench: Total hot run time: 24.22 s |
FE UT Coverage ReportIncrement line coverage `` 🎉 |
FE Regression Coverage ReportIncrement line coverage |
924060929
left a comment
There was a problem hiding this comment.
Reviewed scan-slot/schema-dictionary names, pinned-schema partition field IDs, eager/streaming/count paths and TopN/equality carriers at 4450fac. No blocking issue found. The indexed lookup addresses the earlier review comment. Static source review only; no local build or test execution. P0 Regression was pending when checked.
|
PR approved by anyone and no changes requested. |
|
PR approved by at least one committer and no changes requested. |
Brings in apache#69382, which renamed IcebergScanPlanProvider#requestedLowerNames to requestedColumnNames; the next commit moves the $labels planner onto it. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…rename ### What problem does this PR solve? Issue Number: apache#68916 Related PR: apache#69382 Problem Summary: apache#69382 replaced IcebergScanPlanProvider#requestedLowerNames with requestedColumnNames, which keeps the case of the requested column names. The $labels planner added by this PR still called the removed helper, so the PR no longer compiled once merged into master ("cannot find symbol: method requestedLowerNames"). Call requestedColumnNames like the other system-table path does. The labels schema is all lower case, so the projected schema is unchanged. ### Release note None ### Check List (For Author) - Test: Unit Test, Regression test - All 1646 iceberg connector FE unit tests, the $labels ones included, pass on top of master (0 failures, 5 skipped). - test_iceberg_rest_labels passes. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
#69382) (#69709) pick #69382 4.2 still uses the legacy IcebergScanNode/IcebergUtils path. The fix therefore resolves identity partition keys and values from the query-pinned scan schema by Iceberg field id, so path_partition_keys, schema info, and range partition metadata stay aligned. The master-only fe-connector dictionary/lowercasing changes are not applicable to this path.
What problem does this PR solve?
Issue Number: N/A
Problem Summary:
Iceberg scan carriers could use inconsistent top-level name casing or schema generations, especially for full-schema, Top-N, and snapshot-pinned partition scans. Build all carriers from one authoritative scan schema and resolve partition columns by Iceberg field id.
Release note
Fix mixed-case and time-travel Iceberg reads by keeping scan-schema carrier names consistent.
Check List (For Author)
Test
Regression test:
test_iceberg_scan_schema_carrier_namespassed twice, including normal output comparison.Unit tests: Iceberg connector suite passed (1478 tests, 0 failures, 1 skipped).
Manual test: verified mixed-case ORC, Top-N, pinned snapshot partition rename/filter, and count pushdown with Spark, Iceberg REST, and Doris.
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)