You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8d2bd1e
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](iceberg) Publish Parquet split offsets in data file metadata (apache#69615)
### What problem does this PR solve?
Doris writes Parquet row groups for Iceberg data files but does not
publish their split offsets in commit fragments or manifests. Readers
cannot use those known boundaries when planning splits.
Collect row-group starts from the closed writer's in-memory Footer,
including dictionary pages, and carry them through an optional Thrift
field into Iceberg data-file metadata. Keep collection independent of
column metrics policy; leave older BE fragments, empty files, ORC and
delete files without Parquet offsets.
When a file metadata-table scan requests `split_offsets`, also project
`file_size_in_bytes`: Iceberg uses the size to validate offsets and
otherwise returns null for valid values. Append this dependency after
the requested columns to preserve the BE JNI reader's positional
contract.
### Release note
Publish Parquet row-group split offsets in Iceberg manifests and
preserve them in projected file metadata-table scans.
### Validation
- Maven reactor build passed: 212 `IcebergScanPlanProviderTest` tests
and 31 `IcebergWriterHelperTest` tests, including Manifest round trips
and per-file isolation.
- The projection regression failed before the fix and passes afterward.
It covers files/data_files/all_files/all_data_files,
partitioned/unpartitioned tables and five requested-column orders.
- 6 targeted BE ASAN tests passed for dictionary/plain pages,
multiple/single/empty row groups, metrics-disabled Parquet commits and
unchanged ORC behavior. Changed source, generated Thrift and unity
objects were linked with cached engine libraries; this was not a clean
full BE rebuild.
- FE Checkstyle, clang-format 16, BE build hygiene and connector import
checks passed.
- Checked the optional Thrift field ID against master and release
branches 3.1, 4.0, 4.1 and 4.2.
- Extended the partitioned-write external regression to assert split
offsets, including with column metrics disabled. CI validation of the
latest revision and Spark/Trino interoperability remain pending.
### Check List (For Author)
- Test: Unit Test and Regression test
- Behavior changed: Yes. Newly written Iceberg Parquet data files
publish optional split offsets, which projected file metadata scans can
return.
- Does this need documentation? No.
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java
+46Lines changed: 46 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -3147,6 +3147,52 @@ public void planSystemTableScanProjectsRequestedColumnsExcludingReadableMetrics(
3147
3147
+ "BufferUnderflows on complex/boolean bound columns when materialised unrequested");
0 commit comments