You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d8e75cf
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](iceberg) Project Iceberg system table scans (#65262)
Iceberg system table scans serialize Iceberg SDK `FileScanTask` objects
and the Java metadata scanner materializes rows from the task schema.
Doris did not pass the actually required system table columns into the
Iceberg SDK scan, so metadata tables such as `$files` and `$data_files`
were planned with the full schema. For files metadata tables, the full
schema can include virtual `readable_metrics` even when the SQL only
requests columns such as `file_size_in_bytes`.
This PR applies SDK top-level column projection only for Iceberg system
table scans. Normal Iceberg data table scans continue to rely on Doris
BE scan range params for column pruning.
### What changed?
- Add system-table-only `TableScan.select(...)` projection in
`IcebergScanNode` before planning the Iceberg SDK scan.
- Skip synthesized/global row id and Iceberg row lineage columns when
building the SDK projection list.
- Add regression coverage for ORC and Parquet Iceberg tables with
`map<boolean, boolean>` columns, verifying projected `$data_files` and
`$files` queries.
Copy file name to clipboardExpand all lines: fe/be-java-extensions/iceberg-metadata-scanner/src/main/java/org/apache/doris/iceberg/IcebergSysTableJniScanner.java
+18-39Lines changed: 18 additions & 39 deletions
Original file line number
Diff line number
Diff line change
@@ -28,15 +28,11 @@
28
28
importorg.apache.iceberg.FileScanTask;
29
29
importorg.apache.iceberg.StructLike;
30
30
importorg.apache.iceberg.io.CloseableIterator;
31
-
importorg.apache.iceberg.types.Types.NestedField;
32
-
importorg.apache.iceberg.types.Types.StructType;
33
31
importorg.apache.iceberg.util.SerializationUtil;
34
32
importorg.slf4j.Logger;
35
33
importorg.slf4j.LoggerFactory;
36
34
37
35
importjava.io.IOException;
38
-
importjava.util.ArrayList;
39
-
importjava.util.List;
40
36
importjava.util.Map;
41
37
importjava.util.TimeZone;
42
38
importjava.util.stream.Collectors;
@@ -50,7 +46,7 @@ public class IcebergSysTableJniScanner extends JniScanner {
0 commit comments