Repository navigation
Commit 0814e49
authored
[feature](iceberg) Support Iceberg position deletes system table (#65135)
Checkpoint conclusions:
Goal and scope: the patch adds native Iceberg $position_deletes metadata-table support across FE system-table resolution, split construction, Thrift scan metadata, BE scanner-v1/scanner-v2 readers, regression coverage, and unit tests. I did not find an out-of-scope behavioral change needing a new comment.
Existing review context: prior comments already covered the earlier V1 IOContext/profile issue, smooth-upgrade guard, Avro unsupported format, partition JSON/type/evolution/rename concerns, and the ORC row assertion typo. I did not duplicate those threads.
FE/BE protocol and compatibility: native $position_deletes ranges use top-level Iceberg content to route into the new readers, while ordinary Iceberg data ranges keep delete-file descriptors nested and do not match that route. The smooth-upgrade backend guard now runs before native position-delete ranges are generated.
Data correctness and nullability: partition values are mapped by Iceberg partition field ID into the metadata-table partition struct, missing evolved fields become null, unsupported non-null binary/fixed/UUID partition values fail instead of silently producing wrong data, optional absent row payloads materialize as null, and deletion-vector rows are expanded from the referenced data file plus decoded positions.
Scanner and lifecycle paths: scanner V1/V2 routing, Parquet/ORC position-delete files, Puffin deletion vectors, EOF/final-batch handling, IOContext propagation, profile/stat accounting, close/reset state, and row-count behavior were reviewed without a substantiated new defect.
Tests and coverage: the changed regression and unit tests cover the main feature surface, including scanner V1/V2, Parquet/ORC deletes, V3 deletion-vector metadata, partition evolution/rename, typed partitions, time travel, empty-delete tables, and privilege checks. A possible Puffin-DV split concern was dismissed because Doris depends on Iceberg 1.10.1, where FileFormat.PUFFIN is non-splittable and BaseContentScanTask.split(...) returns the original task for non-splittable files.
Concurrency, persistence, transactions, and config: no new shared mutable concurrency path, edit-log/persistence path, transaction/write path, or user-facing config was introduced by this read-only metadata scan feature.
Observability/performance: existing scanner/profile counters remain in use; I did not find a concrete additional metrics requirement or hot-path regression.1 parent f5c30d7 commit 0814e49
18 files changed
Lines changed: 3218 additions & 46 deletions
File tree
- be
- src
- exec/scan
- format_v2/table
- format/table
- test
- exec/scan
- format/table/iceberg
- fe/fe-core/src
- main/java/org/apache/doris/datasource
- iceberg
- source
- systable
- test/java/org/apache/doris/datasource
- iceberg/source
- systable
- gensrc/thrift
- regression-test
- data/external_table_p0/iceberg
- suites/external_table_p0/iceberg
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
| 72 | + | |
72 | 73 | | |
73 | 74 | | |
74 | 75 | | |
| |||
98 | 99 | | |
99 | 100 | | |
100 | 101 | | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
101 | 116 | | |
102 | 117 | | |
103 | 118 | | |
| |||
1043 | 1058 | | |
1044 | 1059 | | |
1045 | 1060 | | |
| 1061 | + | |
1046 | 1062 | | |
1047 | 1063 | | |
1048 | 1064 | | |
| |||
1168 | 1184 | | |
1169 | 1185 | | |
1170 | 1186 | | |
| 1187 | + | |
| 1188 | + | |
| 1189 | + | |
| 1190 | + | |
| 1191 | + | |
| 1192 | + | |
| 1193 | + | |
| 1194 | + | |
| 1195 | + | |
| 1196 | + | |
| 1197 | + | |
1171 | 1198 | | |
1172 | 1199 | | |
1173 | 1200 | | |
| |||
1180 | 1207 | | |
1181 | 1208 | | |
1182 | 1209 | | |
| 1210 | + | |
| 1211 | + | |
| 1212 | + | |
| 1213 | + | |
| 1214 | + | |
| 1215 | + | |
| 1216 | + | |
| 1217 | + | |
| 1218 | + | |
| 1219 | + | |
| 1220 | + | |
1183 | 1221 | | |
1184 | 1222 | | |
1185 | 1223 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
| 58 | + | |
58 | 59 | | |
59 | 60 | | |
60 | 61 | | |
| |||
71 | 72 | | |
72 | 73 | | |
73 | 74 | | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
74 | 78 | | |
75 | 79 | | |
76 | 80 | | |
| |||
111 | 115 | | |
112 | 116 | | |
113 | 117 | | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
114 | 127 | | |
115 | 128 | | |
116 | 129 | | |
| |||
473 | 486 | | |
474 | 487 | | |
475 | 488 | | |
476 | | - | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
477 | 492 | | |
478 | 493 | | |
479 | 494 | | |
| |||
0 commit comments