Repository navigation
Commit 7604fbf
authored
[fix](be) Fix Iceberg row-level deletes across schema evolution (apache#65502)
Problem Summary:
Iceberg row-level delete handling had correctness and efficiency gaps across the V1 and V2 file scanners:
FileScannerV2 applied delete expressions with ordinary predicate polarity, keeping rows whose delete predicate evaluated to true instead of removing them.
Equality-delete keys could not be resolved when they were hidden from the query projection, renamed through Iceberg name mapping, absent from files written with older schemas, or stored in files without physical field ids.
Missing equality keys were rejected instead of being materialized with their Iceberg initial default, or NULL when no initial default exists.
Binary-like initial defaults could lose bytes in string transport, timestamp defaults could be parsed with the wrong mapping, and time-travel scans could use metadata from the latest schema instead of the selected snapshot schema.
Equality and position delete files shared by multiple data splits were reparsed for every split, and position bounds were not applied.
Complex-column rematerialization could trust an incomplete nested descriptor and produce a child type whose nullability disagreed with its parent Struct type. Applying that correction to Array/Map wrapper children could also trigger a fatal child-name mismatch.
This change:
Builds an explicit keep filter by inverting delete matches and combines multiple delete files with the correct union semantics.
Sends complete Iceberg schema metadata from FE to BE, including historical name mapping and losslessly encoded initial defaults from the selected snapshot schema.
Resolves equality keys consistently by field id or BY_NAME mapping in both V1 and V2 scanners, and materializes missing keys with their typed initial default or NULL.
Supports all-literal equality predicates by materializing constant keys to the current batch size.
Propagates TIMESTAMPTZ mapping to delete-file readers.
Caches parsed equality and position delete data in SplitReadOptions.cache, with filesystem, path, and equality-field layout in the cache identity, and applies inclusive position bounds.
Uses the parent Struct type as the authoritative contract for child nullability while preserving Array and Map wrapper semantics.1 parent a37667d commit 7604fbf
28 files changed
Lines changed: 2712 additions & 269 deletions
File tree
- be
- src
- format_v2
- expr
- table
- format/table
- test
- format_v2
- expr
- json
- table
- format/table/iceberg
- fe/fe-core/src
- main/java/org/apache/doris/datasource
- iceberg
- source
- test/java/org/apache/doris/datasource
- iceberg
- source
- gensrc/thrift
- regression-test/suites/external_table_p0/iceberg
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
105 | 105 | | |
106 | 106 | | |
107 | 107 | | |
108 | | - | |
109 | | - | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
110 | 111 | | |
111 | 112 | | |
112 | 113 | | |
| |||
151 | 152 | | |
152 | 153 | | |
153 | 154 | | |
154 | | - | |
155 | | - | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
156 | 158 | | |
157 | 159 | | |
158 | 160 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
20 | 22 | | |
21 | 23 | | |
22 | 24 | | |
| 25 | + | |
23 | 26 | | |
24 | 27 | | |
25 | 28 | | |
| |||
33 | 36 | | |
34 | 37 | | |
35 | 38 | | |
| 39 | + | |
36 | 40 | | |
37 | 41 | | |
38 | 42 | | |
39 | 43 | | |
40 | 44 | | |
41 | 45 | | |
42 | 46 | | |
| 47 | + | |
43 | 48 | | |
44 | 49 | | |
45 | 50 | | |
| |||
135 | 140 | | |
136 | 141 | | |
137 | 142 | | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
138 | 157 | | |
139 | 158 | | |
140 | 159 | | |
| |||
251 | 270 | | |
252 | 271 | | |
253 | 272 | | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
254 | 278 | | |
255 | 279 | | |
256 | 280 | | |
| |||
352 | 376 | | |
353 | 377 | | |
354 | 378 | | |
| 379 | + | |
355 | 380 | | |
356 | 381 | | |
357 | 382 | | |
| |||
624 | 649 | | |
625 | 650 | | |
626 | 651 | | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
627 | 657 | | |
628 | 658 | | |
629 | 659 | | |
| |||
634 | 664 | | |
635 | 665 | | |
636 | 666 | | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
637 | 779 | | |
638 | 780 | | |
639 | 781 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
258 | 258 | | |
259 | 259 | | |
260 | 260 | | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
261 | 267 | | |
262 | 268 | | |
263 | 269 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
230 | 230 | | |
231 | 231 | | |
232 | 232 | | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
233 | 238 | | |
234 | 239 | | |
235 | 240 | | |
| |||
1237 | 1242 | | |
1238 | 1243 | | |
1239 | 1244 | | |
| 1245 | + | |
| 1246 | + | |
| 1247 | + | |
| 1248 | + | |
| 1249 | + | |
| 1250 | + | |
| 1251 | + | |
| 1252 | + | |
| 1253 | + | |
| 1254 | + | |
| 1255 | + | |
| 1256 | + | |
| 1257 | + | |
| 1258 | + | |
| 1259 | + | |
| 1260 | + | |
| 1261 | + | |
| 1262 | + | |
| 1263 | + | |
| 1264 | + | |
1240 | 1265 | | |
1241 | 1266 | | |
1242 | 1267 | | |
| |||
1245 | 1270 | | |
1246 | 1271 | | |
1247 | 1272 | | |
1248 | | - | |
1249 | | - | |
| 1273 | + | |
1250 | 1274 | | |
1251 | 1275 | | |
1252 | 1276 | | |
| |||
1261 | 1285 | | |
1262 | 1286 | | |
1263 | 1287 | | |
1264 | | - | |
1265 | | - | |
| 1288 | + | |
| 1289 | + | |
1266 | 1290 | | |
1267 | 1291 | | |
1268 | 1292 | | |
| |||
1279 | 1303 | | |
1280 | 1304 | | |
1281 | 1305 | | |
| 1306 | + | |
| 1307 | + | |
| 1308 | + | |
| 1309 | + | |
| 1310 | + | |
| 1311 | + | |
| 1312 | + | |
| 1313 | + | |
| 1314 | + | |
| 1315 | + | |
| 1316 | + | |
| 1317 | + | |
| 1318 | + | |
| 1319 | + | |
| 1320 | + | |
| 1321 | + | |
| 1322 | + | |
| 1323 | + | |
| 1324 | + | |
| 1325 | + | |
| 1326 | + | |
| 1327 | + | |
| 1328 | + | |
| 1329 | + | |
| 1330 | + | |
| 1331 | + | |
| 1332 | + | |
| 1333 | + | |
| 1334 | + | |
| 1335 | + | |
| 1336 | + | |
1282 | 1337 | | |
1283 | 1338 | | |
1284 | 1339 | | |
| |||
2016 | 2071 | | |
2017 | 2072 | | |
2018 | 2073 | | |
2019 | | - | |
2020 | | - | |
2021 | | - | |
2022 | | - | |
2023 | | - | |
2024 | | - | |
2025 | | - | |
2026 | | - | |
2027 | | - | |
2028 | | - | |
2029 | | - | |
2030 | | - | |
2031 | | - | |
2032 | | - | |
2033 | | - | |
2034 | | - | |
2035 | | - | |
2036 | | - | |
2037 | | - | |
2038 | | - | |
2039 | | - | |
2040 | | - | |
2041 | | - | |
| 2074 | + | |
| 2075 | + | |
2042 | 2076 | | |
2043 | 2077 | | |
2044 | 2078 | | |
| |||
0 commit comments