Repository navigation
Commit bf278a5
authored
[feature](iceberg) Support whole-file metadata delete for DELETE (apache#69176)
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
For an Iceberg table, a row-level DELETE always scans the matching rows
and writes delete files (position deletes on v2, deletion vectors on
v3), so a data file whose rows are all gone stays in the snapshot and
every later scan still reads it.
This PR adds the whole-file "metadata delete" fast path that Spark /
Trino use for whole-partition and whole-file predicates: the DELETE is
committed as one metadata operation that drops the matching data files,
so the statement reads no rows, writes no delete files, and later scans
no longer read those files.
- The engine only tries the fast path when the statement predicate can
be extracted exactly (single-table DELETE, leaf is the target table
scan, every conjunct representable). Anything else keeps the current
row-level path. `PARTITION(...)` scoped deletes, UPDATE and MERGE are
unchanged.
- The Iceberg connector lowers the predicate with the strictly
all-or-nothing rewrite matrix and only commits when every candidate file
is matched in full (whole-partition shortcut, strict partition
projection or file metrics); otherwise it falls back to the row-level
delete, so a partially matched file is never dropped.
- Like the engines above, the committed snapshot applies the predicate
to the current table state, and the reported row count is iceberg's
`deleted-records` minus the rows the removed delete files had already
hidden.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label1 parent 883bf91 commit bf278a5
21 files changed
Lines changed: 1629 additions & 46 deletions
File tree
- fe
- fe-connector
- fe-connector-hive/src
- main/java/org/apache/doris/connector/hive
- test/java/org/apache/doris/connector/hive
- fe-connector-iceberg/src
- main/java/org/apache/doris/connector/iceberg
- test/java/org/apache/doris/connector/iceberg
- fe-connector-spi/src
- main/java/org/apache/doris/connector/spi
- test
- java/org/apache/doris/connector/spi
- resources
- fe-core/src
- main/java/org/apache/doris
- datasource/connector/converter
- nereids/trees/plans/commands
- test/java/org/apache/doris
- datasource/connector/converter
- nereids/trees/plans/commands
- regression-test
- data/external_table_p0/iceberg/write
- suites/external_table_p0/iceberg
- write
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
100 | 100 | | |
101 | 101 | | |
102 | 102 | | |
| 103 | + | |
103 | 104 | | |
104 | 105 | | |
105 | 106 | | |
| |||
2149 | 2150 | | |
2150 | 2151 | | |
2151 | 2152 | | |
| 2153 | + | |
| 2154 | + | |
| 2155 | + | |
| 2156 | + | |
| 2157 | + | |
| 2158 | + | |
| 2159 | + | |
| 2160 | + | |
| 2161 | + | |
| 2162 | + | |
| 2163 | + | |
| 2164 | + | |
| 2165 | + | |
| 2166 | + | |
| 2167 | + | |
2152 | 2168 | | |
2153 | 2169 | | |
2154 | 2170 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
| 73 | + | |
73 | 74 | | |
74 | 75 | | |
75 | 76 | | |
| |||
356 | 357 | | |
357 | 358 | | |
358 | 359 | | |
359 | | - | |
| 360 | + | |
360 | 361 | | |
361 | 362 | | |
362 | | - | |
363 | | - | |
364 | | - | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
365 | 366 | | |
366 | 367 | | |
367 | 368 | | |
| |||
387 | 388 | | |
388 | 389 | | |
389 | 390 | | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
390 | 395 | | |
391 | 396 | | |
392 | 397 | | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
393 | 406 | | |
394 | 407 | | |
395 | 408 | | |
| |||
451 | 464 | | |
452 | 465 | | |
453 | 466 | | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
454 | 470 | | |
455 | 471 | | |
456 | 472 | | |
| |||
721 | 737 | | |
722 | 738 | | |
723 | 739 | | |
| 740 | + | |
724 | 741 | | |
725 | 742 | | |
726 | 743 | | |
| |||
747 | 764 | | |
748 | 765 | | |
749 | 766 | | |
750 | | - | |
| 767 | + | |
| 768 | + | |
751 | 769 | | |
752 | 770 | | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
753 | 774 | | |
754 | 775 | | |
755 | 776 | | |
| |||
1069 | 1090 | | |
1070 | 1091 | | |
1071 | 1092 | | |
| 1093 | + | |
| 1094 | + | |
| 1095 | + | |
| 1096 | + | |
| 1097 | + | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
| 1103 | + | |
1072 | 1104 | | |
1073 | 1105 | | |
0 commit comments