Repository navigation
Commit 71f51ed
[fix](iceberg) Keep the promoted type of a dropped equality-delete key
### What problem does this PR solve?
Issue Number: None
Related PR: #66538
Problem Summary: Review follow-up. The previous commit resolves an
equality-delete key that the scan schema no longer has from the scan
snapshot's parent chain first. With Iceberg 1.12 a pending RowDelta is
stamped with the schema that is current when it commits, and nothing checks
its equality field IDs against that schema. So for a key k promoted from INT
to LONG by a schema-only change (no snapshot), a writer that wrote LONG
tombstones, and k dropped before that delete commits, the delete snapshot
records the schema without k and its parent the INT one: no snapshot on the
lineage records LONG. The carrier then said INT, FileScannerV2 compares
equality keys in the carrier's type, and a tombstone such as 2147483648 was
cast to NULL in non-strict mode and deleted every row whose k is NULL
(strict mode failed the query). The previous metadata-list lookup found
LONG.
A dropped key now keeps its lineage definition (name, optionality, initial
default) but takes the widest type its field ID has in any retained schema.
Field IDs are never reused and promotion is monotonic and lossless, so that
type covers every file written for the field, also when it comes from a
schema after a time-travel target.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- IcebergScanPlanProviderTest#equalityCarrierKeepsPromotedTypeOfKeyDroppedBeforeItsDeleteCommits replays the sequence with the SDK; it fails without the widening
- fe-connector-iceberg module tests
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent dc21748 commit 71f51ed
2 files changed
Lines changed: 74 additions & 9 deletions
File tree
- fe/fe-connector/fe-connector-iceberg/src
- main/java/org/apache/doris/connector/iceberg
- test/java/org/apache/doris/connector/iceberg
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2284 | 2284 | | |
2285 | 2285 | | |
2286 | 2286 | | |
2287 | | - | |
2288 | | - | |
2289 | | - | |
| 2287 | + | |
| 2288 | + | |
| 2289 | + | |
| 2290 | + | |
| 2291 | + | |
2290 | 2292 | | |
2291 | 2293 | | |
2292 | 2294 | | |
| |||
2305 | 2307 | | |
2306 | 2308 | | |
2307 | 2309 | | |
| 2310 | + | |
| 2311 | + | |
2308 | 2312 | | |
2309 | 2313 | | |
2310 | 2314 | | |
| |||
2318 | 2322 | | |
2319 | 2323 | | |
2320 | 2324 | | |
2321 | | - | |
| 2325 | + | |
2322 | 2326 | | |
2323 | | - | |
2324 | 2327 | | |
2325 | | - | |
| 2328 | + | |
2326 | 2329 | | |
2327 | 2330 | | |
2328 | 2331 | | |
| |||
2338 | 2341 | | |
2339 | 2342 | | |
2340 | 2343 | | |
2341 | | - | |
2342 | | - | |
| 2344 | + | |
| 2345 | + | |
| 2346 | + | |
| 2347 | + | |
| 2348 | + | |
| 2349 | + | |
| 2350 | + | |
| 2351 | + | |
| 2352 | + | |
| 2353 | + | |
| 2354 | + | |
| 2355 | + | |
| 2356 | + | |
| 2357 | + | |
| 2358 | + | |
| 2359 | + | |
| 2360 | + | |
| 2361 | + | |
| 2362 | + | |
| 2363 | + | |
| 2364 | + | |
| 2365 | + | |
| 2366 | + | |
| 2367 | + | |
| 2368 | + | |
| 2369 | + | |
| 2370 | + | |
| 2371 | + | |
| 2372 | + | |
| 2373 | + | |
| 2374 | + | |
| 2375 | + | |
| 2376 | + | |
| 2377 | + | |
2343 | 2378 | | |
2344 | 2379 | | |
2345 | 2380 | | |
| |||
2356 | 2391 | | |
2357 | 2392 | | |
2358 | 2393 | | |
2359 | | - | |
| 2394 | + | |
2360 | 2395 | | |
2361 | 2396 | | |
2362 | 2397 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
| 65 | + | |
65 | 66 | | |
66 | 67 | | |
67 | 68 | | |
| |||
2113 | 2114 | | |
2114 | 2115 | | |
2115 | 2116 | | |
| 2117 | + | |
| 2118 | + | |
| 2119 | + | |
| 2120 | + | |
| 2121 | + | |
| 2122 | + | |
| 2123 | + | |
| 2124 | + | |
| 2125 | + | |
| 2126 | + | |
| 2127 | + | |
| 2128 | + | |
| 2129 | + | |
| 2130 | + | |
| 2131 | + | |
| 2132 | + | |
| 2133 | + | |
| 2134 | + | |
| 2135 | + | |
| 2136 | + | |
| 2137 | + | |
| 2138 | + | |
| 2139 | + | |
| 2140 | + | |
| 2141 | + | |
| 2142 | + | |
| 2143 | + | |
| 2144 | + | |
| 2145 | + | |
2116 | 2146 | | |
2117 | 2147 | | |
2118 | 2148 | | |
| |||
0 commit comments