Repository navigation
Commit cbf21fb
[fix](iceberg) Relax a dropped equality-delete key only along retained compatible changes
### What problem does this PR solve?
Issue Number: None
Related PR: #69536
Problem Summary: A dropped Iceberg equality-delete key takes its
definition from the scan's snapshot lineage, falling back to the whole
schema list. The previous commit then replaced the key's type with the
widest type it has in any retained schema, comparing types with
TypeUtil.isPromotionAllowed. In Iceberg 1.12.0 that helper knows only
INT to LONG, FLOAT to DOUBLE and decimal precision, so for a v3 table
whose key was promoted from DATE to an unzoned TIMESTAMP the first
retained type, DATE, won and replaced the TIMESTAMP definition the
lineage had found. FileScannerV2 compares equality keys in the carrier's
type, so it cast a noon TIMESTAMP tombstone to DATE and deleted the
midnight row of that day. A schema-only DATE to TIMESTAMP_NS promotion
of a key dropped before its delete committed kept DATE as well.
The lineage also lost a schema-only change of a required key to
optional: a pending delete and an append without the key both commit
with the post-drop schema, so the lineage reaches the original REQUIRED
definition, and both readers reject the appended file with a missing
required field, where the lookup before this PR found OPTIONAL.
Each field of the selected definition now starts from its own type and
optionality and takes, from every retained schema, only a compatible
relaxation: a promotion recognized by TypeUtil.isPromotionAllowed or the
v3 promotion of DATE to an unzoned TIMESTAMP or TIMESTAMP_NS, and
optional when any retained schema has the field optional. A key cannot
be narrowed or switched to an unrelated type, and a key required in
every retained schema stays required.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- IcebergScanPlanProviderTest: a v3 DATE key promoted to TIMESTAMP keeps the lineage's TIMESTAMP; a DATE key promoted to TIMESTAMP_NS without a snapshot and dropped before its delete commits widens to TIMESTAMP_NS; a key made optional without a snapshot and dropped before its delete commits stays optional; a key required in every schema stays required
- fe-connector-iceberg module tests
- Behavior changed: No (fixes a regression of this PR's previous commits)
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent 71f51ed commit cbf21fb
2 files changed
Lines changed: 154 additions & 30 deletions
File tree
- fe/fe-connector/fe-connector-iceberg/src
- main/java/org/apache/doris/connector/iceberg
- test/java/org/apache/doris/connector/iceberg
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2284 | 2284 | | |
2285 | 2285 | | |
2286 | 2286 | | |
2287 | | - | |
2288 | | - | |
2289 | | - | |
2290 | | - | |
2291 | | - | |
| 2287 | + | |
| 2288 | + | |
| 2289 | + | |
| 2290 | + | |
| 2291 | + | |
| 2292 | + | |
2292 | 2293 | | |
2293 | 2294 | | |
2294 | 2295 | | |
| |||
2308 | 2309 | | |
2309 | 2310 | | |
2310 | 2311 | | |
2311 | | - | |
2312 | 2312 | | |
2313 | 2313 | | |
2314 | 2314 | | |
| |||
2322 | 2322 | | |
2323 | 2323 | | |
2324 | 2324 | | |
2325 | | - | |
| 2325 | + | |
2326 | 2326 | | |
2327 | 2327 | | |
2328 | | - | |
| 2328 | + | |
2329 | 2329 | | |
2330 | 2330 | | |
2331 | 2331 | | |
| |||
2342 | 2342 | | |
2343 | 2343 | | |
2344 | 2344 | | |
2345 | | - | |
2346 | | - | |
| 2345 | + | |
| 2346 | + | |
| 2347 | + | |
| 2348 | + | |
| 2349 | + | |
2347 | 2350 | | |
2348 | | - | |
2349 | | - | |
2350 | | - | |
2351 | | - | |
2352 | | - | |
2353 | | - | |
| 2351 | + | |
| 2352 | + | |
| 2353 | + | |
| 2354 | + | |
| 2355 | + | |
| 2356 | + | |
| 2357 | + | |
| 2358 | + | |
2354 | 2359 | | |
2355 | 2360 | | |
2356 | | - | |
2357 | | - | |
2358 | | - | |
| 2361 | + | |
| 2362 | + | |
| 2363 | + | |
| 2364 | + | |
2359 | 2365 | | |
2360 | 2366 | | |
| 2367 | + | |
| 2368 | + | |
| 2369 | + | |
| 2370 | + | |
| 2371 | + | |
2361 | 2372 | | |
2362 | | - | |
| 2373 | + | |
2363 | 2374 | | |
2364 | 2375 | | |
2365 | | - | |
2366 | | - | |
2367 | | - | |
2368 | | - | |
2369 | | - | |
2370 | | - | |
2371 | | - | |
| 2376 | + | |
| 2377 | + | |
| 2378 | + | |
| 2379 | + | |
| 2380 | + | |
| 2381 | + | |
| 2382 | + | |
| 2383 | + | |
| 2384 | + | |
| 2385 | + | |
2372 | 2386 | | |
2373 | | - | |
| 2387 | + | |
| 2388 | + | |
2374 | 2389 | | |
2375 | 2390 | | |
2376 | 2391 | | |
2377 | | - | |
| 2392 | + | |
2378 | 2393 | | |
2379 | 2394 | | |
2380 | 2395 | | |
| |||
2391 | 2406 | | |
2392 | 2407 | | |
2393 | 2408 | | |
2394 | | - | |
| 2409 | + | |
2395 | 2410 | | |
2396 | 2411 | | |
2397 | 2412 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2143 | 2143 | | |
2144 | 2144 | | |
2145 | 2145 | | |
| 2146 | + | |
| 2147 | + | |
| 2148 | + | |
| 2149 | + | |
| 2150 | + | |
| 2151 | + | |
| 2152 | + | |
| 2153 | + | |
| 2154 | + | |
| 2155 | + | |
| 2156 | + | |
| 2157 | + | |
| 2158 | + | |
| 2159 | + | |
| 2160 | + | |
| 2161 | + | |
| 2162 | + | |
| 2163 | + | |
| 2164 | + | |
| 2165 | + | |
| 2166 | + | |
| 2167 | + | |
| 2168 | + | |
| 2169 | + | |
| 2170 | + | |
| 2171 | + | |
| 2172 | + | |
| 2173 | + | |
| 2174 | + | |
| 2175 | + | |
| 2176 | + | |
| 2177 | + | |
| 2178 | + | |
| 2179 | + | |
| 2180 | + | |
| 2181 | + | |
| 2182 | + | |
| 2183 | + | |
| 2184 | + | |
| 2185 | + | |
| 2186 | + | |
| 2187 | + | |
| 2188 | + | |
| 2189 | + | |
| 2190 | + | |
| 2191 | + | |
| 2192 | + | |
| 2193 | + | |
| 2194 | + | |
| 2195 | + | |
| 2196 | + | |
| 2197 | + | |
| 2198 | + | |
| 2199 | + | |
| 2200 | + | |
| 2201 | + | |
| 2202 | + | |
| 2203 | + | |
| 2204 | + | |
| 2205 | + | |
| 2206 | + | |
| 2207 | + | |
| 2208 | + | |
| 2209 | + | |
| 2210 | + | |
| 2211 | + | |
| 2212 | + | |
| 2213 | + | |
| 2214 | + | |
| 2215 | + | |
| 2216 | + | |
| 2217 | + | |
| 2218 | + | |
| 2219 | + | |
| 2220 | + | |
| 2221 | + | |
| 2222 | + | |
| 2223 | + | |
| 2224 | + | |
| 2225 | + | |
| 2226 | + | |
| 2227 | + | |
| 2228 | + | |
| 2229 | + | |
| 2230 | + | |
| 2231 | + | |
| 2232 | + | |
| 2233 | + | |
| 2234 | + | |
| 2235 | + | |
| 2236 | + | |
| 2237 | + | |
| 2238 | + | |
| 2239 | + | |
| 2240 | + | |
| 2241 | + | |
| 2242 | + | |
| 2243 | + | |
| 2244 | + | |
| 2245 | + | |
| 2246 | + | |
| 2247 | + | |
| 2248 | + | |
| 2249 | + | |
| 2250 | + | |
| 2251 | + | |
| 2252 | + | |
| 2253 | + | |
| 2254 | + | |
2146 | 2255 | | |
2147 | 2256 | | |
2148 | 2257 | | |
| |||
0 commit comments