Repository navigation
Commit abf1e6e
fix: only propagate cast statistics through safe conversions (apache#25227)
## Which issue does this PR close?
- Closes apache#25221.
## Rationale for this change
`MIN`/`MAX` over a cast can return incorrect results when aggregate
optimization replaces the scan with converted column statistics. Casting
the original endpoints is only valid if they remain extrema in the
target domain.
For example, a Parquet string column containing `('1', '100', '2')` has
string extrema `'1'` and `'2'`. Previously, `MIN(CAST(a AS INT)),
MAX(CAST(a AS INT))` could return `1, 2` from those statistics instead
of the correct `1, 100`. The same issue affects `BIGINT`.
## What changes are included in this PR?
- Restrict CAST statistics propagation to conversions proven safe for
the source type or its exact integer bounds. Unsupported or unsafe
conversions return unknown column statistics, so aggregates evaluate the
data.
- Preserve the existing same-type passthrough and supported lossless
widening conversions.
- Retain integer narrowing and signed/unsigned conversions when both
original bounds are exact and non-null, and both converted endpoints are
non-null. Successful endpoint conversions establish that the entire
integer range fits in the target type; overflow errors or null endpoints
do not qualify.
- Extend `CastExpr::check_bigger_cast` to recognize `Int32 ↔ Date32`,
and clarify that its contract includes lossless, strictly
order-preserving conversions that preserve nulls. Arrow reinterprets the
same `i32` values as days since the epoch, so these conversions preserve
both statistics and ordering properties. This also retains the
statistics-based `MIN`/`MAX` optimization for ClickBench's `UInt16 →
Int32 → Date32` projection.
## What is the testing strategy for this PR?
- Add projection unit tests for non-monotonic string-to-integer casts
and integer narrowing with safe, overflowing, and inexact bounds.
Removing the statistics guard makes the new string-cast regression test
fail.
- Add end-to-end Parquet regression cases in `parquet_statistics.slt`
for `INT` and `BIGINT`, both expecting `1, 100`.
- Add a bidirectional `Int32 ↔ Date32` test covering `i32::MIN`,
`i32::MAX`, negative values, zero, NULL, and strict ordering properties.
- Run the related projection and cast unit tests, plus the ClickBench
and Parquet statistics SLT files. The existing ClickBench plan
expectation passes unchanged, confirming that its aggregate still folds
to constants.
Local formatting, Clippy, license, spelling, workflow, and documentation
checks also pass.
## Are there any user-facing changes?
Affected casts now produce correct `MIN`/`MAX` results. Conversions
whose safety is not established may require a scan instead of using
statistics-based aggregate optimization. Supported safe integer
conversions and `Int32 ↔ Date32` retain that optimization.1 parent 2949730 commit abf1e6e
3 files changed
Lines changed: 395 additions & 13 deletions
File tree
- datafusion
- physical-expr/src
- expressions
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
248 | 248 | | |
249 | 249 | | |
250 | 250 | | |
251 | | - | |
252 | | - | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
253 | 257 | | |
254 | 258 | | |
255 | 259 | | |
| |||
259 | 263 | | |
260 | 264 | | |
261 | 265 | | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
262 | 270 | | |
263 | 271 | | |
264 | 272 | | |
| |||
268 | 276 | | |
269 | 277 | | |
270 | 278 | | |
271 | | - | |
| 279 | + | |
| 280 | + | |
272 | 281 | | |
273 | 282 | | |
274 | 283 | | |
| |||
289 | 298 | | |
290 | 299 | | |
291 | 300 | | |
292 | | - | |
293 | | - | |
| 301 | + | |
| 302 | + | |
294 | 303 | | |
295 | 304 | | |
296 | 305 | | |
| |||
1505 | 1514 | | |
1506 | 1515 | | |
1507 | 1516 | | |
| 1517 | + | |
| 1518 | + | |
| 1519 | + | |
| 1520 | + | |
| 1521 | + | |
| 1522 | + | |
| 1523 | + | |
| 1524 | + | |
| 1525 | + | |
| 1526 | + | |
| 1527 | + | |
| 1528 | + | |
| 1529 | + | |
| 1530 | + | |
| 1531 | + | |
| 1532 | + | |
| 1533 | + | |
| 1534 | + | |
| 1535 | + | |
| 1536 | + | |
| 1537 | + | |
| 1538 | + | |
| 1539 | + | |
| 1540 | + | |
| 1541 | + | |
| 1542 | + | |
| 1543 | + | |
| 1544 | + | |
| 1545 | + | |
| 1546 | + | |
| 1547 | + | |
| 1548 | + | |
| 1549 | + | |
| 1550 | + | |
| 1551 | + | |
| 1552 | + | |
| 1553 | + | |
| 1554 | + | |
| 1555 | + | |
| 1556 | + | |
| 1557 | + | |
| 1558 | + | |
| 1559 | + | |
| 1560 | + | |
| 1561 | + | |
| 1562 | + | |
| 1563 | + | |
| 1564 | + | |
| 1565 | + | |
| 1566 | + | |
| 1567 | + | |
| 1568 | + | |
| 1569 | + | |
| 1570 | + | |
1508 | 1571 | | |
1509 | 1572 | | |
1510 | 1573 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
872 | 872 | | |
873 | 873 | | |
874 | 874 | | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
875 | 905 | | |
876 | | - | |
877 | | - | |
878 | | - | |
879 | | - | |
880 | | - | |
881 | | - | |
882 | | - | |
883 | | - | |
| 906 | + | |
| 907 | + | |
884 | 908 | | |
885 | 909 | | |
886 | 910 | | |
887 | 911 | | |
888 | 912 | | |
889 | 913 | | |
890 | 914 | | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
891 | 932 | | |
892 | 933 | | |
893 | 934 | | |
| |||
2249 | 2290 | | |
2250 | 2291 | | |
2251 | 2292 | | |
| 2293 | + | |
| 2294 | + | |
| 2295 | + | |
| 2296 | + | |
| 2297 | + | |
| 2298 | + | |
| 2299 | + | |
| 2300 | + | |
| 2301 | + | |
| 2302 | + | |
| 2303 | + | |
| 2304 | + | |
| 2305 | + | |
| 2306 | + | |
| 2307 | + | |
| 2308 | + | |
| 2309 | + | |
| 2310 | + | |
| 2311 | + | |
| 2312 | + | |
| 2313 | + | |
| 2314 | + | |
| 2315 | + | |
| 2316 | + | |
| 2317 | + | |
| 2318 | + | |
| 2319 | + | |
| 2320 | + | |
| 2321 | + | |
| 2322 | + | |
| 2323 | + | |
| 2324 | + | |
| 2325 | + | |
| 2326 | + | |
| 2327 | + | |
| 2328 | + | |
| 2329 | + | |
| 2330 | + | |
| 2331 | + | |
| 2332 | + | |
| 2333 | + | |
| 2334 | + | |
| 2335 | + | |
| 2336 | + | |
| 2337 | + | |
| 2338 | + | |
| 2339 | + | |
| 2340 | + | |
| 2341 | + | |
| 2342 | + | |
| 2343 | + | |
| 2344 | + | |
| 2345 | + | |
| 2346 | + | |
| 2347 | + | |
| 2348 | + | |
| 2349 | + | |
| 2350 | + | |
| 2351 | + | |
| 2352 | + | |
| 2353 | + | |
| 2354 | + | |
| 2355 | + | |
| 2356 | + | |
| 2357 | + | |
| 2358 | + | |
| 2359 | + | |
| 2360 | + | |
| 2361 | + | |
| 2362 | + | |
| 2363 | + | |
| 2364 | + | |
| 2365 | + | |
| 2366 | + | |
| 2367 | + | |
| 2368 | + | |
| 2369 | + | |
| 2370 | + | |
| 2371 | + | |
| 2372 | + | |
| 2373 | + | |
| 2374 | + | |
| 2375 | + | |
| 2376 | + | |
| 2377 | + | |
| 2378 | + | |
| 2379 | + | |
| 2380 | + | |
| 2381 | + | |
| 2382 | + | |
| 2383 | + | |
| 2384 | + | |
| 2385 | + | |
| 2386 | + | |
| 2387 | + | |
| 2388 | + | |
| 2389 | + | |
| 2390 | + | |
| 2391 | + | |
| 2392 | + | |
| 2393 | + | |
| 2394 | + | |
| 2395 | + | |
| 2396 | + | |
| 2397 | + | |
| 2398 | + | |
| 2399 | + | |
| 2400 | + | |
| 2401 | + | |
| 2402 | + | |
| 2403 | + | |
| 2404 | + | |
| 2405 | + | |
| 2406 | + | |
| 2407 | + | |
| 2408 | + | |
| 2409 | + | |
| 2410 | + | |
| 2411 | + | |
| 2412 | + | |
| 2413 | + | |
| 2414 | + | |
| 2415 | + | |
| 2416 | + | |
| 2417 | + | |
| 2418 | + | |
| 2419 | + | |
| 2420 | + | |
| 2421 | + | |
| 2422 | + | |
| 2423 | + | |
| 2424 | + | |
| 2425 | + | |
| 2426 | + | |
| 2427 | + | |
| 2428 | + | |
| 2429 | + | |
| 2430 | + | |
| 2431 | + | |
| 2432 | + | |
| 2433 | + | |
| 2434 | + | |
| 2435 | + | |
| 2436 | + | |
| 2437 | + | |
| 2438 | + | |
| 2439 | + | |
| 2440 | + | |
| 2441 | + | |
| 2442 | + | |
| 2443 | + | |
| 2444 | + | |
| 2445 | + | |
| 2446 | + | |
| 2447 | + | |
| 2448 | + | |
| 2449 | + | |
| 2450 | + | |
| 2451 | + | |
| 2452 | + | |
| 2453 | + | |
| 2454 | + | |
| 2455 | + | |
| 2456 | + | |
| 2457 | + | |
| 2458 | + | |
| 2459 | + | |
| 2460 | + | |
| 2461 | + | |
| 2462 | + | |
| 2463 | + | |
| 2464 | + | |
| 2465 | + | |
| 2466 | + | |
| 2467 | + | |
| 2468 | + | |
| 2469 | + | |
| 2470 | + | |
| 2471 | + | |
| 2472 | + | |
| 2473 | + | |
| 2474 | + | |
| 2475 | + | |
| 2476 | + | |
| 2477 | + | |
| 2478 | + | |
| 2479 | + | |
| 2480 | + | |
| 2481 | + | |
| 2482 | + | |
| 2483 | + | |
2252 | 2484 | | |
2253 | 2485 | | |
2254 | 2486 | | |
| |||
0 commit comments