Repository navigation
Commit 9ff4bf5
fix: only propagate cast statistics through safe conversions (apache#25227)
## Which issue does this PR close?
- Closes apache#25221.
## Rationale for this change
`MIN`/`MAX` over a cast can return incorrect results when aggregate
optimization replaces the scan with converted column statistics. Casting
the original endpoints is only valid if they remain extrema in the
target domain.
For example, a Parquet string column containing `('1', '100', '2')` has
string extrema `'1'` and `'2'`. Previously, `MIN(CAST(a AS INT)),
MAX(CAST(a AS INT))` could return `1, 2` from those statistics instead
of the correct `1, 100`. The same issue affects `BIGINT`.
## What changes are included in this PR?
- Restrict CAST statistics propagation to conversions proven safe for
the source type or its exact integer bounds. Unsupported or unsafe
conversions return unknown column statistics, so aggregates evaluate the
data.
- Preserve the existing same-type passthrough and supported lossless
widening conversions.
- Retain integer narrowing and signed/unsigned conversions when both
original bounds are exact and non-null, and both converted endpoints are
non-null. Successful endpoint conversions establish that the entire
integer range fits in the target type; overflow errors or null endpoints
do not qualify.
- Extend `CastExpr::check_bigger_cast` to recognize `Int32 ↔ Date32`,
and clarify that its contract includes lossless, strictly
order-preserving conversions that preserve nulls. Arrow reinterprets the
same `i32` values as days since the epoch, so these conversions preserve
both statistics and ordering properties. This also retains the
statistics-based `MIN`/`MAX` optimization for ClickBench's `UInt16 →
Int32 → Date32` projection.
## What is the testing strategy for this PR?
- Add projection unit tests for non-monotonic string-to-integer casts
and integer narrowing with safe, overflowing, and inexact bounds.
Removing the statistics guard makes the new string-cast regression test
fail.
- Add end-to-end Parquet regression cases in `parquet_statistics.slt`
for `INT` and `BIGINT`, both expecting `1, 100`.
- Add a bidirectional `Int32 ↔ Date32` test covering `i32::MIN`,
`i32::MAX`, negative values, zero, NULL, and strict ordering properties.
- Run the related projection and cast unit tests, plus the ClickBench
and Parquet statistics SLT files. The existing ClickBench plan
expectation passes unchanged, confirming that its aggregate still folds
to constants.
Local formatting, Clippy, license, spelling, workflow, and documentation
checks also pass.
## Are there any user-facing changes?
Affected casts now produce correct `MIN`/`MAX` results. Conversions
whose safety is not established may require a scan instead of using
statistics-based aggregate optimization. Supported safe integer
conversions and `Int32 ↔ Date32` retain that optimization.
(cherry picked from commit ccfe704)
Conflicts resolved when applying to spiceai-54 (DataFusion 54.1):
- datafusion/physical-expr/src/projection.rs: added this commit's extrema
check (min_value/max_value/preserves_values) and is_within_extrema to the
54.1 project_column_statistics_through_expr. Upstream's same-type early
return in that function (d5bd10d, apache#24094) and its column_statistics_at
helper are not on 54.1 and are not added. check_bigger_cast accepts
identical types, so same-type casts still keep their extrema.
- datafusion/physical-expr/src/expressions/cast.rs: the check_bigger_cast
extension (Int32 <-> Date32, Int64 <-> Date64) and its doc comments apply
as upstream wrote them. Not applied: the comment reword in
cast_expr_properties, which is on strictly_order_preserving code 54.1 does
not have, and the unit test
test_integer_date_cast_preserves_values_and_ordering, which asserts that
same property.
- On 54.1, check_bigger_cast also gates substitute_cast_ordering in
equivalence/properties, so the four added casts can now also replace a
sort key there. They reinterpret the same integer and keep order and NULLs,
which is the property upstream relies on in this commit.1 parent 41bcdda commit 9ff4bf5
3 files changed
Lines changed: 340 additions & 11 deletions
File tree
- datafusion
- physical-expr/src
- expressions
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
165 | 165 | | |
166 | 166 | | |
167 | 167 | | |
168 | | - | |
169 | | - | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
170 | 174 | | |
171 | 175 | | |
172 | 176 | | |
| |||
176 | 180 | | |
177 | 181 | | |
178 | 182 | | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
179 | 187 | | |
180 | 188 | | |
181 | 189 | | |
| |||
188 | 196 | | |
189 | 197 | | |
190 | 198 | | |
191 | | - | |
| 199 | + | |
| 200 | + | |
192 | 201 | | |
193 | 202 | | |
194 | 203 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
792 | 792 | | |
793 | 793 | | |
794 | 794 | | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
795 | 826 | | |
796 | | - | |
797 | | - | |
798 | | - | |
799 | | - | |
800 | | - | |
801 | | - | |
802 | | - | |
803 | | - | |
| 827 | + | |
| 828 | + | |
804 | 829 | | |
805 | 830 | | |
806 | 831 | | |
807 | 832 | | |
808 | 833 | | |
809 | 834 | | |
810 | 835 | | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
811 | 853 | | |
812 | 854 | | |
813 | 855 | | |
| |||
2237 | 2279 | | |
2238 | 2280 | | |
2239 | 2281 | | |
| 2282 | + | |
| 2283 | + | |
| 2284 | + | |
| 2285 | + | |
| 2286 | + | |
| 2287 | + | |
| 2288 | + | |
| 2289 | + | |
| 2290 | + | |
| 2291 | + | |
| 2292 | + | |
| 2293 | + | |
| 2294 | + | |
| 2295 | + | |
| 2296 | + | |
| 2297 | + | |
| 2298 | + | |
| 2299 | + | |
| 2300 | + | |
| 2301 | + | |
| 2302 | + | |
| 2303 | + | |
| 2304 | + | |
| 2305 | + | |
| 2306 | + | |
| 2307 | + | |
| 2308 | + | |
| 2309 | + | |
| 2310 | + | |
| 2311 | + | |
| 2312 | + | |
| 2313 | + | |
| 2314 | + | |
| 2315 | + | |
| 2316 | + | |
| 2317 | + | |
| 2318 | + | |
| 2319 | + | |
| 2320 | + | |
| 2321 | + | |
| 2322 | + | |
| 2323 | + | |
| 2324 | + | |
| 2325 | + | |
| 2326 | + | |
| 2327 | + | |
| 2328 | + | |
| 2329 | + | |
| 2330 | + | |
| 2331 | + | |
| 2332 | + | |
| 2333 | + | |
| 2334 | + | |
| 2335 | + | |
| 2336 | + | |
| 2337 | + | |
| 2338 | + | |
| 2339 | + | |
| 2340 | + | |
| 2341 | + | |
| 2342 | + | |
| 2343 | + | |
| 2344 | + | |
| 2345 | + | |
| 2346 | + | |
| 2347 | + | |
| 2348 | + | |
| 2349 | + | |
| 2350 | + | |
| 2351 | + | |
| 2352 | + | |
| 2353 | + | |
| 2354 | + | |
| 2355 | + | |
| 2356 | + | |
| 2357 | + | |
| 2358 | + | |
| 2359 | + | |
| 2360 | + | |
| 2361 | + | |
| 2362 | + | |
| 2363 | + | |
| 2364 | + | |
| 2365 | + | |
| 2366 | + | |
| 2367 | + | |
| 2368 | + | |
| 2369 | + | |
| 2370 | + | |
| 2371 | + | |
| 2372 | + | |
| 2373 | + | |
| 2374 | + | |
| 2375 | + | |
| 2376 | + | |
| 2377 | + | |
| 2378 | + | |
| 2379 | + | |
| 2380 | + | |
| 2381 | + | |
| 2382 | + | |
| 2383 | + | |
| 2384 | + | |
| 2385 | + | |
| 2386 | + | |
| 2387 | + | |
| 2388 | + | |
| 2389 | + | |
| 2390 | + | |
| 2391 | + | |
| 2392 | + | |
| 2393 | + | |
| 2394 | + | |
| 2395 | + | |
| 2396 | + | |
| 2397 | + | |
| 2398 | + | |
| 2399 | + | |
| 2400 | + | |
| 2401 | + | |
| 2402 | + | |
| 2403 | + | |
| 2404 | + | |
| 2405 | + | |
| 2406 | + | |
| 2407 | + | |
| 2408 | + | |
| 2409 | + | |
| 2410 | + | |
| 2411 | + | |
| 2412 | + | |
| 2413 | + | |
| 2414 | + | |
| 2415 | + | |
| 2416 | + | |
| 2417 | + | |
| 2418 | + | |
| 2419 | + | |
| 2420 | + | |
| 2421 | + | |
| 2422 | + | |
| 2423 | + | |
| 2424 | + | |
| 2425 | + | |
| 2426 | + | |
| 2427 | + | |
| 2428 | + | |
| 2429 | + | |
| 2430 | + | |
| 2431 | + | |
| 2432 | + | |
| 2433 | + | |
| 2434 | + | |
| 2435 | + | |
| 2436 | + | |
| 2437 | + | |
| 2438 | + | |
| 2439 | + | |
| 2440 | + | |
| 2441 | + | |
| 2442 | + | |
| 2443 | + | |
| 2444 | + | |
| 2445 | + | |
| 2446 | + | |
| 2447 | + | |
| 2448 | + | |
| 2449 | + | |
| 2450 | + | |
| 2451 | + | |
| 2452 | + | |
| 2453 | + | |
| 2454 | + | |
| 2455 | + | |
| 2456 | + | |
| 2457 | + | |
| 2458 | + | |
| 2459 | + | |
| 2460 | + | |
| 2461 | + | |
| 2462 | + | |
| 2463 | + | |
| 2464 | + | |
| 2465 | + | |
| 2466 | + | |
| 2467 | + | |
| 2468 | + | |
| 2469 | + | |
| 2470 | + | |
| 2471 | + | |
| 2472 | + | |
2240 | 2473 | | |
2241 | 2474 | | |
2242 | 2475 | | |
| |||
0 commit comments