Skip to content

Commit abf1e6e

Browse files
haohuaijinOmega359
authored andcommitted
fix: only propagate cast statistics through safe conversions (apache#25227)
## Which issue does this PR close? - Closes apache#25221. ## Rationale for this change `MIN`/`MAX` over a cast can return incorrect results when aggregate optimization replaces the scan with converted column statistics. Casting the original endpoints is only valid if they remain extrema in the target domain. For example, a Parquet string column containing `('1', '100', '2')` has string extrema `'1'` and `'2'`. Previously, `MIN(CAST(a AS INT)), MAX(CAST(a AS INT))` could return `1, 2` from those statistics instead of the correct `1, 100`. The same issue affects `BIGINT`. ## What changes are included in this PR? - Restrict CAST statistics propagation to conversions proven safe for the source type or its exact integer bounds. Unsupported or unsafe conversions return unknown column statistics, so aggregates evaluate the data. - Preserve the existing same-type passthrough and supported lossless widening conversions. - Retain integer narrowing and signed/unsigned conversions when both original bounds are exact and non-null, and both converted endpoints are non-null. Successful endpoint conversions establish that the entire integer range fits in the target type; overflow errors or null endpoints do not qualify. - Extend `CastExpr::check_bigger_cast` to recognize `Int32 ↔ Date32`, and clarify that its contract includes lossless, strictly order-preserving conversions that preserve nulls. Arrow reinterprets the same `i32` values as days since the epoch, so these conversions preserve both statistics and ordering properties. This also retains the statistics-based `MIN`/`MAX` optimization for ClickBench's `UInt16 → Int32 → Date32` projection. ## What is the testing strategy for this PR? - Add projection unit tests for non-monotonic string-to-integer casts and integer narrowing with safe, overflowing, and inexact bounds. Removing the statistics guard makes the new string-cast regression test fail. - Add end-to-end Parquet regression cases in `parquet_statistics.slt` for `INT` and `BIGINT`, both expecting `1, 100`. - Add a bidirectional `Int32 ↔ Date32` test covering `i32::MIN`, `i32::MAX`, negative values, zero, NULL, and strict ordering properties. - Run the related projection and cast unit tests, plus the ClickBench and Parquet statistics SLT files. The existing ClickBench plan expectation passes unchanged, confirming that its aggregate still folds to constants. Local formatting, Clippy, license, spelling, workflow, and documentation checks also pass. ## Are there any user-facing changes? Affected casts now produce correct `MIN`/`MAX` results. Conversions whose safety is not established may require a scan instead of using statistics-based aggregate optimization. Supported safe integer conversions and `Int32 ↔ Date32` retain that optimization.
1 parent 2949730 commit abf1e6e

3 files changed

Lines changed: 395 additions & 13 deletions

File tree

‎datafusion/physical-expr/src/expressions/cast.rs‎

Lines changed: 68 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -248,8 +248,12 @@ impl CastExpr {
248248
})
249249
}
250250

251-
/// Check if casting from the specified source type to the target type is a
252-
/// widening cast (e.g. from `Int8` to `Int16`).
251+
/// Check if casting from the source type to the target type is known to be
252+
/// lossless and strictly order-preserving for all source values, preserving nulls.
253+
/// This includes widening casts (e.g. `Int8` to `Int16`) and representation
254+
/// conversions such as `Int32` to `Date32`, which interprets the same integer
255+
/// as days since the epoch, or `Int64` to `Date64`, which interprets the same
256+
/// integer as milliseconds since the epoch.
253257
pub fn check_bigger_cast(cast_type: &DataType, src: &DataType) -> bool {
254258
if cast_type.eq(src) {
255259
return true;
@@ -259,6 +263,10 @@ impl CastExpr {
259263
(Int8, Int16 | Int32 | Int64)
260264
| (Int16, Int32 | Int64)
261265
| (Int32, Int64)
266+
| (Int32, Date32)
267+
| (Date32, Int32)
268+
| (Int64, Date64)
269+
| (Date64, Int64)
262270
| (UInt8, UInt16 | UInt32 | UInt64)
263271
| (UInt16, UInt32 | UInt64)
264272
| (UInt32, UInt64)
@@ -268,7 +276,8 @@ impl CastExpr {
268276
)
269277
}
270278

271-
/// Check if the cast is a widening cast (e.g. from `Int8` to `Int16`).
279+
/// Check if the cast is lossless and strictly order-preserving for all source
280+
/// values, preserving nulls. See [`Self::check_bigger_cast`].
272281
pub fn is_bigger_cast(&self, src: &DataType) -> bool {
273282
Self::check_bigger_cast(self.cast_type(), src)
274283
}
@@ -289,8 +298,8 @@ pub(crate) fn cast_expr_properties(
289298
) -> Result<ExprProperties> {
290299
let unbounded = Interval::make_unbounded(target_type)?;
291300
let source_type = child.range.data_type();
292-
// A widening cast is additionally one-to-one, so it is strictly
293-
// order-preserving; a narrowing cast may collapse distinct values,
301+
// A lossless cast recognized by check_bigger_cast is one-to-one, so it is
302+
// strictly order-preserving; a narrowing cast may collapse distinct values,
294303
// breaking the ordering of subsequent sort keys.
295304
let bigger_cast = CastExpr::check_bigger_cast(target_type, &source_type);
296305
if is_order_preserving_cast_family(&source_type, target_type) || bigger_cast {
@@ -1505,6 +1514,60 @@ mod tests {
15051514
Ok(())
15061515
}
15071516

1517+
#[test]
1518+
fn test_integer_date_cast_preserves_values_and_ordering() {
1519+
use arrow::array::{Date32Array, Date64Array};
1520+
use arrow::compute::SortOptions;
1521+
use datafusion_expr_common::sort_properties::SortProperties;
1522+
1523+
let values = vec![None, Some(i32::MIN), Some(-1), Some(0), Some(i32::MAX)];
1524+
let integers: ArrayRef = Arc::new(Int32Array::from(values.clone()));
1525+
let dates: ArrayRef = Arc::new(Date32Array::from(values));
1526+
let values = vec![None, Some(i64::MIN), Some(-1), Some(0), Some(i64::MAX)];
1527+
let integers64: ArrayRef = Arc::new(Int64Array::from(values.clone()));
1528+
let dates64: ArrayRef = Arc::new(Date64Array::from(values));
1529+
for (input, expected) in [
1530+
(Arc::clone(&integers), Arc::clone(&dates)),
1531+
(dates, integers),
1532+
(Arc::clone(&integers64), Arc::clone(&dates64)),
1533+
(dates64, integers64),
1534+
] {
1535+
let schema = Arc::new(Schema::new(vec![Field::new(
1536+
"a",
1537+
input.data_type().clone(),
1538+
true,
1539+
)]));
1540+
let expr = CastExpr::new(
1541+
col("a", &schema).expect("column exists"),
1542+
expected.data_type().clone(),
1543+
None,
1544+
);
1545+
assert!(expr.is_bigger_cast(input.data_type()));
1546+
let child = ExprProperties::new_unknown()
1547+
.with_range(
1548+
Interval::make_unbounded(input.data_type())
1549+
.expect("supported interval type"),
1550+
)
1551+
.with_order(SortProperties::Ordered(SortOptions::default()))
1552+
.with_strictly_order_preserving(true);
1553+
let properties = expr
1554+
.get_properties(std::slice::from_ref(&child))
1555+
.expect("cast properties");
1556+
assert_eq!(properties.sort_properties, child.sort_properties);
1557+
assert!(properties.strictly_order_preserving);
1558+
assert_eq!(properties.range.data_type(), *expected.data_type());
1559+
1560+
let batch =
1561+
RecordBatch::try_new(schema, vec![input]).expect("valid input batch");
1562+
let actual = expr
1563+
.evaluate(&batch)
1564+
.expect("cast succeeds")
1565+
.into_array(batch.num_rows())
1566+
.expect("array result");
1567+
assert_eq!(actual.as_ref(), expected.as_ref());
1568+
}
1569+
}
1570+
15081571
#[test]
15091572
fn test_check_bigger_cast_precision_loss() {
15101573
use DataType::*;

‎datafusion/physical-expr/src/projection.rs‎

Lines changed: 240 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -872,22 +872,63 @@ fn project_column_statistics_through_expr(
872872
return inner_stats;
873873
}
874874

875+
let min_value = inner_stats
876+
.min_value
877+
.cast_to(target_type)
878+
.unwrap_or(Precision::Absent);
879+
let max_value = inner_stats
880+
.max_value
881+
.cast_to(target_type)
882+
.unwrap_or(Precision::Absent);
883+
let source_type = inner_stats
884+
.min_value
885+
.get_value()
886+
.or_else(|| inner_stats.max_value.get_value())
887+
.map(ScalarValue::data_type);
888+
// Copy extrema only for casts that preserve order and cannot discard values
889+
// or fail within the input domain. Copying string endpoints into a numeric
890+
// domain, for example, does not bound the converted column. Merely casting
891+
// a failing endpoint to NULL also cannot establish the remaining extrema.
892+
let preserves_values = source_type.is_some_and(|source_type| {
893+
CastExpr::check_bigger_cast(target_type, &source_type)
894+
|| is_within_extrema(
895+
&inner_stats.min_value,
896+
&inner_stats.max_value,
897+
&min_value,
898+
&max_value,
899+
)
900+
});
901+
if !preserves_values {
902+
return ColumnStatistics::new_unknown();
903+
}
904+
875905
ColumnStatistics {
876-
min_value: inner_stats
877-
.min_value
878-
.cast_to(target_type)
879-
.unwrap_or(Precision::Absent),
880-
max_value: inner_stats
881-
.max_value
882-
.cast_to(target_type)
883-
.unwrap_or(Precision::Absent),
906+
min_value,
907+
max_value,
884908
null_count: inner_stats.null_count,
885909
distinct_count: inner_stats.distinct_count,
886910
sum_value: Precision::Absent,
887911
byte_size: Precision::Absent,
888912
}
889913
}
890914

915+
/// Whether an integer cast preserves every value between the source extrema.
916+
/// The converted extrema must come from casting the corresponding source bounds.
917+
/// Exact, non-null integer bounds that both cast successfully prove the entire
918+
/// range fits in the target type. For example, Int64 [-128, 127] fits in Int8,
919+
/// whereas [-129, 127] does not. Inexact bounds cannot establish exact extrema.
920+
fn is_within_extrema(
921+
lower: &Precision<ScalarValue>,
922+
upper: &Precision<ScalarValue>,
923+
min: &Precision<ScalarValue>,
924+
max: &Precision<ScalarValue>,
925+
) -> bool {
926+
[lower, upper, min, max].into_iter().all(|bound| {
927+
matches!(bound, Precision::Exact(value)
928+
if value.data_type().is_integer() && !value.is_null())
929+
})
930+
}
931+
891932
fn column_statistics_at(
892933
column_stats: &[ColumnStatistics],
893934
index: usize,
@@ -2249,6 +2290,197 @@ pub(crate) mod tests {
22492290
Ok(())
22502291
}
22512292

2293+
#[test]
2294+
fn test_project_statistics_non_monotonic_cast() {
2295+
let input_schema = Schema::new(vec![Field::new("a", DataType::Utf8, false)]);
2296+
let mut stats = Statistics::new_unknown(&input_schema);
2297+
stats.num_rows = Precision::Exact(3);
2298+
stats.column_statistics[0].min_value = Precision::Exact(ScalarValue::from("1"));
2299+
stats.column_statistics[0].max_value = Precision::Exact(ScalarValue::from("2"));
2300+
let expr: Arc<dyn PhysicalExpr> = Arc::new(CastExpr::new(
2301+
Arc::new(Column::new("a", 0)),
2302+
DataType::Int32,
2303+
None,
2304+
));
2305+
let projection = ProjectionExprs::new(vec![ProjectionExpr {
2306+
expr: Arc::clone(&expr),
2307+
alias: "x".to_string(),
2308+
}]);
2309+
let out = projection
2310+
.project_statistics(
2311+
stats,
2312+
&projection
2313+
.project_schema(&input_schema)
2314+
.expect("valid projection schema"),
2315+
)
2316+
.expect("statistics projection succeeds");
2317+
let batch = RecordBatch::try_new(
2318+
Arc::new(input_schema),
2319+
vec![Arc::new(arrow::array::StringArray::from(vec![
2320+
"1", "100", "2",
2321+
]))],
2322+
)
2323+
.expect("valid input batch");
2324+
let actual = expr
2325+
.evaluate(&batch)
2326+
.expect("cast succeeds")
2327+
.into_array(3)
2328+
.expect("array result");
2329+
assert_eq!(out.column_statistics[0].max_value, Precision::Absent);
2330+
assert_eq!(out.column_statistics[0].min_value, Precision::Absent);
2331+
assert_eq!(
2332+
ScalarValue::try_from_array(&actual, 1).expect("valid scalar value"),
2333+
ScalarValue::Int32(Some(100))
2334+
);
2335+
}
2336+
2337+
#[test]
2338+
fn test_is_within_extrema() {
2339+
use Precision::{Absent, Exact, Inexact};
2340+
use ScalarValue::{Int8, Int64, UInt8};
2341+
2342+
for (lower, upper, target_type, expected) in [
2343+
(
2344+
Exact(Int64(Some(-128))),
2345+
Exact(Int64(Some(127))),
2346+
DataType::Int8,
2347+
true,
2348+
),
2349+
(
2350+
Exact(Int64(Some(-129))),
2351+
Exact(Int64(Some(127))),
2352+
DataType::Int8,
2353+
false,
2354+
),
2355+
(
2356+
Exact(Int64(Some(-128))),
2357+
Exact(Int64(Some(128))),
2358+
DataType::Int8,
2359+
false,
2360+
),
2361+
(
2362+
Exact(Int64(Some(0))),
2363+
Exact(Int64(Some(255))),
2364+
DataType::UInt8,
2365+
true,
2366+
),
2367+
(
2368+
Exact(Int64(Some(-1))),
2369+
Exact(Int64(Some(255))),
2370+
DataType::UInt8,
2371+
false,
2372+
),
2373+
(
2374+
Exact(UInt8(Some(0))),
2375+
Exact(UInt8(Some(127))),
2376+
DataType::Int8,
2377+
true,
2378+
),
2379+
(
2380+
Exact(UInt8(Some(0))),
2381+
Exact(UInt8(Some(128))),
2382+
DataType::Int8,
2383+
false,
2384+
),
2385+
(
2386+
Exact(ScalarValue::from("1")),
2387+
Exact(ScalarValue::from("2")),
2388+
DataType::Int8,
2389+
false,
2390+
),
2391+
(
2392+
Exact(Int64(Some(1))),
2393+
Exact(Int64(Some(2))),
2394+
DataType::Utf8,
2395+
false,
2396+
),
2397+
] {
2398+
let min = lower.cast_to(&target_type).unwrap_or(Absent);
2399+
let max = upper.cast_to(&target_type).unwrap_or(Absent);
2400+
assert_eq!(
2401+
is_within_extrema(&lower, &upper, &min, &max),
2402+
expected,
2403+
"{lower:?}..{upper:?} -> {target_type:?}",
2404+
);
2405+
}
2406+
2407+
// Every bound must be exact and non-null, including converted endpoints.
2408+
let bounds = [
2409+
Exact(Int64(Some(-128))),
2410+
Exact(Int64(Some(127))),
2411+
Exact(Int8(Some(-128))),
2412+
Exact(Int8(Some(127))),
2413+
];
2414+
for index in 0..bounds.len() {
2415+
for invalid in [
2416+
Absent,
2417+
Inexact(bounds[index].get_value().unwrap().clone()),
2418+
Exact(
2419+
ScalarValue::try_from(
2420+
&bounds[index].get_value().unwrap().data_type(),
2421+
)
2422+
.unwrap(),
2423+
),
2424+
] {
2425+
let mut bounds = bounds.clone();
2426+
bounds[index] = invalid;
2427+
assert!(!is_within_extrema(
2428+
&bounds[0], &bounds[1], &bounds[2], &bounds[3],
2429+
));
2430+
}
2431+
}
2432+
}
2433+
2434+
#[test]
2435+
fn test_project_statistics_narrowing_cast_requires_safe_bounds() {
2436+
let schema = Schema::new(vec![Field::new("a", DataType::Int32, true)]);
2437+
for (lower, upper, exact, safe) in [
2438+
(-100, 100, true, true),
2439+
(-200, 100, true, false),
2440+
(-100, 200, true, false),
2441+
(-100, 100, false, false),
2442+
] {
2443+
let precision = |v| {
2444+
if exact {
2445+
Precision::Exact(ScalarValue::Int32(Some(v)))
2446+
} else {
2447+
Precision::Inexact(ScalarValue::Int32(Some(v)))
2448+
}
2449+
};
2450+
let mut stats = Statistics::new_unknown(&schema);
2451+
stats.column_statistics[0].min_value = precision(lower);
2452+
stats.column_statistics[0].max_value = precision(upper);
2453+
let projection = ProjectionExprs::new(vec![ProjectionExpr::new(
2454+
Arc::new(CastExpr::new(
2455+
Arc::new(Column::new("a", 0)),
2456+
DataType::Int8,
2457+
None,
2458+
)),
2459+
"x",
2460+
)]);
2461+
let output = projection
2462+
.project_statistics(
2463+
stats,
2464+
&projection
2465+
.project_schema(&schema)
2466+
.expect("valid projection schema"),
2467+
)
2468+
.expect("statistics projection succeeds");
2469+
if safe {
2470+
assert_eq!(
2471+
output.column_statistics[0].min_value,
2472+
Precision::Exact(ScalarValue::Int8(Some(-100)))
2473+
);
2474+
assert_eq!(
2475+
output.column_statistics[0].max_value,
2476+
Precision::Exact(ScalarValue::Int8(Some(100)))
2477+
);
2478+
} else {
2479+
assert_eq!(output.column_statistics[0], ColumnStatistics::new_unknown());
2480+
}
2481+
}
2482+
}
2483+
22522484
fn get_stats() -> Statistics {
22532485
Statistics {
22542486
num_rows: Precision::Exact(5),

0 commit comments

Comments
 (0)