Skip to content

Commit 9ff4bf5

Browse files
haohuaijinlukekim
authored andcommitted
fix: only propagate cast statistics through safe conversions (apache#25227)
## Which issue does this PR close? - Closes apache#25221. ## Rationale for this change `MIN`/`MAX` over a cast can return incorrect results when aggregate optimization replaces the scan with converted column statistics. Casting the original endpoints is only valid if they remain extrema in the target domain. For example, a Parquet string column containing `('1', '100', '2')` has string extrema `'1'` and `'2'`. Previously, `MIN(CAST(a AS INT)), MAX(CAST(a AS INT))` could return `1, 2` from those statistics instead of the correct `1, 100`. The same issue affects `BIGINT`. ## What changes are included in this PR? - Restrict CAST statistics propagation to conversions proven safe for the source type or its exact integer bounds. Unsupported or unsafe conversions return unknown column statistics, so aggregates evaluate the data. - Preserve the existing same-type passthrough and supported lossless widening conversions. - Retain integer narrowing and signed/unsigned conversions when both original bounds are exact and non-null, and both converted endpoints are non-null. Successful endpoint conversions establish that the entire integer range fits in the target type; overflow errors or null endpoints do not qualify. - Extend `CastExpr::check_bigger_cast` to recognize `Int32 ↔ Date32`, and clarify that its contract includes lossless, strictly order-preserving conversions that preserve nulls. Arrow reinterprets the same `i32` values as days since the epoch, so these conversions preserve both statistics and ordering properties. This also retains the statistics-based `MIN`/`MAX` optimization for ClickBench's `UInt16 → Int32 → Date32` projection. ## What is the testing strategy for this PR? - Add projection unit tests for non-monotonic string-to-integer casts and integer narrowing with safe, overflowing, and inexact bounds. Removing the statistics guard makes the new string-cast regression test fail. - Add end-to-end Parquet regression cases in `parquet_statistics.slt` for `INT` and `BIGINT`, both expecting `1, 100`. - Add a bidirectional `Int32 ↔ Date32` test covering `i32::MIN`, `i32::MAX`, negative values, zero, NULL, and strict ordering properties. - Run the related projection and cast unit tests, plus the ClickBench and Parquet statistics SLT files. The existing ClickBench plan expectation passes unchanged, confirming that its aggregate still folds to constants. Local formatting, Clippy, license, spelling, workflow, and documentation checks also pass. ## Are there any user-facing changes? Affected casts now produce correct `MIN`/`MAX` results. Conversions whose safety is not established may require a scan instead of using statistics-based aggregate optimization. Supported safe integer conversions and `Int32 ↔ Date32` retain that optimization. (cherry picked from commit ccfe704) Conflicts resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/physical-expr/src/projection.rs: added this commit's extrema check (min_value/max_value/preserves_values) and is_within_extrema to the 54.1 project_column_statistics_through_expr. Upstream's same-type early return in that function (d5bd10d, apache#24094) and its column_statistics_at helper are not on 54.1 and are not added. check_bigger_cast accepts identical types, so same-type casts still keep their extrema. - datafusion/physical-expr/src/expressions/cast.rs: the check_bigger_cast extension (Int32 <-> Date32, Int64 <-> Date64) and its doc comments apply as upstream wrote them. Not applied: the comment reword in cast_expr_properties, which is on strictly_order_preserving code 54.1 does not have, and the unit test test_integer_date_cast_preserves_values_and_ordering, which asserts that same property. - On 54.1, check_bigger_cast also gates substitute_cast_ordering in equivalence/properties, so the four added casts can now also replace a sort key there. They reinterpret the same integer and keep order and NULLs, which is the property upstream relies on in this commit.
1 parent 41bcdda commit 9ff4bf5

3 files changed

Lines changed: 340 additions & 11 deletions

File tree

‎datafusion/physical-expr/src/expressions/cast.rs‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -165,8 +165,12 @@ impl CastExpr {
165165
}
166166
}
167167

168-
/// Check if casting from the specified source type to the target type is a
169-
/// widening cast (e.g. from `Int8` to `Int16`).
168+
/// Check if casting from the source type to the target type is known to be
169+
/// lossless and strictly order-preserving for all source values, preserving nulls.
170+
/// This includes widening casts (e.g. `Int8` to `Int16`) and representation
171+
/// conversions such as `Int32` to `Date32`, which interprets the same integer
172+
/// as days since the epoch, or `Int64` to `Date64`, which interprets the same
173+
/// integer as milliseconds since the epoch.
170174
pub fn check_bigger_cast(cast_type: &DataType, src: &DataType) -> bool {
171175
if cast_type.eq(src) {
172176
return true;
@@ -176,6 +180,10 @@ impl CastExpr {
176180
(Int8, Int16 | Int32 | Int64)
177181
| (Int16, Int32 | Int64)
178182
| (Int32, Int64)
183+
| (Int32, Date32)
184+
| (Date32, Int32)
185+
| (Int64, Date64)
186+
| (Date64, Int64)
179187
| (UInt8, UInt16 | UInt32 | UInt64)
180188
| (UInt16, UInt32 | UInt64)
181189
| (UInt32, UInt64)
@@ -188,7 +196,8 @@ impl CastExpr {
188196
)
189197
}
190198

191-
/// Check if the cast is a widening cast (e.g. from `Int8` to `Int16`).
199+
/// Check if the cast is lossless and strictly order-preserving for all source
200+
/// values, preserving nulls. See [`Self::check_bigger_cast`].
192201
pub fn is_bigger_cast(&self, src: &DataType) -> bool {
193202
Self::check_bigger_cast(self.cast_type(), src)
194203
}

‎datafusion/physical-expr/src/projection.rs‎

Lines changed: 241 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -792,22 +792,64 @@ fn project_column_statistics_through_expr(
792792
let inner_stats =
793793
project_column_statistics_through_expr(cast_expr.expr.as_ref(), column_stats);
794794
let target_type = cast_expr.cast_type();
795+
796+
let min_value = inner_stats
797+
.min_value
798+
.cast_to(target_type)
799+
.unwrap_or(Precision::Absent);
800+
let max_value = inner_stats
801+
.max_value
802+
.cast_to(target_type)
803+
.unwrap_or(Precision::Absent);
804+
let source_type = inner_stats
805+
.min_value
806+
.get_value()
807+
.or_else(|| inner_stats.max_value.get_value())
808+
.map(ScalarValue::data_type);
809+
// Copy extrema only for casts that preserve order and cannot discard values
810+
// or fail within the input domain. Copying string endpoints into a numeric
811+
// domain, for example, does not bound the converted column. Merely casting
812+
// a failing endpoint to NULL also cannot establish the remaining extrema.
813+
let preserves_values = source_type.is_some_and(|source_type| {
814+
CastExpr::check_bigger_cast(target_type, &source_type)
815+
|| is_within_extrema(
816+
&inner_stats.min_value,
817+
&inner_stats.max_value,
818+
&min_value,
819+
&max_value,
820+
)
821+
});
822+
if !preserves_values {
823+
return ColumnStatistics::new_unknown();
824+
}
825+
795826
ColumnStatistics {
796-
min_value: inner_stats
797-
.min_value
798-
.cast_to(target_type)
799-
.unwrap_or(Precision::Absent),
800-
max_value: inner_stats
801-
.max_value
802-
.cast_to(target_type)
803-
.unwrap_or(Precision::Absent),
827+
min_value,
828+
max_value,
804829
null_count: inner_stats.null_count,
805830
distinct_count: inner_stats.distinct_count,
806831
sum_value: Precision::Absent,
807832
byte_size: Precision::Absent,
808833
}
809834
}
810835

836+
/// Whether an integer cast preserves every value between the source extrema.
837+
/// The converted extrema must come from casting the corresponding source bounds.
838+
/// Exact, non-null integer bounds that both cast successfully prove the entire
839+
/// range fits in the target type. For example, Int64 [-128, 127] fits in Int8,
840+
/// whereas [-129, 127] does not. Inexact bounds cannot establish exact extrema.
841+
fn is_within_extrema(
842+
lower: &Precision<ScalarValue>,
843+
upper: &Precision<ScalarValue>,
844+
min: &Precision<ScalarValue>,
845+
max: &Precision<ScalarValue>,
846+
) -> bool {
847+
[lower, upper, min, max].into_iter().all(|bound| {
848+
matches!(bound, Precision::Exact(value)
849+
if value.data_type().is_integer() && !value.is_null())
850+
})
851+
}
852+
811853
impl<'a> IntoIterator for &'a ProjectionExprs {
812854
type Item = &'a ProjectionExpr;
813855
type IntoIter = std::slice::Iter<'a, ProjectionExpr>;
@@ -2237,6 +2279,197 @@ pub(crate) mod tests {
22372279
Ok(())
22382280
}
22392281

2282+
#[test]
2283+
fn test_project_statistics_non_monotonic_cast() {
2284+
let input_schema = Schema::new(vec![Field::new("a", DataType::Utf8, false)]);
2285+
let mut stats = Statistics::new_unknown(&input_schema);
2286+
stats.num_rows = Precision::Exact(3);
2287+
stats.column_statistics[0].min_value = Precision::Exact(ScalarValue::from("1"));
2288+
stats.column_statistics[0].max_value = Precision::Exact(ScalarValue::from("2"));
2289+
let expr: Arc<dyn PhysicalExpr> = Arc::new(CastExpr::new(
2290+
Arc::new(Column::new("a", 0)),
2291+
DataType::Int32,
2292+
None,
2293+
));
2294+
let projection = ProjectionExprs::new(vec![ProjectionExpr {
2295+
expr: Arc::clone(&expr),
2296+
alias: "x".to_string(),
2297+
}]);
2298+
let out = projection
2299+
.project_statistics(
2300+
stats,
2301+
&projection
2302+
.project_schema(&input_schema)
2303+
.expect("valid projection schema"),
2304+
)
2305+
.expect("statistics projection succeeds");
2306+
let batch = RecordBatch::try_new(
2307+
Arc::new(input_schema),
2308+
vec![Arc::new(arrow::array::StringArray::from(vec![
2309+
"1", "100", "2",
2310+
]))],
2311+
)
2312+
.expect("valid input batch");
2313+
let actual = expr
2314+
.evaluate(&batch)
2315+
.expect("cast succeeds")
2316+
.into_array(3)
2317+
.expect("array result");
2318+
assert_eq!(out.column_statistics[0].max_value, Precision::Absent);
2319+
assert_eq!(out.column_statistics[0].min_value, Precision::Absent);
2320+
assert_eq!(
2321+
ScalarValue::try_from_array(&actual, 1).expect("valid scalar value"),
2322+
ScalarValue::Int32(Some(100))
2323+
);
2324+
}
2325+
2326+
#[test]
2327+
fn test_is_within_extrema() {
2328+
use Precision::{Absent, Exact, Inexact};
2329+
use ScalarValue::{Int8, Int64, UInt8};
2330+
2331+
for (lower, upper, target_type, expected) in [
2332+
(
2333+
Exact(Int64(Some(-128))),
2334+
Exact(Int64(Some(127))),
2335+
DataType::Int8,
2336+
true,
2337+
),
2338+
(
2339+
Exact(Int64(Some(-129))),
2340+
Exact(Int64(Some(127))),
2341+
DataType::Int8,
2342+
false,
2343+
),
2344+
(
2345+
Exact(Int64(Some(-128))),
2346+
Exact(Int64(Some(128))),
2347+
DataType::Int8,
2348+
false,
2349+
),
2350+
(
2351+
Exact(Int64(Some(0))),
2352+
Exact(Int64(Some(255))),
2353+
DataType::UInt8,
2354+
true,
2355+
),
2356+
(
2357+
Exact(Int64(Some(-1))),
2358+
Exact(Int64(Some(255))),
2359+
DataType::UInt8,
2360+
false,
2361+
),
2362+
(
2363+
Exact(UInt8(Some(0))),
2364+
Exact(UInt8(Some(127))),
2365+
DataType::Int8,
2366+
true,
2367+
),
2368+
(
2369+
Exact(UInt8(Some(0))),
2370+
Exact(UInt8(Some(128))),
2371+
DataType::Int8,
2372+
false,
2373+
),
2374+
(
2375+
Exact(ScalarValue::from("1")),
2376+
Exact(ScalarValue::from("2")),
2377+
DataType::Int8,
2378+
false,
2379+
),
2380+
(
2381+
Exact(Int64(Some(1))),
2382+
Exact(Int64(Some(2))),
2383+
DataType::Utf8,
2384+
false,
2385+
),
2386+
] {
2387+
let min = lower.cast_to(&target_type).unwrap_or(Absent);
2388+
let max = upper.cast_to(&target_type).unwrap_or(Absent);
2389+
assert_eq!(
2390+
is_within_extrema(&lower, &upper, &min, &max),
2391+
expected,
2392+
"{lower:?}..{upper:?} -> {target_type:?}",
2393+
);
2394+
}
2395+
2396+
// Every bound must be exact and non-null, including converted endpoints.
2397+
let bounds = [
2398+
Exact(Int64(Some(-128))),
2399+
Exact(Int64(Some(127))),
2400+
Exact(Int8(Some(-128))),
2401+
Exact(Int8(Some(127))),
2402+
];
2403+
for index in 0..bounds.len() {
2404+
for invalid in [
2405+
Absent,
2406+
Inexact(bounds[index].get_value().unwrap().clone()),
2407+
Exact(
2408+
ScalarValue::try_from(
2409+
&bounds[index].get_value().unwrap().data_type(),
2410+
)
2411+
.unwrap(),
2412+
),
2413+
] {
2414+
let mut bounds = bounds.clone();
2415+
bounds[index] = invalid;
2416+
assert!(!is_within_extrema(
2417+
&bounds[0], &bounds[1], &bounds[2], &bounds[3],
2418+
));
2419+
}
2420+
}
2421+
}
2422+
2423+
#[test]
2424+
fn test_project_statistics_narrowing_cast_requires_safe_bounds() {
2425+
let schema = Schema::new(vec![Field::new("a", DataType::Int32, true)]);
2426+
for (lower, upper, exact, safe) in [
2427+
(-100, 100, true, true),
2428+
(-200, 100, true, false),
2429+
(-100, 200, true, false),
2430+
(-100, 100, false, false),
2431+
] {
2432+
let precision = |v| {
2433+
if exact {
2434+
Precision::Exact(ScalarValue::Int32(Some(v)))
2435+
} else {
2436+
Precision::Inexact(ScalarValue::Int32(Some(v)))
2437+
}
2438+
};
2439+
let mut stats = Statistics::new_unknown(&schema);
2440+
stats.column_statistics[0].min_value = precision(lower);
2441+
stats.column_statistics[0].max_value = precision(upper);
2442+
let projection = ProjectionExprs::new(vec![ProjectionExpr::new(
2443+
Arc::new(CastExpr::new(
2444+
Arc::new(Column::new("a", 0)),
2445+
DataType::Int8,
2446+
None,
2447+
)),
2448+
"x",
2449+
)]);
2450+
let output = projection
2451+
.project_statistics(
2452+
stats,
2453+
&projection
2454+
.project_schema(&schema)
2455+
.expect("valid projection schema"),
2456+
)
2457+
.expect("statistics projection succeeds");
2458+
if safe {
2459+
assert_eq!(
2460+
output.column_statistics[0].min_value,
2461+
Precision::Exact(ScalarValue::Int8(Some(-100)))
2462+
);
2463+
assert_eq!(
2464+
output.column_statistics[0].max_value,
2465+
Precision::Exact(ScalarValue::Int8(Some(100)))
2466+
);
2467+
} else {
2468+
assert_eq!(output.column_statistics[0], ColumnStatistics::new_unknown());
2469+
}
2470+
}
2471+
}
2472+
22402473
fn get_stats() -> Statistics {
22412474
Statistics {
22422475
num_rows: Precision::Exact(5),

0 commit comments

Comments
 (0)