Skip to content

Commit 707a7b1

Browse files
DeviousCardiclaude
andcommitted
fix: do not evaluate a volatile coalesce/nvl argument twice
`CoalesceFunc::simplify` rewrites `coalesce(a, b)` into `CASE WHEN a IS NOT NULL THEN a ELSE b END`. That rewrite names every non-final argument *twice* -- once in the `WHEN` predicate and once in the `THEN` result -- so for a volatile argument the two mentions are two independent draws. Since apache#17357 removed the runtime kernel (`invoke_with_args` was left as `internal_err!("coalesce should have been simplified to case")`), there was no other path and no volatility guard, so every volatile `coalesce` went through the duplicating rewrite. `nvl` delegates `simplify`/`invoke_with_args`/`short_circuits`/ `conditional_arguments` to `CoalesceFunc`, so it had the identical bug. Two observable symptoms, both reproduced in the new sqllogictests against the unpatched code: 1. Wrong results. Over 100000 rows, `count(coalesce(nullif(floor(random()*2), 0), -1))` returned 75112 instead of 100000 -- 24888 impossible NULLs, exactly the 0.5*0.5 two-draw rate, from rows where the `WHEN` draw was non-null but the independent `THEN` draw was null. 2. A hard failure. `return_field_from_args` marks the result non-nullable when any argument is non-nullable (here the `-1` fallback), so those impossible NULLs also trip `Arrow error: Invalid argument error: Column 'c' is declared as non-nullable but contains null values`. Fix (two parts, one file): * Guard `simplify`: if any argument `is_volatile()`, return `ExprSimplifyResult::Original` and leave the `coalesce` call intact. Single-argument `coalesce` is still unwrapped, volatile or not, because it is named only once. * Restore the runtime kernel in `invoke_with_args`, recovered from e5dcc8c (`new_null_array` + `is_not_null`/`is_null` + `zip`). It walks the already-evaluated `ColumnarValue`s and therefore evaluates each argument exactly once. Why this rather than a new physical expression (apache#25476): this is ~60 lines in a single file with no new public API, no new physical expression and no protobuf change, which answers the complexity objection raised on that PR directly. Nothing outside `coalesce.rs` moves. Trade-off -- the restored kernel is eager, which is exactly what apache#17357 removed, so this is deliberately confined to the volatile case: * Non-volatile `coalesce` is untouched and keeps the lazy `CASE` rewrite. `select.slt:1686` (`select coalesce(1, y/x)` with `x = 0`) still plans to `Projection: Int64(1)` and still never divides by zero. * Volatile `coalesce` is now eager, so `coalesce(random(), y/x)` will evaluate `y/x` and can raise divide-by-zero where it previously did not. That is accepted: today the same query silently returns wrong answers (or aborts with the Arrow nullability error above), and a surfaced error beats a silently wrong result. It also only restores the pre-apache#17357 behaviour, and only for the narrow slice of calls that actually contain a volatile argument. There is no way to get both per-row laziness and single-evaluation out of the `CASE` rewrite, because the rewrite is what duplicates the argument. `short_circuits()` stays `true` and `conditional_arguments` is unchanged. Its contract is "some subexpressions *may* not be evaluated", so an eager kernel does not violate it -- common-subexpression elimination simply declines to hoist out of the lazy arguments, which costs an optimization and never correctness. Tests: unit tests for the restored kernel (arrays, array + scalar fallback, all scalars, all-null scalars, all-null arrays, empty args) and for the simplify guard (non-volatile still expands to `CASE`; direct, nested and single-argument volatile cases); sqllogictests asserting the deterministic `count(c) = count(*)` row count, that no value comes from outside the two operands, and `EXPLAIN` output pinning that the volatile shape stays `coalesce(...)`/`nvl(...)` and does not expand to `CASE`, while the non-volatile shape still does. Closes apache#25477. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 1e09a2a commit 707a7b1

2 files changed

Lines changed: 343 additions & 2 deletions

File tree

‎datafusion/functions/src/core/coalesce.rs‎

Lines changed: 268 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,9 @@
1515
// specific language governing permissions and limitations
1616
// under the License.
1717

18+
use arrow::array::{BooleanArray, new_null_array};
19+
use arrow::compute::kernels::zip::zip;
20+
use arrow::compute::{and, is_not_null, is_null};
1821
use arrow::datatypes::{DataType, Field, FieldRef};
1922
use datafusion_common::{Result, exec_err, internal_err, plan_err};
2023
use datafusion_expr::binary::try_type_union_resolution;
@@ -103,6 +106,20 @@ impl ScalarUDFImpl for CoalesceFunc {
103106
));
104107
}
105108

109+
// The `CASE WHEN a IS NOT NULL THEN a ELSE b END` rewrite below mentions
110+
// every non-final argument *twice* (once in the `WHEN` predicate and once
111+
// in the `THEN` result). That is fine for deterministic arguments, but for
112+
// a volatile argument the two mentions are two independent draws, so
113+
// `coalesce(random_nullable_expr, default)` can take the `THEN` branch
114+
// after the `WHEN` draw was non-null and still produce NULL.
115+
//
116+
// See https://github.com/apache/datafusion/issues/25477. For volatile
117+
// arguments we keep `coalesce` intact and let the runtime kernel in
118+
// `invoke_with_args` evaluate each argument exactly once.
119+
if args.iter().any(|arg| arg.is_volatile()) {
120+
return Ok(ExprSimplifyResult::Original(args));
121+
}
122+
106123
let n = args.len();
107124
let (init, last_elem) = args.split_at(n - 1);
108125
let whens = init
@@ -117,8 +134,63 @@ impl ScalarUDFImpl for CoalesceFunc {
117134
}
118135

119136
/// coalesce evaluates to the first value which is not NULL
120-
fn invoke_with_args(&self, _args: ScalarFunctionArgs) -> Result<ColumnarValue> {
121-
internal_err!("coalesce should have been simplified to case")
137+
///
138+
/// This kernel is only reached when [`Self::simplify`] declined to rewrite the
139+
/// call into a `CASE` expression, which today only happens when one of the
140+
/// arguments is volatile. It evaluates each argument exactly once.
141+
fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
142+
let args = args.args;
143+
// do not accept 0 arguments.
144+
if args.is_empty() {
145+
return exec_err!(
146+
"coalesce was called with {} arguments. It requires at least 1.",
147+
args.len()
148+
);
149+
}
150+
151+
let return_type = args[0].data_type();
152+
let mut return_array = args.iter().filter_map(|x| match x {
153+
ColumnarValue::Array(array) => Some(array.len()),
154+
_ => None,
155+
});
156+
157+
if let Some(size) = return_array.next() {
158+
// start with nulls as default output
159+
let mut current_value = new_null_array(&return_type, size);
160+
let mut remainder = BooleanArray::from(vec![true; size]);
161+
162+
for arg in args {
163+
match arg {
164+
ColumnarValue::Array(ref array) => {
165+
let to_apply = and(&remainder, &is_not_null(array.as_ref())?)?;
166+
current_value = zip(&to_apply, array, &current_value)?;
167+
remainder = and(&remainder, &is_null(array)?)?;
168+
}
169+
ColumnarValue::Scalar(value) => {
170+
if value.is_null() {
171+
continue;
172+
} else {
173+
let last_value = value.to_scalar()?;
174+
current_value = zip(&remainder, &last_value, &current_value)?;
175+
break;
176+
}
177+
}
178+
}
179+
if remainder.iter().all(|x| x == Some(false)) {
180+
break;
181+
}
182+
}
183+
Ok(ColumnarValue::Array(current_value))
184+
} else {
185+
let result = args
186+
.iter()
187+
.find_map(|x| match x {
188+
ColumnarValue::Scalar(s) if !s.is_null() => Some(x.clone()),
189+
_ => None,
190+
})
191+
.unwrap_or_else(|| args[0].clone());
192+
Ok(result)
193+
}
122194
}
123195

124196
fn conditional_arguments<'a>(
@@ -146,3 +218,197 @@ impl ScalarUDFImpl for CoalesceFunc {
146218
self.doc()
147219
}
148220
}
221+
222+
#[cfg(test)]
223+
mod tests {
224+
use std::sync::Arc;
225+
226+
use arrow::array::{Array, ArrayRef, Int32Array, StringArray};
227+
use arrow::datatypes::DataType;
228+
use datafusion_common::ScalarValue;
229+
use datafusion_common::config::ConfigOptions;
230+
use datafusion_expr::{ColumnarValue, Expr, ScalarUDFImpl, lit};
231+
232+
use super::*;
233+
234+
fn invoke(args: Vec<ColumnarValue>, number_rows: usize) -> ColumnarValue {
235+
let return_type = args[0].data_type();
236+
let arg_fields = args
237+
.iter()
238+
.enumerate()
239+
.map(|(i, a)| Field::new(format!("a{i}"), a.data_type(), true).into())
240+
.collect();
241+
CoalesceFunc::new()
242+
.invoke_with_args(ScalarFunctionArgs {
243+
args,
244+
arg_fields,
245+
number_rows,
246+
return_field: Field::new("f", return_type, true).into(),
247+
config_options: Arc::new(ConfigOptions::default()),
248+
})
249+
.expect("coalesce kernel failed")
250+
}
251+
252+
/// The restored runtime kernel picks the first non-null value per row.
253+
#[test]
254+
fn coalesce_kernel_arrays() {
255+
let a: ArrayRef = Arc::new(Int32Array::from(vec![Some(1), None, None, None]));
256+
let b: ArrayRef = Arc::new(Int32Array::from(vec![Some(9), Some(2), None, None]));
257+
let c: ArrayRef =
258+
Arc::new(Int32Array::from(vec![Some(8), Some(8), Some(3), None]));
259+
260+
let result = invoke(
261+
vec![
262+
ColumnarValue::Array(a),
263+
ColumnarValue::Array(b),
264+
ColumnarValue::Array(c),
265+
],
266+
4,
267+
)
268+
.into_array(4)
269+
.unwrap();
270+
271+
let expected: ArrayRef =
272+
Arc::new(Int32Array::from(vec![Some(1), Some(2), Some(3), None]));
273+
assert_eq!(&result, &expected);
274+
}
275+
276+
/// A trailing non-null scalar fills every remaining row, so the output has no nulls.
277+
#[test]
278+
fn coalesce_kernel_array_with_scalar_fallback() {
279+
let a: ArrayRef = Arc::new(Int32Array::from(vec![Some(1), None, Some(3), None]));
280+
281+
let result = invoke(
282+
vec![
283+
ColumnarValue::Array(a),
284+
ColumnarValue::Scalar(ScalarValue::Int32(Some(-1))),
285+
],
286+
4,
287+
)
288+
.into_array(4)
289+
.unwrap();
290+
291+
let expected: ArrayRef =
292+
Arc::new(Int32Array::from(vec![Some(1), Some(-1), Some(3), Some(-1)]));
293+
assert_eq!(&result, &expected);
294+
assert_eq!(result.null_count(), 0);
295+
}
296+
297+
/// All-scalar input short-circuits to the first non-null scalar.
298+
#[test]
299+
fn coalesce_kernel_all_scalars() {
300+
let result = invoke(
301+
vec![
302+
ColumnarValue::Scalar(ScalarValue::Utf8(None)),
303+
ColumnarValue::Scalar(ScalarValue::Utf8(Some("datafusion".into()))),
304+
ColumnarValue::Scalar(ScalarValue::Utf8(Some("unused".into()))),
305+
],
306+
1,
307+
);
308+
match result {
309+
ColumnarValue::Scalar(ScalarValue::Utf8(Some(s))) => {
310+
assert_eq!(s, "datafusion")
311+
}
312+
other => panic!("expected Utf8 scalar, got {other:?}"),
313+
}
314+
}
315+
316+
/// All-null scalars produce a null scalar rather than an error.
317+
#[test]
318+
fn coalesce_kernel_all_null_scalars() {
319+
let result = invoke(
320+
vec![
321+
ColumnarValue::Scalar(ScalarValue::Int32(None)),
322+
ColumnarValue::Scalar(ScalarValue::Int32(None)),
323+
],
324+
1,
325+
);
326+
match result {
327+
ColumnarValue::Scalar(v) => assert!(v.is_null()),
328+
other => panic!("expected null scalar, got {other:?}"),
329+
}
330+
}
331+
332+
/// All-null arrays produce an all-null array of the right type and length.
333+
#[test]
334+
fn coalesce_kernel_all_null_arrays() {
335+
let a: ArrayRef = Arc::new(StringArray::from(vec![None as Option<&str>, None]));
336+
let b: ArrayRef = Arc::new(StringArray::from(vec![None as Option<&str>, None]));
337+
338+
let result = invoke(vec![ColumnarValue::Array(a), ColumnarValue::Array(b)], 2)
339+
.into_array(2)
340+
.unwrap();
341+
342+
assert_eq!(result.len(), 2);
343+
assert_eq!(result.null_count(), 2);
344+
assert_eq!(result.data_type(), &DataType::Utf8);
345+
}
346+
347+
#[test]
348+
fn coalesce_kernel_rejects_empty_args() {
349+
let err = CoalesceFunc::new()
350+
.invoke_with_args(ScalarFunctionArgs {
351+
args: vec![],
352+
arg_fields: vec![],
353+
number_rows: 1,
354+
return_field: Field::new("f", DataType::Int32, true).into(),
355+
config_options: Arc::new(ConfigOptions::default()),
356+
})
357+
.unwrap_err();
358+
assert!(err.to_string().contains("It requires at least 1"));
359+
}
360+
361+
fn simplify(args: Vec<Expr>) -> ExprSimplifyResult {
362+
CoalesceFunc::new()
363+
.simplify(args, &SimplifyContext::default())
364+
.expect("simplify failed")
365+
}
366+
367+
/// Non-volatile arguments keep the lazy `CASE` rewrite.
368+
#[test]
369+
fn simplify_non_volatile_expands_to_case() {
370+
let result = simplify(vec![lit(1i64), lit(2i64)]);
371+
match result {
372+
ExprSimplifyResult::Simplified(Expr::Case(_)) => {}
373+
other => panic!("expected a CASE expression, got {other:?}"),
374+
}
375+
}
376+
377+
/// A volatile argument must NOT be expanded, because the `CASE` rewrite would
378+
/// name it twice and evaluate it twice. See issue #25477.
379+
#[test]
380+
fn simplify_volatile_is_left_intact() {
381+
let random =
382+
datafusion_expr::expr::ScalarFunction::new_udf(crate::math::random(), vec![]);
383+
let args = vec![Expr::ScalarFunction(random), lit(-1i64)];
384+
match simplify(args.clone()) {
385+
ExprSimplifyResult::Original(original) => assert_eq!(original, args),
386+
other => panic!("expected the original coalesce args, got {other:?}"),
387+
}
388+
}
389+
390+
/// A volatile argument nested inside a larger expression is also caught.
391+
#[test]
392+
fn simplify_nested_volatile_is_left_intact() {
393+
let random = Expr::ScalarFunction(
394+
datafusion_expr::expr::ScalarFunction::new_udf(crate::math::random(), vec![]),
395+
);
396+
let args = vec![random + lit(1.0f64), lit(-1.0f64)];
397+
match simplify(args.clone()) {
398+
ExprSimplifyResult::Original(original) => assert_eq!(original, args),
399+
other => panic!("expected the original coalesce args, got {other:?}"),
400+
}
401+
}
402+
403+
/// A single argument is still unwrapped, volatile or not (it is named once).
404+
#[test]
405+
fn simplify_single_volatile_arg_is_unwrapped() {
406+
let random = Expr::ScalarFunction(
407+
datafusion_expr::expr::ScalarFunction::new_udf(crate::math::random(), vec![]),
408+
);
409+
match simplify(vec![random.clone()]) {
410+
ExprSimplifyResult::Simplified(e) => assert_eq!(e, random),
411+
other => panic!("expected the argument itself, got {other:?}"),
412+
}
413+
}
414+
}

‎datafusion/sqllogictest/test_files/coalesce.slt‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -458,3 +458,78 @@ world
458458

459459
statement ok
460460
drop table ree_t;
461+
462+
##########
463+
# Volatile arguments must be evaluated exactly once
464+
# https://github.com/apache/datafusion/issues/25477
465+
##########
466+
467+
# `coalesce` on a volatile argument must NOT be rewritten to
468+
# `CASE WHEN a IS NOT NULL THEN a ELSE b END`, because that names `a` twice and
469+
# therefore draws it twice. It must stay a `coalesce` call so the runtime kernel
470+
# evaluates each argument exactly once.
471+
query TT
472+
explain select coalesce(nullif(floor(random() * 2), 0), -1) as c;
473+
----
474+
logical_plan
475+
01)Projection: coalesce(nullif(floor(random() * Float64(2)), Float64(0)), Float64(-1)) AS c
476+
02)--EmptyRelation: rows=1
477+
physical_plan
478+
01)ProjectionExec: expr=[coalesce(nullif(floor(random() * 2), 0), -1) as c]
479+
02)--PlaceholderRowExec
480+
481+
statement ok
482+
create table volatile_t as select * from generate_series(1, 100000) as t(i);
483+
484+
# The fallback `-1` makes the result non-nullable, and with a single evaluation
485+
# per row the result really never is null: every row is either the non-null
486+
# volatile draw or the literal. Before the fix this both produced ~25% spurious
487+
# NULLs and tripped the non-nullable assertion in Arrow.
488+
query II
489+
select count(*), count(coalesce(nullif(floor(random() * 2), 0), -1))
490+
from volatile_t;
491+
----
492+
100000 100000
493+
494+
# Every produced value must come from one of the two operands.
495+
query I
496+
select count(*)
497+
from (
498+
select coalesce(nullif(floor(random() * 2), 0), -1) as c from volatile_t
499+
)
500+
where c not in (1, -1);
501+
----
502+
0
503+
504+
# `nvl` delegates to `coalesce`, so it inherits the fix.
505+
query TT
506+
explain select nvl(nullif(floor(random() * 2), 0), -1) as c;
507+
----
508+
logical_plan
509+
01)Projection: nvl(nullif(floor(random() * Float64(2)), Float64(0)), Float64(-1)) AS c
510+
02)--EmptyRelation: rows=1
511+
physical_plan
512+
01)ProjectionExec: expr=[nvl(nullif(floor(random() * 2), 0), -1) as c]
513+
02)--PlaceholderRowExec
514+
515+
query II
516+
select count(*), count(nvl(nullif(floor(random() * 2), 0), -1))
517+
from volatile_t;
518+
----
519+
100000 100000
520+
521+
statement ok
522+
drop table volatile_t;
523+
524+
# Non-volatile `coalesce` keeps the lazy CASE rewrite (see also
525+
# select.slt, which asserts `coalesce(1, y/x)` never divides by zero).
526+
query TT
527+
explain select coalesce(column1, column2) as c from (values (1, 2), (null, 3)) t;
528+
----
529+
logical_plan
530+
01)Projection: CASE WHEN t.column1 IS NOT NULL THEN t.column1 ELSE t.column2 END AS c
531+
02)--SubqueryAlias: t
532+
03)----Values: (Int64(1), Int64(2)), (Int64(NULL), Int64(3))
533+
physical_plan
534+
01)ProjectionExec: expr=[CASE WHEN column1@0 IS NOT NULL THEN column1@0 ELSE column2@1 END as c]
535+
02)--DataSourceExec: partitions=1, partition_sizes=[1]

0 commit comments

Comments
 (0)