Skip to content

Commit 636590b

Browse files
adriangbclaudejayzhan211
authored
test: count evaluations of volatile calls in sqllogictests (apache#25458)
## Which issue does this PR close? - Part of apache#25459. It does not fix a bug. It adds a test helper for the class of bug where the optimizer evaluates a volatile call more times than the query says. ## Rationale for this change Tests for these bugs used `random()` and inferred the number of evaluations from the result: they compared two reads of the same value, ran 10000–100000 rows and checked that no row broke a filter, or wrapped arguments in `case when random() >= 0 then x end`. These tests are indirect, and a reader cannot see what they check. A function that counts its own evaluations makes them exact. An earlier version of this PR added a check that walked the optimized plan. Run on every query of the sqllogictest suite with the bugs reverted, it caught none of them, and it could not see duplication in the physical plan. This version replaces it. ## What changes are included in this PR? Three functions, registered in every sqllogictest context (`datafusion/sqllogictest/src/test_context/evaluation_counter.rs`): - `counted_volatile(x)` returns `x` and is volatile, like `random()` - `counted_immutable(x)` returns `x` and is immutable - `evaluations()` returns the number of rows that the two functions above were evaluated for since its last call, and resets the count ``` query I SELECT count(*) FROM (SELECT counted_volatile(a) AS r FROM t) WHERE r > 5; ---- 5 query I SELECT evaluations(); ---- 10 ``` The count is per row, not per call, so it does not depend on batch size, and it is correct for `CASE` and `COALESCE`, which evaluate a branch on part of a batch. Commits, one per step: 1. Add the functions. 2. Port the tests for apache#25415 (`push_down_filter_regression.slt`). 3. Port the tests for apache#23220 and apache#24678 (`projection_pushdown.slt`). 4. Port the tests for apache#25477 (`coalesce.slt`). 5. Port the tests that two volatile calls are not merged (`functions.slt`, `aggregates_simplify.slt`). 6. Add `volatile_evaluations.slt`: filter pushdown, joins, unions, `HAVING`, and two known double evaluations: apache#26910 (preimage, a bug) and apache#25457 (`BETWEEN`, accepted behaviour). Not ported: `cse.slt` (its `EXPLAIN` tests check which subexpressions are hoisted, not only how many evaluations), `cte_materialized.slt` (its results are already exact), and the files that use `random()` only as an expression that cannot be pushed down. ## Are these changes tested? Each ported test was checked against its fix reverted: | Fix | Count with the fix reverted | |---|---| | apache#25416 (volatile group key) | 13 and 17, expected 10 | | apache#24720 (leaf extraction) | 20, expected 10 | | apache#23395 (file scan projection, physical plan) | 6, expected 3 | | apache#25561 (`COALESCE`) | 30, expected 20 | ## Are there any user-facing changes? No. The functions exist only in the sqllogictest runner. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Jay Zhan <[email protected]>
1 parent 934691e commit 636590b

8 files changed

Lines changed: 439 additions & 61 deletions

File tree

‎datafusion/sqllogictest/src/test_context.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ use log::info;
6060
use tempfile::TempDir;
6161

6262
mod custom_sort_pushdown;
63+
mod evaluation_counter;
6364
mod range_partitioning;
6465

6566
/// Context for running tests
@@ -162,6 +163,7 @@ impl TestContext {
162163
let state = state_builder.build();
163164

164165
let mut test_ctx = TestContext::new(SessionContext::new_with_state(state));
166+
evaluation_counter::register_evaluation_counter(test_ctx.session_ctx());
165167

166168
match file_name {
167169
"sort_pushdown.slt" => {
Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
// SPDX-License-Identifier: Apache-2.0
2+
// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership
3+
// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements
4+
5+
//! Functions that count how many rows a call is evaluated for, so a test can
6+
//! check that the optimizer does not evaluate a call more (or fewer) times
7+
//! than the query says:
8+
//!
9+
//! - `counted_volatile(x)` returns `x` and is volatile, like `random()`.
10+
//! - `counted_immutable(x)` returns `x` and is immutable.
11+
//! - `evaluations()` returns the number of rows that the two functions above
12+
//! were evaluated for since the last call to `evaluations()`, and resets it.
13+
//!
14+
//! The count is per row, not per call: a function is invoked once per batch,
15+
//! and `CASE` evaluates its branches on a subset of a batch.
16+
//!
17+
//! ```text
18+
//! statement ok
19+
//! SELECT k FROM (SELECT counted_volatile(a) AS k FROM t) WHERE k > 1;
20+
//!
21+
//! query I
22+
//! SELECT evaluations();
23+
//! ----
24+
//! 3
25+
//! ```
26+
27+
use std::hash::{Hash, Hasher};
28+
use std::sync::Arc;
29+
use std::sync::atomic::{AtomicUsize, Ordering};
30+
31+
use arrow::datatypes::DataType;
32+
use datafusion::common::{Result, ScalarValue};
33+
use datafusion::logical_expr::{
34+
ColumnarValue, ScalarFunctionArgs, ScalarUDF, ScalarUDFImpl, Signature, Volatility,
35+
};
36+
use datafusion::prelude::SessionContext;
37+
38+
/// Registers `counted_volatile`, `counted_immutable` and `evaluations` with `ctx`. Each
39+
/// context gets its own counter.
40+
pub fn register_evaluation_counter(ctx: &SessionContext) {
41+
let counter = Arc::new(AtomicUsize::new(0));
42+
for (name, volatility) in [
43+
("counted_volatile", Volatility::Volatile),
44+
("counted_immutable", Volatility::Immutable),
45+
] {
46+
ctx.register_udf(ScalarUDF::new_from_impl(Counted {
47+
name,
48+
signature: Signature::any(1, volatility),
49+
counter: Arc::clone(&counter),
50+
}));
51+
}
52+
ctx.register_udf(ScalarUDF::new_from_impl(Evaluations {
53+
signature: Signature::nullary(Volatility::Volatile),
54+
counter,
55+
}));
56+
}
57+
58+
/// Returns its argument and adds the number of rows to `counter`.
59+
#[derive(Debug)]
60+
struct Counted {
61+
name: &'static str,
62+
signature: Signature,
63+
counter: Arc<AtomicUsize>,
64+
}
65+
66+
impl ScalarUDFImpl for Counted {
67+
fn name(&self) -> &str {
68+
self.name
69+
}
70+
71+
fn signature(&self) -> &Signature {
72+
&self.signature
73+
}
74+
75+
fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
76+
Ok(arg_types[0].clone())
77+
}
78+
79+
fn invoke_with_args(&self, mut args: ScalarFunctionArgs) -> Result<ColumnarValue> {
80+
self.counter.fetch_add(args.number_rows, Ordering::SeqCst);
81+
Ok(args.args.swap_remove(0))
82+
}
83+
}
84+
85+
/// Returns the count and resets it.
86+
#[derive(Debug)]
87+
struct Evaluations {
88+
signature: Signature,
89+
counter: Arc<AtomicUsize>,
90+
}
91+
92+
impl ScalarUDFImpl for Evaluations {
93+
fn name(&self) -> &str {
94+
"evaluations"
95+
}
96+
97+
fn signature(&self) -> &Signature {
98+
&self.signature
99+
}
100+
101+
fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
102+
Ok(DataType::Int64)
103+
}
104+
105+
fn invoke_with_args(&self, _args: ScalarFunctionArgs) -> Result<ColumnarValue> {
106+
let count = self.counter.swap(0, Ordering::SeqCst);
107+
Ok(ColumnarValue::Scalar(ScalarValue::Int64(Some(
108+
count as i64,
109+
))))
110+
}
111+
}
112+
113+
// Two instances are equal only if they share a counter.
114+
115+
impl PartialEq for Counted {
116+
fn eq(&self, other: &Self) -> bool {
117+
self.name == other.name && Arc::ptr_eq(&self.counter, &other.counter)
118+
}
119+
}
120+
121+
impl Eq for Counted {}
122+
123+
impl Hash for Counted {
124+
fn hash<H: Hasher>(&self, state: &mut H) {
125+
self.name.hash(state);
126+
Arc::as_ptr(&self.counter).hash(state);
127+
}
128+
}
129+
130+
impl PartialEq for Evaluations {
131+
fn eq(&self, other: &Self) -> bool {
132+
Arc::ptr_eq(&self.counter, &other.counter)
133+
}
134+
}
135+
136+
impl Eq for Evaluations {}
137+
138+
impl Hash for Evaluations {
139+
fn hash<H: Hasher>(&self, state: &mut H) {
140+
Arc::as_ptr(&self.counter).hash(state);
141+
}
142+
}

‎datafusion/sqllogictest/test_files/aggregates_simplify.slt‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -259,11 +259,16 @@ physical_plan
259259
01)AggregateExec: mode=Single, gby=[], aggr=[sum(sum_simplify_t.column1 + Int64(1)) FILTER (WHERE sum_simplify_t.column1 > Int64(1)), sum(sum_simplify_t.column1 + Int64(1)) FILTER (WHERE sum_simplify_t.column1 > Int64(0))]
260260
02)--DataSourceExec: partitions=1, partition_sizes=[1]
261261

262-
# volatile aggregate arguments
263-
query B
264-
SELECT SUM(random() + 1) < SUM(random() + 2) FROM sum_simplify_t;
262+
# volatile aggregate arguments: each call is evaluated one time per row
263+
query II
264+
SELECT SUM(counted_volatile(column1) + 1), SUM(counted_volatile(column1) + 2) FROM sum_simplify_t;
265+
----
266+
7 10
267+
268+
query I
269+
SELECT evaluations();
265270
----
266-
true
271+
8
267272

268273
query TT
269274
EXPLAIN SELECT SUM(random() + 1) < SUM(random() + 2) FROM sum_simplify_t;

‎datafusion/sqllogictest/test_files/coalesce.slt‎

Lines changed: 42 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -463,26 +463,34 @@ physical_plan
463463
01)ProjectionExec: expr=[coalesce(nullif(floor(random() * 2), 0), -1) as c]
464464
02)--PlaceholderRowExec
465465

466-
# With two draws, about 25% of rows were NULL (and Arrow rejected them because
467-
# the `-1` fallback makes the result non-nullable). With one draw, every row is
468-
# either 1 or -1.
469-
statement ok
470-
create table volatile_t as select * from generate_series(1, 100000) as t(i);
466+
# A volatile first argument is evaluated one time per row, not once for the
467+
# null test and again for the value.
468+
query II
469+
select
470+
count(coalesce(nullif(counted_volatile(value) % 2, 0), -1)),
471+
sum(coalesce(nullif(counted_volatile(value) % 2, 0), -1))
472+
from generate_series(1, 10);
473+
----
474+
10 0
475+
476+
query I
477+
select evaluations();
478+
----
479+
20
471480

472-
query III
481+
query II
473482
select
474-
count(coalesce(nullif(floor(random() * 2), 0), -1)),
475-
count(nvl(nullif(floor(random() * 2), 0), -1)),
476-
count(*) filter (where coalesce(nullif(floor(random() * 2), 0), -1) not in (1, -1))
477-
from volatile_t;
483+
count(nvl(nullif(counted_volatile(value) % 2, 0), -1)),
484+
sum(nvl(nullif(counted_volatile(value) % 2, 0), -1))
485+
from generate_series(1, 10);
478486
----
479-
100000 100000 0
487+
10 0
480488

481-
statement ok
482-
drop table volatile_t;
489+
query I
490+
select evaluations();
491+
----
492+
20
483493

484-
# `case when random() >= 0 then x end` is volatile but always equals `x`, which
485-
# makes the results of the volatile code path deterministic.
486494
statement ok
487495
create table volatile_args(a int, b int, c int, s varchar, x int, y int) as values
488496
(1, 9, 8, null, 0, 1),
@@ -492,25 +500,37 @@ create table volatile_args(a int, b int, c int, s varchar, x int, y int) as valu
492500

493501
query IIIIT
494502
select
495-
coalesce(case when random() >= 0 then a end, b, c),
496-
coalesce(case when random() >= 0 then a end, -1),
497-
coalesce(case when random() >= 0 then a end, null::int),
498-
coalesce(a, case when random() >= 0 then b end, c),
499-
coalesce(case when random() >= 0 then s end, 'none')
503+
coalesce(counted_volatile(a), b, c),
504+
coalesce(counted_volatile(a), -1),
505+
coalesce(counted_volatile(a), null::int),
506+
coalesce(a, counted_volatile(b), c),
507+
coalesce(counted_volatile(s), 'none')
500508
from volatile_args;
501509
----
502510
1 1 1 1 none
503511
2 -1 NULL 2 b
504512
3 -1 NULL 3 none
505513
NULL -1 NULL NULL none
506514

515+
# Four rows for each of the first three and the last call, and three for
516+
# `counted_volatile(b)`, which is reached only where `a` is null.
517+
query I
518+
select evaluations();
519+
----
520+
19
521+
507522
query TI
508523
select
509-
coalesce(case when random() >= 0 then null::varchar end, 'datafusion', 'unused'),
510-
coalesce(case when random() >= 0 then null::int end, null::int);
524+
coalesce(counted_volatile(null::varchar), 'datafusion', 'unused'),
525+
coalesce(counted_volatile(null::int), null::int);
511526
----
512527
datafusion NULL
513528

529+
query I
530+
select evaluations();
531+
----
532+
2
533+
514534
# Arguments before the first volatile one keep the CASE rewrite; only the rest
515535
# is left to `coalesce`.
516536
query TT

‎datafusion/sqllogictest/test_files/functions.slt‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1304,9 +1304,14 @@ ac
13041304

13051305
# Verify that multiple calls to volatile functions like `random()` are not combined / optimized away
13061306
query B
1307-
SELECT r FROM (SELECT r1 == r2 r, r1, r2 FROM (SELECT random()+1 r1, random()+1 r2) WHERE r1 > 0 AND r2 > 0)
1307+
SELECT r FROM (SELECT r1 == r2 r, r1, r2 FROM (SELECT counted_volatile(1)+1 r1, counted_volatile(1)+1 r2) WHERE r1 > 0 AND r2 > 0)
13081308
----
1309-
false
1309+
true
1310+
1311+
query I
1312+
SELECT evaluations();
1313+
----
1314+
2
13101315

13111316
# Verify that rand() is accepted as an alias for random()
13121317
query TB

‎datafusion/sqllogictest/test_files/projection_pushdown.slt‎

Lines changed: 45 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -2109,7 +2109,7 @@ SET datafusion.execution.target_partitions = 4;
21092109
# reused. Projection pushdown must not merge the outer projection into the file
21102110
# scan when doing so would inline and duplicate the volatile expression.
21112111
# Reproduces only against a file scan (not an in-memory table); if the volatile
2112-
# expression is duplicated, the two references diverge and `x = y` is false.
2112+
# expression is duplicated, it is evaluated two times per row.
21132113
#####################
21142114

21152115
statement ok
@@ -2121,15 +2121,20 @@ statement ok
21212121
CREATE EXTERNAL TABLE volatile_scan STORED AS PARQUET
21222122
LOCATION 'test_files/scratch/projection_pushdown/volatile.parquet';
21232123

2124-
# The two references to the aliased `random()` value must be equal on every
2125-
# row: the expression is evaluated once and reused, not inlined twice.
2126-
query B rowsort
2127-
SELECT s.x = s.y
2128-
FROM (SELECT r AS x, r AS y FROM (SELECT random() AS r FROM volatile_scan) AS t) AS s;
2124+
# The aliased volatile value is evaluated one time per row and reused, not
2125+
# inlined twice.
2126+
query II rowsort
2127+
SELECT s.x, s.y
2128+
FROM (SELECT r AS x, r AS y FROM (SELECT counted_volatile(id) AS r FROM volatile_scan) AS t) AS s;
21292129
----
2130-
true
2131-
true
2132-
true
2130+
1 1
2131+
2 2
2132+
3 3
2133+
2134+
query I
2135+
SELECT evaluations();
2136+
----
2137+
3
21332138

21342139
#####################
21352140
# Section: leaf expression extraction does not duplicate volatile expressions
@@ -2176,37 +2181,52 @@ logical_plan
21762181
statement ok
21772182
set datafusion.explain.logical_plan_only = false;
21782183

2179-
# Every row's `field` is the `a` field of that row's own struct.
2180-
query B
2181-
SELECT bool_and(field = s['a'])
2184+
# The struct is evaluated one time per row; `field` reads it.
2185+
query II
2186+
SELECT count(s), sum(field)
21822187
FROM (
21832188
SELECT s, s['a'] AS field
2184-
FROM (SELECT named_struct('a', random()) AS s FROM generate_series(1, 1000))
2189+
FROM (SELECT named_struct('a', counted_volatile(value)) AS s FROM generate_series(1, 10))
21852190
);
21862191
----
2187-
true
2192+
10 55
2193+
2194+
query I
2195+
SELECT evaluations();
2196+
----
2197+
10
21882198

2189-
# Same invariant when the extraction is routed into one side of a join.
2190-
query B
2191-
SELECT bool_and(field = s['a'])
2199+
# Same when the extraction is routed into one side of a join.
2200+
query II
2201+
SELECT count(s), sum(field)
21922202
FROM (
21932203
SELECT l.s, l.s['a'] AS field
2194-
FROM (SELECT named_struct('a', random()) AS s FROM generate_series(1, 1000)) AS l
2204+
FROM (SELECT named_struct('a', counted_volatile(value)) AS s FROM generate_series(1, 10)) AS l
21952205
INNER JOIN (SELECT 1 AS k) AS r ON true
21962206
);
21972207
----
2198-
true
2208+
10 55
2209+
2210+
query I
2211+
SELECT evaluations();
2212+
----
2213+
10
21992214

2200-
# Every surviving row satisfies the predicate it was filtered on.
2201-
query B
2202-
SELECT bool_and(s['a'] > 0.5)
2215+
# The filter tests the struct that is returned, not a second evaluation.
2216+
query I
2217+
SELECT count(s)
22032218
FROM (
22042219
SELECT s
2205-
FROM (SELECT named_struct('a', random()) AS s FROM generate_series(1, 1000))
2206-
WHERE s['a'] > 0.5
2220+
FROM (SELECT named_struct('a', counted_volatile(value)) AS s FROM generate_series(1, 10))
2221+
WHERE s['a'] > 5
22072222
);
22082223
----
2209-
true
2224+
5
2225+
2226+
query I
2227+
SELECT evaluations();
2228+
----
2229+
10
22102230

22112231
#####################
22122232
# Section: expensive expressions are not re-inlined by projection pushdown

0 commit comments

Comments
 (0)