Skip to content

Commit e72572d

Browse files
geoffreyclaudelukekim
authored andcommitted
fix: prevent panic and incorrect results for COUNT with ORDER BY (apache#24997)
## Which issue does this PR close? - Closes apache#25055. ## Rationale for this change `COUNT` does not depend on input order, but it inherited the default `AggregateOrderSensitivity::HardRequirement`. As detailed in apache#25055, this could pass ordering keys to count accumulators as additional arguments, causing incorrect results or a panic, and could introduce unnecessary sorting. ## What changes are included in this PR? Override `Count::order_sensitivity` to return `AggregateOrderSensitivity::Insensitive`. This makes aggregate ordering keys ineffective for physical planning: they are excluded from accumulator inputs and no `SortExec` is required solely for `COUNT`'s `ORDER BY`. ## What is the testing strategy for this PR? Regression tests in `aggregate.slt` cover grouped, multi-argument, and non-grouped counts. The data includes both a null ordering key, which must not affect the count, and a null counted argument, which must still be excluded. A bare global `COUNT(a ORDER BY b)` over the in-memory test table is folded by `AggregateStatistics` to `num_rows - null_count(a)`, so `CountAccumulator` never runs. The test uses `a + 0`, which preserves `a`'s nullness but prevents that fold, ensuring the regression exercises the accumulator. An `EXPLAIN` assertion verifies that the count does not require a sort. ## Are there any user-facing changes? `COUNT(... ORDER BY ...)` returns the correct count without panicking. Nulls in counted arguments retain their usual behavior; nulls in ordering keys no longer incorrectly exclude rows. (cherry picked from commit 224cc56) Conflict resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/sqllogictest/test_files/aggregate.slt: upstream appends this commit's COUNT ... ORDER BY block after other blocks that are not on 54.1, including the nested-aggregate block. Added only this commit's 43 lines, at the end of the 54.1 file.
1 parent 140c13f commit e72572d

2 files changed

Lines changed: 48 additions & 1 deletion

File tree

‎datafusion/functions-aggregate/src/count.rs‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ use datafusion_expr::{
3939
WindowFunctionDefinition,
4040
expr::WindowFunction,
4141
function::{AccumulatorArgs, StateFieldsArgs},
42-
utils::format_state_name,
42+
utils::{AggregateOrderSensitivity, format_state_name},
4343
};
4444
use datafusion_functions_aggregate_common::aggregate::count_distinct::PrimitiveDistinctCountGroupsAccumulator;
4545
use datafusion_functions_aggregate_common::aggregate::{
@@ -378,6 +378,10 @@ impl AggregateUDFImpl for Count {
378378
ReversedUDAF::Identical
379379
}
380380

381+
fn order_sensitivity(&self) -> AggregateOrderSensitivity {
382+
AggregateOrderSensitivity::Insensitive
383+
}
384+
381385
fn default_value(&self, _data_type: &DataType) -> Result<ScalarValue> {
382386
Ok(ScalarValue::Int64(Some(0)))
383387
}

‎datafusion/sqllogictest/test_files/aggregate.slt‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9018,3 +9018,46 @@ SET datafusion.execution.target_partitions = 4;
90189018

90199019
statement ok
90209020
DROP TABLE hits_raw;
9021+
9022+
# Tests for COUNT(... ORDER BY ...)
9023+
statement ok
9024+
CREATE TABLE count_order_by_t (a INT, b INT, c INT) AS VALUES
9025+
(1, NULL, 10),
9026+
(2, 20, 10),
9027+
(3, 30, 20),
9028+
(NULL, 40, 20);
9029+
9030+
# Single-argument grouped count with ORDER BY
9031+
query II rowsort
9032+
SELECT c, COUNT(a ORDER BY b) FROM count_order_by_t GROUP BY c;
9033+
----
9034+
10 2
9035+
20 1
9036+
9037+
# Multi-argument count with nullable ORDER BY key
9038+
query I
9039+
SELECT COUNT(a, c ORDER BY b) FROM count_order_by_t;
9040+
----
9041+
3
9042+
9043+
# Single-argument non-grouped count with nullable ORDER BY key
9044+
# A bare COUNT(a ORDER BY b) is folded to num_rows - null_count(a), bypassing
9045+
# CountAccumulator. a + 0 preserves nullness and forces execution.
9046+
query I
9047+
SELECT COUNT(a + 0 ORDER BY b) FROM count_order_by_t;
9048+
----
9049+
3
9050+
9051+
# COUNT does not require sorting its input, even with an explicit ORDER BY.
9052+
query TT
9053+
EXPLAIN SELECT COUNT(a + 0 ORDER BY b) FROM count_order_by_t;
9054+
----
9055+
logical_plan
9056+
01)Aggregate: groupBy=[[]], aggr=[[count(CAST(count_order_by_t.a AS Int64) + Int64(0)) ORDER BY [count_order_by_t.b ASC NULLS LAST]]]
9057+
02)--TableScan: count_order_by_t projection=[a, b]
9058+
physical_plan
9059+
01)AggregateExec: mode=Single, gby=[], aggr=[count(count_order_by_t.a + Int64(0)) ORDER BY [count_order_by_t.b ASC NULLS LAST]]
9060+
02)--DataSourceExec: partitions=1, partition_sizes=[1]
9061+
9062+
statement ok
9063+
DROP TABLE count_order_by_t;

0 commit comments

Comments
 (0)