Skip to content

Commit a9579bd

Browse files
cdelmonte-zgjayzhan211alamb
authored
fix: duplicate ordering state field names in partial aggregates (apache#25196)
## Which issue does this PR close? * Closes apache#17715. ## Rationale for this change Partial aggregate state schemas can contain duplicate field names when multiple order-sensitive aggregate expressions use the same ordering expression. For example, `first_value(value ORDER BY timestamp)` and `last_value(value ORDER BY timestamp)` both exposed the ordering state field as `timestamp@0`. This required `DFSchema::try_from` to skip `check_names()` as a workaround. However, the `state_fields()` contract requires state field names to be unique within the query. ## What changes are included in this PR? * Namespace ordering state fields with the aggregate name and the ordering-field position. * Apply the same naming scheme to the default `AggregateUDFImpl::state_fields()` implementation and to `FirstValue` / `LastValue`. * Re-enable `DFSchema::check_names()` in `TryFrom<SchemaRef> for DFSchema`. * Update the partial aggregate regression test to assert unique state field names. * Add a unit test for the default `state_fields()` implementation, including duplicate ordering field names within a single aggregate. For example, an ordering state field now has a name such as: ```text first_value(value)[ordering_0] ``` rather than the unqualified: ```text timestamp@0 ``` The ordering position also disambiguates repeated ordering fields within the same aggregate. ## What is the testing strategy for this PR? The change is covered by: * `test_partial_aggregate_state_fields_have_unique_names` * `test_default_state_fields_namespaces_ordering_fields` The following test suites pass locally: ```text cargo test -p datafusion-expr cargo test -p datafusion-functions-aggregate cargo test -p datafusion-physical-plan cargo test -p datafusion-common cargo test -p datafusion --test core_integration cargo test -p datafusion-ffi --features integration-tests ``` `cargo fmt --all -- --check` and `git diff --check` also pass. ## Are there any user-facing changes? No public API signatures were changed. A new ordering_state_fields utility was added to centralize the naming of ordering state fields. Ordering state field names now use the aggregate namespace and ordering position to guarantee uniqueness, allowing DFSchema::check_names() to be re-enabled. --------- Co-authored-by: Jay Zhan <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>
1 parent 7f3e657 commit a9579bd

7 files changed

Lines changed: 132 additions & 24 deletions

File tree

‎datafusion/common/src/dfschema.rs‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1138,11 +1138,7 @@ impl TryFrom<SchemaRef> for DFSchema {
11381138
field_qualifiers: vec![None; field_count],
11391139
functional_dependencies: FunctionalDependencies::empty(),
11401140
};
1141-
// Without checking names, because schema here may have duplicate field names.
1142-
// For example, Partial AggregateMode will generate duplicate field names from
1143-
// state_fields.
1144-
// See <https://github.com/apache/datafusion/issues/17715>
1145-
// dfschema.check_names()?;
1141+
dfschema.check_names()?;
11461142
Ok(dfschema)
11471143
}
11481144
}

‎datafusion/core/tests/dataframe/mod.rs‎

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -7611,7 +7611,7 @@ async fn execute_logical_plan_rejects_duplicate_unqualified_names_in_replace_vie
76117611
}
76127612

76137613
#[tokio::test]
7614-
async fn test_duplicate_state_fields_for_dfschema_construct() -> Result<()> {
7614+
async fn test_partial_aggregate_state_fields_have_unique_names() -> Result<()> {
76157615
let ctx = SessionContext::new();
76167616

76177617
// Simple schema with just the fields we need
@@ -7686,28 +7686,22 @@ async fn test_duplicate_state_fields_for_dfschema_construct() -> Result<()> {
76867686
)
76877687
.expect("Failed to build partial agg");
76887688

7689-
// Assert that the schema field names match the expected names
76907689
let expected_field_names = vec![
76917690
"date",
76927691
"ticker",
76937692
"first_value(value)[first_value]",
7694-
"timestamp@0",
7693+
"first_value(value)[ordering_0]",
76957694
"first_value(value)[first_value_is_set]",
76967695
"last_value(value)[last_value]",
7697-
"timestamp@0",
7696+
"last_value(value)[ordering_0]",
76987697
"last_value(value)[last_value_is_set]",
76997698
];
77007699

77017700
let binding = partial_agg.schema();
77027701
let actual_field_names: Vec<_> = binding.fields().iter().map(|f| f.name()).collect();
77037702
assert_eq!(actual_field_names, expected_field_names);
77047703

7705-
// Ensure that DFSchema::try_from does not fail
7706-
let partial_agg_exec_schema = DFSchema::try_from(partial_agg.schema());
7707-
assert!(
7708-
partial_agg_exec_schema.is_ok(),
7709-
"Expected get AggregateExec schema to succeed with duplicate state fields"
7710-
);
7704+
DFSchema::try_from(partial_agg.schema())?;
77117705

77127706
Ok(())
77137707
}

‎datafusion/expr/src/udaf.rs‎

Lines changed: 75 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,7 @@ use crate::function::{
4141
};
4242
use crate::groups_accumulator::GroupsAccumulator;
4343
use crate::udf_eq::UdfEq;
44-
use crate::utils::AggregateOrderSensitivity;
45-
use crate::utils::format_state_name;
44+
use crate::utils::{AggregateOrderSensitivity, format_state_name, ordering_state_fields};
4645
use crate::{Accumulator, Expr, expr_vec_fmt};
4746
use crate::{Documentation, Signature};
4847

@@ -614,7 +613,7 @@ pub trait AggregateUDFImpl: Debug + DynEq + DynHash + Send + Sync + Any {
614613
Ok(fields
615614
.into_iter()
616615
.map(Arc::new)
617-
.chain(args.ordering_fields.to_vec())
616+
.chain(ordering_state_fields(args.name, args.ordering_fields))
618617
.collect())
619618
}
620619

@@ -1762,7 +1761,7 @@ pub enum DistinctHandling {
17621761
#[cfg(test)]
17631762
mod test {
17641763
use crate::{AggregateUDF, AggregateUDFImpl};
1765-
use arrow::datatypes::{DataType, FieldRef};
1764+
use arrow::datatypes::{DataType, Field, FieldRef};
17661765
use datafusion_common::Result;
17671766
use datafusion_expr_common::accumulator::Accumulator;
17681767
use datafusion_expr_common::signature::{Signature, Volatility};
@@ -1771,6 +1770,7 @@ mod test {
17711770
};
17721771
use std::cmp::Ordering;
17731772
use std::hash::{DefaultHasher, Hash, Hasher};
1773+
use std::sync::Arc;
17741774

17751775
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
17761776
struct AMeanUdf {
@@ -1847,6 +1847,44 @@ mod test {
18471847
}
18481848
}
18491849

1850+
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
1851+
struct DefaultStateFieldsUdf {
1852+
signature: Signature,
1853+
}
1854+
1855+
impl DefaultStateFieldsUdf {
1856+
fn new() -> Self {
1857+
Self {
1858+
signature: Signature::uniform(
1859+
1,
1860+
vec![DataType::Float64],
1861+
Volatility::Immutable,
1862+
),
1863+
}
1864+
}
1865+
}
1866+
1867+
impl AggregateUDFImpl for DefaultStateFieldsUdf {
1868+
fn name(&self) -> &str {
1869+
"default_state_fields"
1870+
}
1871+
1872+
fn signature(&self) -> &Signature {
1873+
&self.signature
1874+
}
1875+
1876+
fn return_type(&self, _args: &[DataType]) -> Result<DataType> {
1877+
unimplemented!()
1878+
}
1879+
1880+
fn accumulator(
1881+
&self,
1882+
_acc_args: AccumulatorArgs,
1883+
) -> Result<Box<dyn Accumulator>> {
1884+
unimplemented!()
1885+
}
1886+
}
1887+
18501888
#[test]
18511889
fn test_partial_eq() {
18521890
let a1 = AggregateUDF::from(AMeanUdf::new());
@@ -1875,4 +1913,37 @@ mod test {
18751913
value.hash(hasher);
18761914
hasher.finish()
18771915
}
1916+
1917+
#[test]
1918+
fn test_default_state_fields_namespaces_ordering_fields() -> Result<()> {
1919+
let udf = DefaultStateFieldsUdf::new();
1920+
1921+
let input_fields = vec![Arc::new(Field::new("value", DataType::Float64, true))];
1922+
1923+
let ordering_fields = vec![
1924+
Arc::new(Field::new("timestamp@0", DataType::Int64, true)),
1925+
Arc::new(Field::new("timestamp@0", DataType::Int64, true)),
1926+
];
1927+
1928+
let fields = udf.state_fields(StateFieldsArgs {
1929+
name: "my_agg(value)",
1930+
input_fields: &input_fields,
1931+
return_field: Arc::new(Field::new("result", DataType::Float64, true)),
1932+
ordering_fields: &ordering_fields,
1933+
is_distinct: false,
1934+
})?;
1935+
1936+
let names: Vec<_> = fields.iter().map(|f| f.name().as_str()).collect();
1937+
1938+
assert_eq!(
1939+
names,
1940+
vec![
1941+
"my_agg(value)[value]",
1942+
"my_agg(value)[ordering_0]",
1943+
"my_agg(value)[ordering_1]",
1944+
]
1945+
);
1946+
1947+
Ok(())
1948+
}
18781949
}

‎datafusion/expr/src/utils.rs‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ use crate::{
2828
};
2929
use datafusion_expr_common::signature::{Signature, TypeSignature};
3030

31-
use arrow::datatypes::{DataType, Field, Schema};
31+
use arrow::datatypes::{DataType, Field, FieldRef, Schema};
3232
use datafusion_common::tree_node::{
3333
Transformed, TransformedResult, TreeNode, TreeNodeRecursion,
3434
};
@@ -1485,6 +1485,23 @@ pub fn format_state_name(name: &str, state_name: &str) -> String {
14851485
format!("{name}[{state_name}]")
14861486
}
14871487

1488+
/// Creates aggregate state fields for ordering expressions with unique names.
1489+
///
1490+
/// Each field is renamed using the aggregate name and its ordering position.
1491+
pub fn ordering_state_fields(
1492+
name: &str,
1493+
ordering_fields: &[FieldRef],
1494+
) -> impl Iterator<Item = FieldRef> {
1495+
ordering_fields.iter().enumerate().map(|(idx, field)| {
1496+
Arc::new(
1497+
field
1498+
.as_ref()
1499+
.clone()
1500+
.with_name(format_state_name(name, &format!("ordering_{idx}"))),
1501+
)
1502+
})
1503+
}
1504+
14881505
/// Determine the set of [`Column`]s produced by the subquery.
14891506
pub fn collect_subquery_cols(
14901507
exprs: &[Expr],

‎datafusion/ffi/src/udaf/mod.rs‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -812,6 +812,9 @@ mod tests {
812812
);
813813

814814
let a_field = Arc::new(Field::new("a", DataType::Float64, true));
815+
816+
let expected_ordering_field =
817+
Arc::new(a_field.as_ref().clone().with_name("a[ordering_0]"));
815818
let state_fields = foreign_udaf.state_fields(StateFieldsArgs {
816819
name: "a",
817820
input_fields: &[Field::new("f", DataType::Float64, true).into()],
@@ -821,7 +824,7 @@ mod tests {
821824
})?;
822825

823826
assert_eq!(state_fields.len(), 3);
824-
assert_eq!(state_fields[1], a_field);
827+
assert_eq!(state_fields[1], expected_ordering_field);
825828
Ok(())
826829
}
827830

‎datafusion/functions-aggregate/src/first_last.rs‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,9 @@ use datafusion_common::{
4040
not_impl_err,
4141
};
4242
use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
43-
use datafusion_expr::utils::{AggregateOrderSensitivity, format_state_name};
43+
use datafusion_expr::utils::{
44+
AggregateOrderSensitivity, format_state_name, ordering_state_fields,
45+
};
4446
use datafusion_expr::{
4547
Accumulator, AggregateUDFImpl, Documentation, EmitTo, Expr, ExprFunctionExt,
4648
GroupsAccumulator, ReversedUDAF, Signature, SortExpr, Volatility,
@@ -332,7 +334,7 @@ impl AggregateUDFImpl for FirstValue {
332334
)
333335
.into(),
334336
];
335-
fields.extend(args.ordering_fields.iter().cloned());
337+
fields.extend(ordering_state_fields(args.name, args.ordering_fields));
336338
fields.push(
337339
Field::new(
338340
format_state_name(args.name, "first_value_is_set"),
@@ -1252,7 +1254,7 @@ impl AggregateUDFImpl for LastValue {
12521254
)
12531255
.into(),
12541256
];
1255-
fields.extend(args.ordering_fields.iter().cloned());
1257+
fields.extend(ordering_state_fields(args.name, args.ordering_fields));
12561258
fields.push(
12571259
Field::new(
12581260
format_state_name(args.name, "last_value_is_set"),

‎docs/source/library-user-guide/upgrading/56.0.0.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -413,6 +413,31 @@ Most existing call sites using `Vec` require no changes. Code that relies on
413413
the exact non-generic function signature of `DataFrame::from_columns` may need
414414
to be updated to account for the new generic API.
415415

416+
### Aggregate ordering state field names are now namespaced
417+
418+
Ordering expressions exposed as top-level aggregate state fields now use names
419+
derived from the aggregate name and their ordering position.
420+
421+
For example:
422+
423+
```text
424+
first_value(value)[ordering_0]
425+
```
426+
427+
is now used instead of an unqualified ordering field name such as:
428+
429+
```text
430+
timestamp@0
431+
```
432+
433+
This guarantees unique aggregate state field names and allows
434+
`DFSchema::check_names()` to be enforced.
435+
436+
**Who is affected:**
437+
438+
Users or integrations that inspect aggregate state field names directly,
439+
including custom UDAFs and FFI integrations.
440+
416441
### Physical filter pushdown resolves columns by position
417442

418443
`datafusion_physical_plan::filter_pushdown::ChildFilterDescription::from_child`

0 commit comments

Comments
 (0)