You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a274b3c
Browse filesBrowse the repository at this point in the historyBrowse files
fix: duplicate ordering state field names in partial aggregates (apache#25196)
## Which issue does this PR close?
* Closesapache#17715.
## Rationale for this change
Partial aggregate state schemas can contain duplicate field names when
multiple order-sensitive aggregate expressions use the same ordering
expression.
For example, `first_value(value ORDER BY timestamp)` and
`last_value(value ORDER BY timestamp)` both exposed the ordering state
field as `timestamp@0`.
This required `DFSchema::try_from` to skip `check_names()` as a
workaround. However, the `state_fields()` contract requires state field
names to be unique within the query.
## What changes are included in this PR?
* Namespace ordering state fields with the aggregate name and the
ordering-field position.
* Apply the same naming scheme to the default
`AggregateUDFImpl::state_fields()` implementation and to `FirstValue` /
`LastValue`.
* Re-enable `DFSchema::check_names()` in `TryFrom<SchemaRef> for
DFSchema`.
* Update the partial aggregate regression test to assert unique state
field names.
* Add a unit test for the default `state_fields()` implementation,
including duplicate ordering field names within a single aggregate.
For example, an ordering state field now has a name such as:
```text
first_value(value)[ordering_0]
```
rather than the unqualified:
```text
timestamp@0
```
The ordering position also disambiguates repeated ordering fields within
the same aggregate.
## What is the testing strategy for this PR?
The change is covered by:
* `test_partial_aggregate_state_fields_have_unique_names`
* `test_default_state_fields_namespaces_ordering_fields`
The following test suites pass locally:
```text
cargo test -p datafusion-expr
cargo test -p datafusion-functions-aggregate
cargo test -p datafusion-physical-plan
cargo test -p datafusion-common
cargo test -p datafusion --test core_integration
cargo test -p datafusion-ffi --features integration-tests
```
`cargo fmt --all -- --check` and `git diff --check` also pass.
## Are there any user-facing changes?
No public API signatures were changed. A new ordering_state_fields
utility was added to centralize the naming of ordering state fields.
Ordering state field names now use the aggregate namespace and ordering
position to guarantee uniqueness, allowing DFSchema::check_names() to be
re-enabled.
---------
Co-authored-by: Jay Zhan <[email protected]>
Co-authored-by: Andrew Lamb <[email protected]>
0 commit comments