Skip to content

Commit c280a4b

Browse files
authored
Fix GroupOrdering memory descriptor double-counting (apache#26074)
## Which issue does this PR close? - Part of apache#23393 ## Rationale for this change `GroupOrdering::size()` currently counts both the outer enum descriptor and the inline active variant descriptor. Because the active variant is stored inside `GroupOrdering`, this double-counts inline memory for partial and full ordering states. This change makes the `GroupOrdering` enum the single owner of the descriptor charge. Partial ordering continues to include retained `order_indices` capacity and state-owned allocations, while full ordering adds no additional descriptor charge. ## What changes are included in this PR? - Make `GroupOrdering::size()` charge `size_of::<GroupOrdering>()` exactly once for `None`, `Partial`, and `Full`. - Rename `GroupOrderingPartial::size()` to `heap_size()` and make it report only: - retained `order_indices` allocation - allocations owned by the partial ordering state - Remove `GroupOrderingFull::size()`, since full ordering has no additional heap allocation to add beyond the outer enum descriptor. - Document the ownership boundary in `GroupOrdering::size()` and `GroupOrderingPartial::heap_size()`. - Add deterministic size-accounting tests for none, full, and partial ordering states. ## Are these changes tested? Yes. This PR adds: - `test_size_none`, which verifies that `GroupOrdering::None` reports exactly `size_of::<GroupOrdering>()`. - `test_size_full`, which verifies that full ordering reports exactly one `GroupOrdering` descriptor before and after `new_groups`, `remove_groups`, `input_done`, and `reset`. - `test_size_partial_retained_allocations`, which verifies that: - grown-and-truncated `order_indices` capacity remains charged - a variable-width `ScalarValue::Utf8` sort key is included in the reported size - replacing the sort key charges the replacement rather than retaining the old key - `input_done` and `reset` drop state-owned key allocations while retaining the `order_indices` capacity - repopulating the state causes the key allocation to be charged again The tests use deterministic `size_of`, vector capacity, and `ScalarValue::size()` expectations rather than allocator-observed byte counts. ## Are there any user-facing changes? No public API changes are introduced. This changes internal memory accounting reported by `GroupOrdering::size()` so that inline descriptor memory is no longer double-counted. Grouping and aggregation behavior is otherwise unchanged by this patch. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.
1 parent 07077ff commit c280a4b

3 files changed

Lines changed: 47 additions & 14 deletions

File tree

‎datafusion/physical-plan/src/aggregates/order/full.rs‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,6 @@
1616
// under the License.
1717

1818
use datafusion_expr::EmitTo;
19-
use std::mem::size_of;
2019

2120
/// Tracks grouping state when the data is ordered entirely by its
2221
/// group keys
@@ -143,10 +142,6 @@ impl GroupOrderingFull {
143142
}
144143
};
145144
}
146-
147-
pub(crate) fn size(&self) -> usize {
148-
size_of::<Self>()
149-
}
150145
}
151146

152147
impl Default for GroupOrderingFull {

‎datafusion/physical-plan/src/aggregates/order/mod.rs‎

Lines changed: 42 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -148,12 +148,15 @@ impl GroupOrdering {
148148
}
149149

150150
/// Returns the size of memory used by the ordering state, in bytes.
151+
///
152+
/// Includes the enum descriptor once (covering the inline active variant)
153+
/// and any heap allocations retained by that variant.
151154
pub fn size(&self) -> usize {
152155
size_of::<Self>()
153156
+ match self {
154-
GroupOrdering::None => 0,
155-
GroupOrdering::Partial(partial) => partial.size(),
156-
GroupOrdering::Full(full) => full.size(),
157+
// These variants hold only inline state, already counted above.
158+
GroupOrdering::None | GroupOrdering::Full(_) => 0,
159+
GroupOrdering::Partial(partial) => partial.heap_size(),
157160
}
158161
}
159162
}
@@ -164,7 +167,42 @@ mod tests {
164167

165168
use std::sync::Arc;
166169

167-
use arrow::array::Int32Array;
170+
use arrow::array::{Int32Array, StringArray};
171+
use datafusion_common::ScalarValue;
172+
173+
#[test]
174+
fn test_size_inline_only() -> Result<()> {
175+
let expected = size_of::<GroupOrdering>();
176+
assert_eq!(GroupOrdering::None.size(), expected);
177+
let ordering = GroupOrdering::try_new(&InputOrderMode::Sorted)?;
178+
assert_eq!(ordering.size(), expected);
179+
Ok(())
180+
}
181+
182+
#[test]
183+
fn test_size_partial_retained_allocations() -> Result<()> {
184+
// The direct partial constructor retains spare order-index capacity.
185+
let mut order_indices = Vec::with_capacity(32);
186+
order_indices.push(0);
187+
let expected =
188+
size_of::<GroupOrdering>() + order_indices.capacity() * size_of::<usize>();
189+
let mut ordering =
190+
GroupOrdering::Partial(GroupOrderingPartial::try_new(order_indices)?);
191+
assert_eq!(ordering.size(), expected);
192+
193+
// A variable-width key checks both the scalar descriptor and its payload.
194+
let key = "retained sort key";
195+
let batch_group_values: Vec<ArrayRef> =
196+
vec![Arc::new(StringArray::from(vec!["a", key]))];
197+
ordering.new_groups(&batch_group_values, &[0, 1], 2)?;
198+
let in_progress = expected + ScalarValue::Utf8(Some(key.to_owned())).size();
199+
assert_eq!(ordering.size(), in_progress);
200+
201+
// Completing drops the key, but retains order-index capacity.
202+
ordering.input_done();
203+
assert_eq!(ordering.size(), expected);
204+
Ok(())
205+
}
168206

169207
#[test]
170208
fn test_oom_emit_to_none_ordering() {

‎datafusion/physical-plan/src/aggregates/order/partial.rs‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,6 @@
1616
// under the License.
1717

1818
use std::cmp::Ordering;
19-
use std::mem::size_of;
2019
use std::sync::Arc;
2120

2221
use arrow::array::ArrayRef;
@@ -102,7 +101,7 @@ enum State {
102101
}
103102

104103
impl State {
105-
fn size(&self) -> usize {
104+
fn heap_size(&self) -> usize {
106105
match self {
107106
State::Taken => 0,
108107
State::Start => 0,
@@ -267,9 +266,10 @@ impl GroupOrderingPartial {
267266
Ok(())
268267
}
269268

270-
/// Return the size of memory allocated by this structure
271-
pub(crate) fn size(&self) -> usize {
272-
size_of::<Self>() + self.order_indices.allocated_size() + self.state.size()
269+
/// Returns retained heap allocations, excluding the inline descriptor
270+
/// already counted by [`super::GroupOrdering::size`].
271+
pub(crate) fn heap_size(&self) -> usize {
272+
self.order_indices.allocated_size() + self.state.heap_size()
273273
}
274274
}
275275

0 commit comments

Comments
 (0)