Skip to content

Commit 5009f42

Browse files
authored
Account for TopK common-prefix converter memory (#26286)
## Which issue does this PR close? - Part of #23393 ## Rationale for this change `TopK` underreports retained memory when a common sort prefix is present because its memory accounting omits the optional `common_sort_prefix_converter`. This can cause memory reservations to underestimate TopK's retained memory, weakening memory-limit enforcement. ## What changes are included in this PR? - Include `common_sort_prefix_converter.size()` exactly once in `TopK::size()` when the converter is present. - Preserve the existing size calculation when the converter is absent. - Add a regression test covering both construction paths and checking that the memory reservation matches the reported size after batch insertion. ## Are these changes tested? Added `test_topk_size_common_sort_prefix_converter`, which verifies: - A common-prefix converter is created and reports a nonzero size. - Without a common prefix, `TopK::size()` retains its existing formula. - With a common prefix, the reported size increases by the converter's size. - After `insert_batch()`, the memory reservation matches `TopK::size()`. The test does not rely on global allocator measurements. ## Are there any user-facing changes? No changes to query results or public APIs. The change corrects internal memory accounting for TopK with a common sort prefix, potentially improving memory-limit enforcement. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.
1 parent cdc2fc0 commit 5009f42

1 file changed

Lines changed: 45 additions & 0 deletions

File tree

  • datafusion/physical-plan/src/topk

‎datafusion/physical-plan/src/topk/mod.rs‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -882,6 +882,10 @@ impl TopK {
882882
fn size(&self) -> usize {
883883
size_of::<Self>()
884884
+ self.row_converter.size()
885+
+ self
886+
.common_sort_prefix_converter
887+
.as_ref()
888+
.map_or(0, RowConverter::size)
885889
+ self.scratch_rows.size()
886890
+ self.heap.size()
887891
}
@@ -3306,6 +3310,47 @@ mod tests {
33063310
)
33073311
}
33083312

3313+
#[test]
3314+
fn test_topk_size_common_sort_prefix_converter() -> Result<()> {
3315+
let schema = make_ab_schema();
3316+
let sort_expr = PhysicalSortExpr {
3317+
expr: col("a", schema.as_ref())?,
3318+
options: SortOptions::default(),
3319+
};
3320+
let make_topk = |common_sort_prefix| {
3321+
TopK::try_new(
3322+
0,
3323+
Arc::clone(&schema),
3324+
common_sort_prefix,
3325+
LexOrdering::from([sort_expr.clone()]),
3326+
3,
3327+
2,
3328+
Arc::new(RuntimeEnv::default()),
3329+
&ExecutionPlanMetricsSet::new(),
3330+
make_topk_filter(),
3331+
)
3332+
};
3333+
let without_prefix = make_topk(vec![])?;
3334+
let with_prefix = make_topk(vec![sort_expr.clone()])?;
3335+
let converter_size = with_prefix
3336+
.common_sort_prefix_converter
3337+
.as_ref()
3338+
.expect("common sort prefix converter")
3339+
.size();
3340+
assert!(converter_size > 0);
3341+
assert!(without_prefix.common_sort_prefix_converter.is_none());
3342+
assert_eq!(
3343+
without_prefix.size(),
3344+
size_of::<TopK>()
3345+
+ without_prefix.row_converter.size()
3346+
+ without_prefix.scratch_rows.size()
3347+
+ without_prefix.heap.size()
3348+
);
3349+
assert_eq!(with_prefix.size(), without_prefix.size() + converter_size);
3350+
3351+
Ok(())
3352+
}
3353+
33093354
/// Reads the `output_batches` and `output_rows` metrics recorded by an emit.
33103355
fn output_batches_and_rows(metrics: &ExecutionPlanMetricsSet) -> (usize, usize) {
33113356
let metrics = metrics.clone_inner();

0 commit comments

Comments
 (0)