Skip to content

MinMaxStructState::emit_to(EmitTo::First(n)) subtracts row counts from a byte counter #26891

Description

@parthchandra

Describe the bug

Filed as a follow up to review comments in apache/datafusion-comet#7191 (comment).

In datafusion-functions-aggregate/src/min_max/min_max_struct.rs, set_value adds get_array_memory_size() bytes to total_data_bytes for each stored struct. But emit_to(EmitTo::First(n)) subtracts s.len() for each emitted value, which is 1 for a one-row StructArray. So size() keeps counting memory for groups that were already emitted. It only resets on EmitTo::All.

Ordered aggregation emits First(n) after every batch, so the reported size grows with the number of groups seen. Reproduction: a Partial AggregateExec feeding a Final one over sorted input, with 4,096 groups of 4 KiB struct values, 64-row batches, a 2 MiB GreedyMemoryPool and the disk manager disabled. It fails with ResourcesExhausted in OrderedFinalAggregateStream. Subtracting get_array_memory_size() of the emitted values instead makes it complete.

The code looks copied from min_max_bytes, where len() is a byte count.

To Reproduce

No response

Expected behavior

No response

Additional context

No response

Willingness to contribute

None

Activity

  1. added theissue type on Oct 29, 2026
  2. dragonknightk commented on Nov 5, 2026

    @dragonknightk

    I'll take this one. The fix is in #27115: it now subtracts the actual byte size of the emitted values, and adds a sqllogictest that fails with Resources exhausted before the fix. Reviews welcome when you have a moment (・ω・)ノ

  3. dragonknightk commented on Nov 10, 2026

    @dragonknightk

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions