You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
MinMaxStructState::emit_to(EmitTo::First(n)) subtracts row counts from a byte counter #26891
In datafusion-functions-aggregate/src/min_max/min_max_struct.rs, set_value adds get_array_memory_size() bytes to total_data_bytes for each stored struct. But emit_to(EmitTo::First(n)) subtracts s.len() for each emitted value, which is 1 for a one-row StructArray. So size() keeps counting memory for groups that were already emitted. It only resets on EmitTo::All.
Ordered aggregation emits First(n) after every batch, so the reported size grows with the number of groups seen. Reproduction: a Partial AggregateExec feeding a Final one over sorted input, with 4,096 groups of 4 KiB struct values, 64-row batches, a 2 MiB GreedyMemoryPool and the disk manager disabled. It fails with ResourcesExhausted in OrderedFinalAggregateStream. Subtracting get_array_memory_size() of the emitted values instead makes it complete.
The code looks copied from min_max_bytes, where len() is a byte count.
I'll take this one. The fix is in #27115: it now subtracts the actual byte size of the emitted values, and adds a sqllogictest that fails with Resources exhausted before the fix. Reviews welcome when you have a moment (・ω・)ノ
Activity
dragonknightk commentedon Nov 5, 2026
I'll take this one. The fix is in #27115: it now subtracts the actual byte size of the emitted values, and adds a sqllogictest that fails with Resources exhausted before the fix. Reviews welcome when you have a moment (・ω・)ノ
dragonknightk commentedon Nov 10, 2026
take