Repository navigation
Commit 96e8fdf
authored
chore: fix
## Rationale for this change
Related to apache#23393
## Rationale for this change
`SlidingDistinctCountAccumulator::size()` returned a flat
`size_of_val(self)` (64 B) regardless of how many distinct values were
held. Since `MemoryPool` reservations trust this value, memory limits
were
silently bypassed for `COUNT(DISTINCT …)` window aggregates — real heap
could reach GBs while the pool reported ~64 B.
## What changes are included in this PR?
Fix `SlidingDistinctCountAccumulator::size()` in
`datafusion/functions-aggregate/src/count.rs` to mirror the sibling
`DistinctCountAccumulator::full_size` pattern:
```rust
size_of_val(self)
+ (size_of::<ScalarValue>() + size_of::<usize>()) *
self.counts.capacity()
+ self.counts.keys().map(|k| k.size() - size_of_val(k)).sum::<usize>()
+ self.data_type.size()
- size_of_val(&self.data_type)
```
Accounts for every persistent field of the struct: `self` bytes +
HashMap bucket capacity + per-key inner heap + `DataType` inner heap.
Uses `.capacity()` (not `.len()`) so post-`retract_batch` state still
reports reserved memory correctly.SlidingDistinctCountAccumulator::size() to include budget for distinct values (apache#23399)1 parent e271f65 commit 96e8fdf
4 files changed
Lines changed: 17 additions & 1 deletion
File tree
- datafusion
- expr-common/src
- functions-aggregate/src
- physical-plan/src/aggregates/group_values
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
92 | 92 | | |
93 | 93 | | |
94 | 94 | | |
| 95 | + | |
| 96 | + | |
95 | 97 | | |
96 | 98 | | |
97 | 99 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
248 | 248 | | |
249 | 249 | | |
250 | 250 | | |
| 251 | + | |
| 252 | + | |
251 | 253 | | |
252 | 254 | | |
253 | 255 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
555 | 555 | | |
556 | 556 | | |
557 | 557 | | |
| 558 | + | |
| 559 | + | |
558 | 560 | | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
559 | 569 | | |
560 | 570 | | |
561 | 571 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
102 | | - | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
103 | 105 | | |
104 | 106 | | |
105 | 107 | | |
| |||
0 commit comments