Repository navigation
Commit 36d81dd
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes apache#123` indicates that this PR will close issue apache#123.
-->
- Part of apache#23393 .
## Rationale for this change
`SlidingMinAccumulator::size` and `SlidingMaxAccumulator::size` only
reported
the stack size of their `ScalarValue` field plus its heap payload,
ignoring the
memory held by the underlying `MovingMin` / `MovingMax` sliding-window
buffers.
For windowed `MIN`/`MAX` over string or list data, the two per-element
stacks
can hold megabytes of `ScalarValue` payload that the memory pool was
never
told about, understating accumulator memory usage.
## What changes are included in this PR?
- Add a private `heap_size(elem_heap)` method to `MovingMin<T>` and
`MovingMax<T>`
that reports the two stack buffers' capacity in bytes plus each stored
element's heap payload.
- Factor the shared implementation into a `moving_stacks_heap_size` free
helper
so the two types stay in sync.
- Include the buffer bytes in `SlidingMinAccumulator::size` and
`SlidingMaxAccumulator::size` via the new method.
## Are these changes tested?
Yes. Two new unit tests in
`datafusion/functions-aggregate/src/min_max.rs`:
- `moving_min_max_heap_size_i32` — fixed-width `T`, verifies buffer-only
accounting with and without pushed elements.
- `moving_min_max_heap_size_counts_elems` — `T = String`, verifies each
of the
two slots in a `(T, T)` pair contributes independently to the heap
payload
(mirroring the two independent `Clone`s made by `push`).
1 parent a34cc22 commit 36d81dd
1 file changed
Lines changed: 80 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
55 | | - | |
| 55 | + | |
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
| |||
433 | 433 | | |
434 | 434 | | |
435 | 435 | | |
436 | | - | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
437 | 439 | | |
438 | 440 | | |
439 | 441 | | |
| |||
721 | 723 | | |
722 | 724 | | |
723 | 725 | | |
724 | | - | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
725 | 729 | | |
726 | 730 | | |
727 | 731 | | |
| |||
857 | 861 | | |
858 | 862 | | |
859 | 863 | | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
860 | 888 | | |
861 | 889 | | |
862 | 890 | | |
| |||
975 | 1003 | | |
976 | 1004 | | |
977 | 1005 | | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
978 | 1013 | | |
979 | 1014 | | |
980 | 1015 | | |
| |||
1213 | 1248 | | |
1214 | 1249 | | |
1215 | 1250 | | |
| 1251 | + | |
| 1252 | + | |
| 1253 | + | |
| 1254 | + | |
| 1255 | + | |
| 1256 | + | |
| 1257 | + | |
| 1258 | + | |
| 1259 | + | |
| 1260 | + | |
| 1261 | + | |
| 1262 | + | |
| 1263 | + | |
| 1264 | + | |
| 1265 | + | |
| 1266 | + | |
| 1267 | + | |
| 1268 | + | |
| 1269 | + | |
| 1270 | + | |
| 1271 | + | |
| 1272 | + | |
| 1273 | + | |
| 1274 | + | |
| 1275 | + | |
| 1276 | + | |
| 1277 | + | |
| 1278 | + | |
| 1279 | + | |
| 1280 | + | |
| 1281 | + | |
| 1282 | + | |
| 1283 | + | |
| 1284 | + | |
| 1285 | + | |
| 1286 | + | |
| 1287 | + | |
| 1288 | + | |
| 1289 | + | |
| 1290 | + | |
| 1291 | + | |
| 1292 | + | |
1216 | 1293 | | |
1217 | 1294 | | |
1218 | 1295 | | |
| |||
0 commit comments