Repository navigation
Commit d80d213
perf: reuse GroupsAccumulatorAdapter scratch buffers across batches (apache#25605)
## Which issue does this PR close?
- Closes apache#25604.
## Rationale for this change
`GroupsAccumulatorAdapter` runs every aggregate that has no
`GroupsAccumulator` of its own. For each input batch,
`invoke_per_accumulator` allocated three new buffers,
`groups_with_rows`, `offsets` and `batch_indices`, and dropped them at
the end of the batch. @alamb suggested to reuse them in
apache#25123 (comment).
## What changes are included in this PR?
1. **Reuse `groups_with_rows` and `offsets`.** A new `Scratch` struct on
the adapter holds the buffers. Each batch clears them and fills them
again, so they allocate only when a batch needs more capacity than an
earlier batch. `invoke_per_accumulator` takes the buffers out of `self`
for the batch and puts them back on every return path, errors included.
The body moves unchanged into `invoke_per_accumulator_with_scratch`.
2. **Reuse `batch_indices`.** The adapter moves the buffer into the
`UInt32Array` for the `take` kernels without a copy. The kernels only
borrow the array, thus the adapter then gets the buffer back without a
copy through `into_parts`.
3. **Shrink oversized buffers.** A buffer keeps the capacity of the
largest batch that it has seen. After each batch, the adapter shrinks a
buffer whose capacity is more than 64 Ki entries and more than 4 times
what the batch used. It shrinks to the smaller of the two, so it never
keeps more than either limit allows. Thus one unusually large batch does
not make the adapter keep that memory for all the batches after it.
The adapter charges the change in the capacity of the buffers to
`allocation_bytes`, thus `GroupsAccumulator::size` includes them and the
memory pool sees them. It releases them when an emit leaves no group,
thus an adapter that has emitted all groups still reports 0 bytes.
Each adapter belongs to one partition's aggregate stream, and
`GroupsAccumulator` methods take `&mut self`, so the buffers are never
shared between partitions.
I have not measured a speed change. This removes three allocations for
each batch, while the same batch calls `Accumulator::update_batch` once
for each group that it touches. Thus I expect the change to be small.
## What is the testing strategy for this PR?
- `adapter_charges_retained_indices_once_and_releases_them`,
`adapter_clears_successful_group_indices_after_later_error` and
`adapter_reconciles_allocation_after_later_error_with_grouped_metric`
check the exact memory charge, which now includes the scratch buffers.
The first one also checks that the adapter keeps some scratch capacity
after a batch, that `batch_indices` comes back from the `take` kernels,
and that `size()` is 0 after `evaluate(EmitTo::All)`.
- `adapter_releases_oversized_scratch` sends a batch with 128 Ki groups
and then a batch with one row. It checks that the second batch shrinks
each large buffer to exactly 4 times what it used, and that `size()`
goes down by the released amount.
To make sure that these tests catch an error, I removed each part in
turn:
- Without the charge for the scratch buffers, the three accounting tests
fail.
- Without the release on emit,
`adapter_charges_retained_indices_once_and_releases_them` fails.
- Without the shrink of oversized buffers, without it for
`batch_indices` alone, with a release to capacity 0 instead of the
shrink, or with a shrink to 64 Ki entries instead of the smaller limit,
`adapter_releases_oversized_scratch` fails.
The unit tests of `datafusion-functions-aggregate-common`, the
`memory_limit` tests in `core_integration` and the sqllogictest suite
pass.
## Are there any user-facing changes?
No. There are no changes to any public API or to any result.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 <[email protected]>1 parent 9b469be commit d80d213
1 file changed
Lines changed: 177 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
111 | 111 | | |
112 | 112 | | |
113 | 113 | | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
114 | 171 | | |
115 | 172 | | |
116 | 173 | | |
| |||
154 | 211 | | |
155 | 212 | | |
156 | 213 | | |
| 214 | + | |
157 | 215 | | |
158 | 216 | | |
159 | 217 | | |
| |||
235 | 293 | | |
236 | 294 | | |
237 | 295 | | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
238 | 331 | | |
239 | 332 | | |
240 | 333 | | |
241 | | - | |
| 334 | + | |
242 | 335 | | |
243 | 336 | | |
244 | 337 | | |
| |||
263 | 356 | | |
264 | 357 | | |
265 | 358 | | |
266 | | - | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
267 | 367 | | |
268 | 368 | | |
269 | 369 | | |
270 | | - | |
| 370 | + | |
271 | 371 | | |
272 | 372 | | |
273 | 373 | | |
| |||
277 | 377 | | |
278 | 378 | | |
279 | 379 | | |
280 | | - | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
281 | 383 | | |
282 | 384 | | |
283 | 385 | | |
284 | 386 | | |
285 | | - | |
286 | | - | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
287 | 394 | | |
288 | 395 | | |
289 | 396 | | |
| |||
398 | 505 | | |
399 | 506 | | |
400 | 507 | | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
401 | 517 | | |
402 | 518 | | |
403 | 519 | | |
| |||
456 | 572 | | |
457 | 573 | | |
458 | 574 | | |
| 575 | + | |
459 | 576 | | |
460 | 577 | | |
461 | 578 | | |
| |||
520 | 637 | | |
521 | 638 | | |
522 | 639 | | |
| 640 | + | |
523 | 641 | | |
524 | 642 | | |
525 | 643 | | |
| |||
820 | 938 | | |
821 | 939 | | |
822 | 940 | | |
823 | | - | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
824 | 949 | | |
825 | 950 | | |
826 | 951 | | |
| |||
1085 | 1210 | | |
1086 | 1211 | | |
1087 | 1212 | | |
| 1213 | + | |
1088 | 1214 | | |
1089 | 1215 | | |
1090 | 1216 | | |
| |||
1129 | 1255 | | |
1130 | 1256 | | |
1131 | 1257 | | |
| 1258 | + | |
1132 | 1259 | | |
1133 | 1260 | | |
1134 | 1261 | | |
| |||
1158 | 1285 | | |
1159 | 1286 | | |
1160 | 1287 | | |
| 1288 | + | |
| 1289 | + | |
| 1290 | + | |
| 1291 | + | |
| 1292 | + | |
| 1293 | + | |
| 1294 | + | |
| 1295 | + | |
| 1296 | + | |
| 1297 | + | |
| 1298 | + | |
| 1299 | + | |
| 1300 | + | |
| 1301 | + | |
| 1302 | + | |
| 1303 | + | |
| 1304 | + | |
| 1305 | + | |
| 1306 | + | |
| 1307 | + | |
| 1308 | + | |
| 1309 | + | |
| 1310 | + | |
| 1311 | + | |
| 1312 | + | |
| 1313 | + | |
| 1314 | + | |
| 1315 | + | |
| 1316 | + | |
| 1317 | + | |
| 1318 | + | |
| 1319 | + | |
| 1320 | + | |
| 1321 | + | |
| 1322 | + | |
| 1323 | + | |
| 1324 | + | |
| 1325 | + | |
| 1326 | + | |
| 1327 | + | |
| 1328 | + | |
| 1329 | + | |
| 1330 | + | |
1161 | 1331 | | |
1162 | 1332 | | |
1163 | 1333 | | |
| |||
0 commit comments