Repository navigation
Commit 741d221
## Which issue does this PR close?
- Part of #23393 on the `branch-55` release line.
- Backports the merged fix in #26154.
## Rationale for this change
In 55.x, a grouped `MIN`/`MAX` over strings or binary values reports
less memory than it holds. Its `size()` counts the current length of
each group's value rather than the capacity that the value's buffer
keeps. So when a long value is replaced by a shorter one, the buffer
keeps its old capacity but the reported size drops. Unused capacity in
the outer per-group vector is not counted either. Under a memory limit,
an aggregate can then hold more memory than the pool knows about, and it
spills later than it should, or not at all.
#26154 fixed this on `main`, but it is not in 55.1.0 or 55.2.0 and first
ships in 56.0.0. This backport is for a 55.3.0 patch release, if one is
made.
## What changes are included in this PR?
- A cherry-pick with `git cherry-pick -x` of the commit that #26154
merged (0d79252). This PR changes
nothing else.
- `MinMaxBytesState` tracks the retained capacity of each group's value
incrementally, keeping `size()` O(1). `size()` reports that capacity
plus the outer vector's capacity. The change is confined to
`datafusion/functions-aggregate/src/min_max/min_max_bytes.rs`.
- The cherry-pick conflicts only in the test module. On `main`, the file
already had a `tests` module containing
`preserving_selected_min_values`, which uses `GroupSelection` and
`evaluate_preserving`, and neither exists on `branch-55`. The resolution
adds the `tests` module with only the test from #26154,
`size_tracks_growth_and_emit_reuse`. The non-test changes apply as they
are on `main`.
## What is the testing strategy for this PR?
Covered by `size_tracks_growth_and_emit_reuse`, the test #26154 added.
On `branch-55`:
- `cargo fmt --all -- --check`
- `cargo clippy -p datafusion-functions-aggregate --all-targets -- -D
warnings`
- `cargo test -p datafusion-functions-aggregate`: 196 unit tests and 2
doc tests passed, including `size_tracks_growth_and_emit_reuse`
## Are there any user-facing changes?
No. Query results and public APIs are unchanged. Grouped byte
`MIN`/`MAX` now reports its memory more accurately, so under a memory
limit it may spill or fail to reserve memory earlier than before.
Co-authored-by: kosiew <[email protected]>
1 parent 6fceca2 commit 741d221
1 file changed
Lines changed: 88 additions & 11 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
383 | 383 | | |
384 | 384 | | |
385 | 385 | | |
386 | | - | |
387 | | - | |
| 386 | + | |
388 | 387 | | |
| 388 | + | |
| 389 | + | |
389 | 390 | | |
390 | 391 | | |
391 | 392 | | |
| |||
400 | 401 | | |
401 | 402 | | |
402 | 403 | | |
| 404 | + | |
403 | 405 | | |
404 | 406 | | |
405 | 407 | | |
406 | 408 | | |
407 | 409 | | |
408 | 410 | | |
409 | 411 | | |
410 | | - | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
411 | 415 | | |
412 | 416 | | |
413 | 417 | | |
414 | 418 | | |
415 | 419 | | |
416 | 420 | | |
| 421 | + | |
417 | 422 | | |
418 | 423 | | |
| 424 | + | |
419 | 425 | | |
420 | 426 | | |
421 | 427 | | |
| |||
476 | 482 | | |
477 | 483 | | |
478 | 484 | | |
479 | | - | |
| 485 | + | |
480 | 486 | | |
481 | | - | |
| 487 | + | |
482 | 488 | | |
483 | 489 | | |
484 | 490 | | |
485 | 491 | | |
| 492 | + | |
486 | 493 | | |
487 | 494 | | |
488 | 495 | | |
489 | 496 | | |
490 | 497 | | |
491 | 498 | | |
492 | 499 | | |
493 | | - | |
| 500 | + | |
494 | 501 | | |
495 | | - | |
496 | | - | |
497 | | - | |
498 | | - | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
499 | 509 | | |
500 | 510 | | |
501 | 511 | | |
502 | 512 | | |
503 | 513 | | |
504 | | - | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
505 | 582 | | |
506 | 583 | | |
0 commit comments