Repository navigation
Commit 8f77b41
authored
fix: correct NaN grouping and eviction in TopK aggregation (apache#26445)
## Which issue does this PR close?
Closes apache#26444.
## Rationale for this change
Repeated identical NaN keys can form separate TopK groups, and evicting
a NaN group can panic. Valid `GROUP BY ... ORDER BY ... LIMIT` queries
can therefore return incorrect results or fail.
## What changes are included in this PR?
- Compare primitive grouping keys with Arrow equality so identical NaN
payloads match.
- Normalize signed zero with `HashValue::normalize_key`, consistent with
ordinary grouping while preserving distinct NaN payloads.
- Remove occupied entries by their known store index during eviction and
NULL aggregate transitions.
A separate prerequisite commit retains the RSS sampling fix from apache#26208.
It samples the current process before query execution so short queries
cannot leave the recorded maximum at zero. Memory thresholds are
unchanged.
## What is the testing strategy for this PR?
The SQL regressions cover repeated NaN keys and eviction. At main commit
`7f493a565`, the eviction regression fails with `internal error: entered
unreachable code`; it passes with this fix. Hash-table tests cover
Float16/32/64, distinct NaN payloads, signed zero, and NULL aggregate
transitions.
Validation was rerun after merging main at `350ca7650`. The extended
workspace suite, including the migrated `extended_tests` target, full
SQL logic suite, and `cargo test --profile ci -p datafusion-cli` passed.
`cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D
warnings`, and `UV_FROZEN=true uv run ./dev/rust_lint.sh` also passed.
The existing `topk_aggregate` primitive-key benchmark was run against
main at `c61fb09ac` using the unoptimized CI profile on a shared host.
Timing varied too much to draw a performance conclusion.
## Are there any user-facing changes?
Identical NaN keys form one TopK group, and evicting a NaN group
succeeds. Signed-zero grouping matches ordinary aggregation. There are
no public API changes.
AI-assisted contribution.1 parent 6341b5c commit 8f77b41
3 files changed
Lines changed: 137 additions & 27 deletions
File tree
- datafusion
- core/tests/memory_limit_validation
- physical-plan/src/aggregates/topk
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | 33 | | |
38 | 34 | | |
39 | | - | |
40 | | - | |
41 | | - | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
42 | 47 | | |
43 | 48 | | |
44 | 49 | | |
45 | 50 | | |
46 | 51 | | |
47 | 52 | | |
48 | | - | |
| 53 | + | |
49 | 54 | | |
50 | 55 | | |
51 | 56 | | |
52 | 57 | | |
53 | 58 | | |
54 | 59 | | |
55 | 60 | | |
56 | | - | |
| 61 | + | |
57 | 62 | | |
58 | 63 | | |
59 | 64 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
9 | 8 | | |
10 | 9 | | |
11 | | - | |
12 | | - | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| |||
119 | 119 | | |
120 | 120 | | |
121 | 121 | | |
122 | | - | |
123 | | - | |
124 | | - | |
125 | | - | |
| 122 | + | |
126 | 123 | | |
127 | 124 | | |
128 | 125 | | |
| |||
211 | 208 | | |
212 | 209 | | |
213 | 210 | | |
214 | | - | |
| 211 | + | |
| 212 | + | |
215 | 213 | | |
216 | 214 | | |
217 | 215 | | |
| |||
225 | 223 | | |
226 | 224 | | |
227 | 225 | | |
228 | | - | |
| 226 | + | |
229 | 227 | | |
230 | 228 | | |
231 | 229 | | |
232 | 230 | | |
233 | 231 | | |
234 | 232 | | |
235 | | - | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
236 | 238 | | |
237 | 239 | | |
238 | 240 | | |
239 | 241 | | |
240 | 242 | | |
241 | | - | |
| 243 | + | |
| 244 | + | |
242 | 245 | | |
243 | 246 | | |
244 | 247 | | |
| |||
293 | 296 | | |
294 | 297 | | |
295 | 298 | | |
296 | | - | |
| 299 | + | |
297 | 300 | | |
298 | 301 | | |
299 | 302 | | |
| |||
311 | 314 | | |
312 | 315 | | |
313 | 316 | | |
314 | | - | |
315 | | - | |
316 | | - | |
317 | | - | |
318 | | - | |
| 317 | + | |
| 318 | + | |
319 | 319 | | |
320 | 320 | | |
321 | 321 | | |
| |||
609 | 609 | | |
610 | 610 | | |
611 | 611 | | |
| 612 | + | |
612 | 613 | | |
613 | 614 | | |
614 | 615 | | |
615 | | - | |
| 616 | + | |
616 | 617 | | |
617 | 618 | | |
618 | 619 | | |
619 | 620 | | |
620 | 621 | | |
621 | 622 | | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
622 | 699 | | |
623 | 700 | | |
624 | 701 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
858 | 858 | | |
859 | 859 | | |
860 | 860 | | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
0 commit comments