Repository navigation
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #26208 +/- ##
==========================================
- Coverage 82.80% 82.80% -0.01%
==========================================
Files 1149 1149
Lines 452205 452205
Branches 452205 452205
==========================================
- Hits 374458 374448 -10
- Misses 54950 54958 +8
- Partials 22797 22799 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
kosiew
pushed a commit
to kosiew/datafusion
that referenced
this pull request
Oct 28, 2026
) ## Which issue does this PR close? Closes apache#26444. ## Rationale for this change Repeated identical NaN keys can form separate TopK groups, and evicting a NaN group can panic. Valid `GROUP BY ... ORDER BY ... LIMIT` queries can therefore return incorrect results or fail. ## What changes are included in this PR? - Compare primitive grouping keys with Arrow equality so identical NaN payloads match. - Normalize signed zero with `HashValue::normalize_key`, consistent with ordinary grouping while preserving distinct NaN payloads. - Remove occupied entries by their known store index during eviction and NULL aggregate transitions. A separate prerequisite commit retains the RSS sampling fix from apache#26208. It samples the current process before query execution so short queries cannot leave the recorded maximum at zero. Memory thresholds are unchanged. ## What is the testing strategy for this PR? The SQL regressions cover repeated NaN keys and eviction. At main commit `7f493a565`, the eviction regression fails with `internal error: entered unreachable code`; it passes with this fix. Hash-table tests cover Float16/32/64, distinct NaN payloads, signed zero, and NULL aggregate transitions. Validation was rerun after merging main at `350ca7650`. The extended workspace suite, including the migrated `extended_tests` target, full SQL logic suite, and `cargo test --profile ci -p datafusion-cli` passed. `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, and `UV_FROZEN=true uv run ./dev/rust_lint.sh` also passed. The existing `topk_aggregate` primitive-key benchmark was run against main at `c61fb09ac` using the unoptimized CI profile on a shared host. Timing varied too much to draw a performance conclusion. ## Are there any user-facing changes? Identical NaN keys form one TopK group, and evicting a NaN group succeeds. Signed-zero grouping matches ordinary aggregation. There are no public API changes. AI-assisted contribution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Local reproduction of a zero-RSS memory-validation failure. Depends on the window-expression lint prerequisite: #26207.
Rationale for this change
A short query can finish before the asynchronous RSS monitor starts, leaving max_rss at zero and failing the memory-limit validation. System::new_all also gathers information unrelated to the current process.
What changes are included in this PR?
Capture current-process memory before running the query, initialize the maximum from that sample, and reuse the same System in the monitor. Both refreshes exclude task enumeration. Memory thresholds are unchanged.
What is the testing strategy for this PR?
The previously failing sort_with_mem_limit_1_runner now passes. The existing memory_limit_validation_runner_works_runner still rejects its impossible 20 MB cap. The core integration suite passed 1,227 tests with one ignored. The extended workspace run is in progress.
Are there any user-facing changes?
None; this changes test infrastructure.
AI-assisted contribution authored with Codex.