You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
TopK aggregation duplicates NaN groups and can panic during eviction #26444
With TopK aggregation enabled, repeated identical NaN grouping keys can produce separate groups. Evicting a NaN group can also panic with internal error: entered unreachable code in aggregates/topk/hash_table.rs.
To Reproduce
Reproduced on main at 5f2158269e24ac45e2740dcfe6e79a6f69f28330:
SETdatafusion.execution.target_partitions =1;
SETdatafusion.optimizer.enable_topk_aggregation = true;
SELECT k, MAX(v) FROM (VALUES
(CAST('NaN'AS DOUBLE), 1),
(CAST('NaN'AS DOUBLE), 2),
(CAST(0AS DOUBLE), 3)
) AS t(k, v)
GROUP BY k ORDER BYMAX(v) DESC NULLS LAST LIMIT3;
This returns an extra group:
0 3
NaN 2
NaN 1
Changing LIMIT 3 to LIMIT 1 triggers the eviction panic. Disabling datafusion.optimizer.enable_topk_aggregation gives the expected results.
Expected behavior
The query with LIMIT 3 returns 0, 3 and NaN, 2. With LIMIT 1, it returns only 0, 3, without a panic.
Additional context
#16256 fixed analogous NaN key equality in ordinary aggregation. This report concerns TopK's key lookup and eviction paths. It uses repeated NaNs with identical bits and does not require the broader all-NaN canonicalization proposed in #26091.
Willingness to contribute
I plan to contribute a fix for this bug independently. A draft PR with the fix and regression coverage will follow.
Activity
fix: correct NaN grouping and eviction in TopK aggregation (apache#26445