Repository navigation
[EPIC] A Collection of Wrong Results Bugs - Q4 2026 #26672
Description
This ticket lists all the open wrong results results bugs; Please add any new issues you find in comments.
Specifically where DataFusion produces a result but that result is incorrect.
Rationale / Call To Action
As a foundational piece of technology, it is important that users can count on DataFusion's results to be correct. For the most part I think we do a very good job and we have many tests
As part of my ongoing quest to figure out how to prioritize all the proposals / PRs to DataFusion, I think correctness bugs are quite high on the list so I wish to track them here.
I have also been needled several times in discussions with various professors from TUM (who shall remain nameless) that they don't think DataFusion takes correctness seriously. While I disagree with this sentiment, I do think it is worth time focusing on correctness.
The List ™️
Correlated subqueries and decorrelation
- Incorrect results for NOT IN subqueries with nulls #8241
- Returns the wrong grouped row when an outer
GROUP BYconsumes a subquery containingORDER BY ... OFFSET. #23534 - Wrong results: correlated
EXISTS/INsubqueries over a groupless aggregate get the wrong outer rows #24960 - Wrong results: correlated
NOT IN (subquery)ignores NULLs in the subquery #25347 - Wrong results: correlated
NOT INloses the correlation when it is an equality on theINvalue column #25480 - Wrong results: LATERAL with SELECT DISTINCT and a non-equality correlated filter returns duplicate rows #25808
- Wrong results: correlated aggregate subquery with a lossy
CASTin its filter returns one row per inner value #26014 - Wrong results: a window function in an
INsubquery sees every row when the correlated filter repeats theINpredicate #26032 - Wrong results: Correlated subquery confuses same-named columns of two of its relations #26234
- Unsupported: correlated scalar subquery with nested aggregates #26508
- REGRESSION: Multi-key null-aware join silently defaults to one NOT IN value key #26569
Sort order and equivalence / ordering properties
- Sort options for DataType::Dictionary is incorrect #4649
- Incorrect join ordering properties can cause required sorts to be removed #25052
- Incorrect ORDER BY results with mismatched Parquet timestamp units #25817
- Wrong ORDER BY results after an outer join when the nullable side equates a column to
coalesce/CASE#26036 - Incorrect results: ORDER BY b, a over a subquery with ORDER BY b LIMIT n is not sorted by (b, a) when SortPreservingMergeExec has a fetch #26162
- Incorrect ORDER BY results after sort pushdown through custom execution plans #26656
TopK, LIMIT and dynamic filters
- TopK dynamic filter returns wrong rows when one sort key contains another (DynamicFilterPhysicalExpr remaps the inner column first) #26029
- Recursive CTE: MIN/MAX aggregate dynamic filter survives reset_state, later iterations return wrong results #26054
- enable_window_topn returns wrong results when a projection under the filter drops a column (wrong ranking function chosen) #26416
- Wrong results: UNION ALL ... ORDER BY ... LIMIT when a sort key is constant per arm (partitioned TopK shares a threshold over a shortened key) #26454
- Wrong result:
count(*) OVER ()sees only the LIMIT rows when ORDER BY ... LIMIT sits on top of another window (target_partitions > 1) #26617 - Wrong results:
ORDER BY <double>, ... LIMIT ndrops-0.0rows when 0.0 arrives in an earlier batch (TopK dynamic filter) #26618 - Wrong results: a filter like
k / 10 = 5makes ORDER BY and GROUP BY treatkas constant #26779
Logical optimizer rewrites (filter pushdown, unwrap cast, volatile expressions)
- Incorrect result when use volatile function alias in having clause #7976
- Unsafe / incorrect
unwrap_castoptimizations for narrowing casts (e.g. Decimal) #23095 - Narrowing
try_castis incorrectly unwrapped from comparisons, dropping NULL-on-overflow semantics #24702 - BETWEEN evaluates its operand two times, so a volatile operand returns wrong rows #25457
- Wrong results: COALESCE on a volatile operand evaluates it two times, and fails when the output is declared non-nullable #25477
- Wrong results: eliminate_cross_join drops an equi-join key that spans two cross-joined relations #26058
- Wrong results: with
pushdown_filters = true, a filter on a volatile subquery column is evaluated again in the Parquet scan #26267 - Wrong results: a filter on a constant column keeps the row of an aggregate without GROUP BY #26545
-
push_down_filtersubstitutes the wrong expression whencolandCAST(col AS ...)share a schema name (HAVING and correlated IN give type errors) #26619 - Projection schema can go stale after SimplifyExpressions; OptimizeProjections silently normalizes it #24284
Aggregation, GROUP BY, constraints and set operations
- Group by duplicate GROUPING SETS returns wrong result #5672
- Some aggregates silently ignore
IGNORE NULLSandORDER BYon arguments #9924 - INTERSECT ALL returns wrong number of records from RHS #12955
- EXCEPT ALL returns the wrong number of records #12956
- Incorrect results of aggregation with grouping sets with single target partition #16965
- Incorrect query results for GROUP BY with UNIQUE constraint #21507
- Incorrect results with unique constraints #23634
- Incorrect results: GROUP BY column dropped based on a nullable UNIQUE constraint #23819
- Incorrect results: GROUP BY returns two rows for the NULL group with a nullable UNIQUE constraint #23820
- Wrong results: EmitTo::First(n) collision-list compaction in GroupValuesColumn can overwrite a not-yet-visited entry's list #25233
-
nth_valueandapprox_percentile_contsilently ignoreDISTINCT#26497 - sum() over DECIMAL silently wraps on overflow instead of erroring #26648
Joins
- NestedLoopJoinExec memory-limited fallback returns an empty result (wrong count = 0) instead of spilling or erroring #25701
- Sort-merge join ignores a join filter that references no columns (wrong FULL JOIN results) #26002
Parquet pruning, statistics and file reading
- Incorrect Schema Adaption for CSV #4918
- Parquet Statistics Pruning Ignores ColumnOrder, resulting in potentially incorrect statistics #8342
- Incorrect results: Bloom filters on
UInt8,Int8,UInt16andInt16columns always return false negatives #9779 - Incorrect statistics read for struct array in parquet #10609
- Pruning of floating point Parquet columns is incorrect when
NaNis present #15812 - Wrong results: filter marked as pushed down at plan time can be silently dropped per-file when schema evolution makes the runtime pushdown checker reject the adapted predicate #25268
- Incorrect Parquet pruning for timestamp-to-time casts across midnight #26451
- Plain
SELECTof a Parquet float column returns wrong values: NaN rows are replaced by the statistics constant when min == max #26490 - Parquet bloom filter pruning drops rows holding the other signed zero:
WHERE c = 0.0misses-0.0rows (and vice versa) #26491
Type coercion, casts and scalar function semantics
- [dbolympics] Comparing a
Durationto anIntervalwith a days field returns wrong results (e.g.ts - ts <= INTERVAL '3 days') #26671 - Inconsistent and incorrect
structfield coercion #14396 -
date_partreturning wrong results due to overflows #14738 - Incorrect type coercion for IS SIMILAR TO #20342
- concat_ws on array arguments silently stringifies the arrays #21703
-
CASEexpressions drop field metadata, silently stripping Arrow extension types and breaking metadata-aware UDFs #24860 - Recursive CTE silently casts recursive term to the static term's type, producing wrong results or a non-terminating query #25893
- Comparing a timestamp cast to a time zone against a literal returns wrong results #25941
- Spark
negativereturns wrong results for decimal columns datafusion-spark#33 - Struct union type coercion takes field nullability from the wrong field when field order differs #26296
-
EXTRACT EPOCHinconsistently extracts fractional seconds #26673 -
EXTRACT(NANOSECOND)onDurationandIntervalcounts whole seconds twice #26678 -
date_partdeclares a non-nullable result for non-null input, but can return NULL #26685
Writing files (round trip)
-
COPY … PARTITIONED BYwrites partition values containing=unescaped (p0=x=y); DuckDB and Polars cannot read the dataset or silently drop the partition column #26492 - CSV
ESCAPEoption is used when reading but ignored when writing:INSERT/COPYfiles cannot be read back with the same options (rows merged, backslashes dropped) #26514
SQL planning
Chore
Already Fixed in 56.0.0
- Scalar subquery returning no rows is incorrectly marked non-nullable #24513
- When performing set comparison subqueries (such as > ANY / > ALL / = ANY, etc.) on the same table (without using aliases), silently returns wrong results (0 rows or all rows). #25258
- Correlated
EXISTSsubquery withOFFSETreturns wrong results #25283 - Wrong results: correlated NOT IN with a non-equality correlation returns no rows (null-aware LeftAnti join ignores the residual filter) #25336
- Wrong results: constant NOT IN (subquery) in a WHERE clause ignores NULLs in the subquery #25340
- Wrong results: nullable constant
NOT IN (subquery)returns every row when the subquery column isNOT NULL#25473 - Wrong results:
NOT IN (subquery)ignores NULLs from a nullable outer expression overNOT NULLcolumns #25474 - Wrong results: a correlated filter under the nullable side of an outer join is pulled above the join #25507
- Wrong results: a correlated filter under an aggregate with a grouping set is pulled above the aggregate #25519
- Wrong results: a correlated filter under a window function is pulled above the window #25792
- Wrong results: a correlated filter under the right side of an ASOF JOIN is pulled above the join #25834
- Integer interval inference can incorrectly remove ORDER BY #25232
-
ORDER BY date_trunc(...)can return incorrectly ordered results for named timezones #25353 -
ORDER BY CAST(ts AS TIME)can return incorrectly ordered results across midnight #25506 - Parquet scan claims equivalences from a pruning-only predicate, giving wrong ORDER BY and LIMIT results #25779
- Wrong results:
SortPreservingMergeExeckeeps orderings of its input that do not hold after the merge #26646 -
LimitPushdowncan return incorrect rows for fetchedSortPreservingMergeExecover ordered Parquet scans #24272 - Wrong results: TopK dynamic filter + pushdown_filters re-reads already-delivered row groups (duplicate rows displace the true top-k) #24352
- EnsureRequirements can silently drop fetch during distribution reoptimization #24807
-
enable_join_dynamic_filter_pushdownsilently drops rows: the pushed filter is remapped by column NAME and lands on the wrong same-named column #25244 - Join dynamic filter pushdown through
ProjectionExeccan map duplicate output aliases to the wrong expression #25263 - Filter pushdown picks the wrong column when a join output has duplicated column names, causing wrong results with TopK dynamic filters #25296
- ORDER BY ... LIMIT returns wrong rows for List/Struct sort keys with NULL elements #25998
-
CoalescePartitionsExec::cardinality_effectreturnsEqualwhenfetchis set #26050 -
LEAD/LAG IGNORE NULLSreturns incorrect results acrossNULLgaps and withLIMIT#25471 - Wrong results: window aggregates over
RANGE ... N PRECEDINGdepend on batch boundaries when the ORDER BY value is NULL #25859 - Window referencing a named window with an added frame, OVER (w ROWS ...), silently drops PARTITION BY and ORDER BY #26364
- FilterPushdown physical optimizer incorrectly remaps filters through ProjectionExec with duplicate column names #21246
- Expression simplifications may produce incorrect results with NULLs #24246
- Incorrect results: leaf expression pushdown duplicates volatile expressions (random() evaluated twice) #24678
- unions_to_filter drops computed projections and returns wrong results #25071
- power and log simplifications return wrong results for base 1, 0 or negative #25137
- Wrong results: leaf expression pushdown removes a computed column that has the same name as its input column #25414
- Wrong results from
INwhen a list item is aCASEthat reads a column #26082 - Handling of Aggregates with multiple groupings sets is incorrect #17910
- Incorrect results: ORDER BY drops a sort key based on a nullable UNIQUE constraint #23818
- Streaming aggregation produces wrong result with grouping sets #24421
- percentile_cont is wrong for
Decimal32with lower interpolation precision #24576 - A fixed GROUP BY column is incorrectly used to decide when groups are complete #25018
- COUNT(... ORDER BY ...) can panic or undercount with nullable ordering keys #25055
- Wrong results: a list
unnestkeeps its input's GROUP BY key, so a later DISTINCT or GROUP BY drops rows #25844 -
corr,covar_pop,covar_sampandregr_*silently ignoreDISTINCT#26471 - Mark joins don't support null mark columns #21309
-
PiecewiseMergeJoinchanges state before draining its output coalescer: the Left/Full unmatched pass repeats forever, queued Right/Full rows can be dropped #24297 - PiecewiseMergeJoin drops rows when a stream batch spans multiple output chunks (small batch_size) #24348
- Memory-limited NestedLoopJoin silently drops unmatched rows when it spills #24745
- PiecewiseMergeJoin returns wrong results for List keys with NULL elements and fails on Struct keys #25957
- Incorrect filtering results when
collect_statistics = trueand column statistics are partially disabled #18241 - incorrect results when using NOT physical expression in
physical_expr::analyze#19264 - Parquet filter pushdown silently drops a
get_fieldpredicate when the file needs schema adaptation (wrong results) #24109 - Wrong results: dynamic row-group pruning drops row groups without slicing the carried RowSelection #24355
- Parquet scan silently drops one of two columns with the same name #24381
- CAST statistics can produce incorrect MIN/MAX results #25221
- Missing Parquet row-group statistics can produce incorrect file MIN/MAX #25226
- Parquet runtime pruning can drop NULLS FIRST TopK rows when null_count is missing #25239
- Incorrect filter statistics for overflowing integer multiplication #25794
- make_array returns wrong row count for null-array input #21841
- Projection field metadata is lost during physical plan optimization #24721
- Spark
map_from_arrays/map_from_entriesreturn wrong results for sliced lists #25419 - date_bin can return incorrectly ordered NULLs after timestamp overflow #25690
-
WITH ORDINALITYis silently dropped on table functions #24388
Activity
comphead commentedon Oct 24, 2026
Adding #26700 to test operators SLT with different batch sizes
kumarUjjawal commentedon Oct 24, 2026
This hasn't been resolved for a long time. I will take a look at this, I see @alamb you have reviewed the last PR which got closed due to inactivity, will see if I can borrow some of the ideas from there.
kumarUjjawal commentedon Oct 24, 2026
Seems this is already fixed on main
alamb commentedon Oct 24, 2026
Thanks @kumarUjjawal 🙏 let's just close it out
AdamGS commentedon Oct 25, 2026
This is a great list and seems fun, I'll try and pick some up.
alamb commentedon Oct 25, 2026
Thanks @AdamGS -- many of them already have PRs even -- it just needs someone to look at and help them along
kumarUjjawal commentedon Oct 26, 2026
#22906 fixes the decimal scale-narrowing bug. The logical and physical simplifiers now use shared cast-family checks, and this narrowing cast stays in the predicate instead of becoming d = 1.2300.
Verified on main at 350ca76:
matthewmturner commentedon Oct 27, 2026
Im also going to try picking some of these up
alamb commentedon Oct 28, 2026
I would like to propose slowing down feature merge velocity for a while to work down bugs
test: Add regression tests for some `unwrap_cast` related bugs that d…
Phoenix500526 commentedon Nov 3, 2026
I've been exploring the logical optimizer code and would be happy to pick up some related issues or help review existing PRs.
Phoenix500526 commentedon Nov 4, 2026
Hi @alamb, @DKER2 pointed out a problem with volatile filters in a comment on apache/datafusion#26545. I found that it also occurs with an ordinary GROUP BY, so it’s separate from the empty-input aggregation bug reported in #26545. I opened #27074 to track it and submitted a fix in #27076. Could we add #27074 to the list here?
jayzhan211 commentedon Nov 7, 2026
Two more wrong results bugs, both in the default configuration: #27194 (
ORDER BY ... LIMITsubquery over a sorted Parquet file) and #27195 (TopK aggregation when a computed column reuses the aggregate's name).