Skip to content

fix: consider nulls_first when propagating ordered SortProperties - #24206

Merged
jayzhan211 merged 4 commits into
apache:mainfrom
Nagato-Yuzuru:issue-11596
Aug 16, 2026
Merged

jayzhan211 merged 4 commits into
apache:mainfrom
Nagato-Yuzuru:issue-11596

Conversation

@Nagato-Yuzuru

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

In the original implementation, cases where null_first is not same were not correctly handled and were still treated as ordered.

Example: a sorted ASC NULLS FIRST, b sorted ASC NULLS LAST:

a b a + b
NULL 1 NULL
1 2 3
2 4 6
3 NULL NULL

Since a + b is NULL wherever either input is, the result has nulls at both ends. The previous code returned Ordered(ASC, nulls_first: true) for this case; it now returns Unordered.

What changes are included in this PR?

Unit Tests were added and issues were fixed. Additional processing was performed on and_or (see #11596 (comment) )

SortProperties::{add, sub, gt_or_gteq, and_or} now propagate an Ordered result only when both operands agree on nulls_first; otherwise they return Unordered.

This is conservative. If one input were known to be non-null the old result could be valid. But SortProperties carries no nullability information, so Unordered is the only sound answer at this layer.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

@github-actions github-actions Bot added the logical-expr Logical plan and expressions label Aug 9, 2026
@Nagato-Yuzuru
Nagato-Yuzuru marked this pull request as ready for review August 9, 2026 18:19
@Nagato-Yuzuru Nagato-Yuzuru changed the title Fixed where binary expressions were passed in the wrong order when the null directions differed fix: consider nulls_first when propagating ordered SortProperties Aug 9, 2026
@Nagato-Yuzuru

Copy link
Copy Markdown
Contributor Author

The old query plan can lead to this sort error

In datafusion-cli

DataFusion CLI v54.1.0

>set datafusion.execution.target_partitions = 1;

-- Materialize the data through SQL so this repro is self-contained.
COPY (SELECT * FROM (VALUES
  (1,                 CAST(NULL AS INT)),
  (2,                 10),
  (3,                 1),
  (CAST(NULL AS INT), 0)
) AS t(inc_col, desc_col))
TO '/tmp/nulls_demo.csv' STORED AS CSV;

CREATE EXTERNAL TABLE demo (
  inc_col INTEGER,
  desc_col INTEGER
)
STORED AS CSV
WITH ORDER (inc_col ASC)     -- ASC  defaults to NULLS LAST  -> nulls at the end
WITH ORDER (desc_col DESC)   -- DESC defaults to NULLS FIRST -> nulls at the start
LOCATION '/tmp/nulls_demo.csv'
OPTIONS ('format.has_header' 'true');

-- Both columns really do match their declared orderings.
SELECT * FROM demo;

SELECT CAST((inc_col > desc_col) AS integer) AS c FROM demo ORDER BY c;

Get

+---------+----------+
| inc_col | desc_col |
+---------+----------+
| 1       | NULL     |   inc_col:  1, 2, 3, NULL  -> ASC NULLS LAST  ✓
| 2       | 10       |   desc_col: NULL, 10, 1, 0 -> DESC NULLS FIRST ✓
| 3       | 1        |
| NULL    | 0        |
+---------+----------+


+------+
| c    |
+------+
| NULL |
| 0    |
| 1    |
| NULL |
+------+

Order C was silently violated

@github-actions github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Aug 9, 2026
@codecov-commenter

codecov-commenter commented Aug 9, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.02%. Comparing base (eec8b94) to head (db9e889).
⚠️ Report is 34 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24206      +/-   ##
==========================================
- Coverage   81.05%   81.02%   -0.03%     
==========================================
  Files        1106     1106              
  Lines      382287   384267    +1980     
  Branches   382287   384267    +1980     
==========================================
+ Hits       309851   311364    +1513     
- Misses      54121    54560     +439     
- Partials    18315    18343      +28     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

OPTIONS ('format.has_header' 'true');

query TT
EXPLAIN SELECT CAST((inc_col>desc_col) as integer) as c from annotated_data_finite_nulls_last order by c;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have sqltest like the example in the description covered, like null + numeric -> null, then we expect to see the null first -> unordered

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for comment. The add case is added.

i found that the + slt would produce an identical plan with and without this patch, and wouldn't guard the change. The add/sub half of the fix is only observable in the unit tests; gt_or_gteq and and_or are the parts reachable end to end.

#23910 made BinaryExpr::arithmetic_sort_properties fall back to Unordered unless it can prove the arithmetic doesn't overflow.

 let cannot_overflow = !range.is_unbounded()
            && !unsigned_subtraction_may_underflow(self.op, l_range, r_range, range);
        if !wraps_in_domain && (self.fail_on_overflow || cannot_overflow) {
            sort_properties
        } else {
            SortProperties::Unordered
        }

Here if endpoint null, so range.is_unbounded() holds for any col + col and the result is Unordered regardless of null placement. SortProperties::add's return value is discarded before nulls_first can matter. The SortExec still survives if declaring both columns ASC NULLS FIRST.

Before current patch:

DataFusion CLI v54.1.0
> set datafusion.execution.target_partitions = 1;
0 row(s) fetched.
Elapsed 0.001 seconds.

> COPY (VALUES (NULL, 1), (1, 2), (2, 4), (3, NULL))
TO 'test_files/scratch/order/mixed_null_placement.csv'
OPTIONS ('format.has_header' 'true');
+-------+
| count |
+-------+
| 4     |
+-------+
1 row(s) fetched.
Elapsed 0.002 seconds.

> CREATE EXTERNAL TABLE mixed_null_placement (
  a BIGINT,
  b BIGINT
)
STORED AS CSV
WITH ORDER (a ASC NULLS FIRST)
WITH ORDER (b ASC NULLS LAST)
LOCATION 'test_files/scratch/order/mixed_null_placement.csv'
OPTIONS ('format.has_header' 'true');
0 row(s) fetched.
Elapsed 0.001 seconds.

> SELECT a, b, a + b AS s FROM mixed_null_placement ORDER BY s ASC NULLS FIRST;
+------+------+------+
| a    | b    | s    |
+------+------+------+
| NULL | 1    | NULL |
| 3    | NULL | NULL |
| 1    | 2    | 3    |
| 2    | 4    | 6    |
+------+------+------+
4 row(s) fetched.
Elapsed 0.003 seconds.

> EXPLAIN SELECT a + b AS s FROM mixed_null_placement ORDER BY s;
+---------------+-------------------------------+
| plan_type     | plan                          |
+---------------+-------------------------------+
| physical_plan | ┌───────────────────────────┐ |
|               | │          SortExec         │ |
|               | │    --------------------   │ |
|               | │     s@0 ASC NULLS LAST    │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │       DataSourceExec      │ |
|               | │    --------------------   │ |
|               | │          files: 1         │ |
|               | │        format: csv        │ |
|               | └───────────────────────────┘ |
|               |                               |
+---------------+-------------------------------+
1 row(s) fetched.
Elapsed 0.001 seconds.

@jayzhan211
jayzhan211 added this pull request to the merge queue Aug 16, 2026
@jayzhan211

Copy link
Copy Markdown
Contributor

Thanks @Nagato-Yuzuru

Merged via the queue into apache:main with commit 4042812 Aug 16, 2026
41 checks passed
imtherealnaska pushed a commit to imtherealnaska/datafusion that referenced this pull request Aug 16, 2026
…ache#24206)

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes apache#123` indicates that this PR will close issue apache#123.
-->

- Closes apache#11596.

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.

Please explain the problem you are trying to solve in terms of the
user-visible
behavior, rather than the implementation.

For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of
the
implementation. "COUNT(DISTINCT) returns wrong results when the column
contains
nulls" is the user-visible problem.
-->

In the original implementation, cases where `null_first` is not same
were not correctly handled and were still treated as `ordered`.


Example: `a` sorted `ASC NULLS FIRST`, `b` sorted `ASC NULLS LAST`:

| a    | b    | a + b |
| ---- | ---- | ----- |
| NULL | 1    | NULL  |
| 1    | 2    | 3     |
| 2    | 4    | 6     |
| 3    | NULL | NULL  |

Since `a + b` is `NULL` wherever either input is, the result has nulls
at both ends. The previous code returned `Ordered(ASC, nulls_first:
true)` for this case; it now returns `Unordered`.


## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here, but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
Unit Tests were added and issues were fixed. Additional processing was
performed on `and_or` (see
apache#11596 (comment)
)

SortProperties::{add, sub, gt_or_gteq, and_or} now propagate an Ordered
result only when both operands agree on nulls_first; otherwise they
return Unordered.

This is conservative. If one input were known to be non-null the old
result could be valid. But SortProperties carries no nullability
information, so Unordered is the only sound answer at this layer.
## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
Yes.

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.

If there are any breaking changes to public APIs, please add the `api
change` label.
-->
No.
haochunchang pushed a commit to haochunchang/datafusion that referenced this pull request Oct 28, 2026
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes apache#123` indicates that this PR will close issue apache#123.
-->

Part of apache#24243 

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.

Please explain the problem you are trying to solve in terms of the
user-visible
behavior, rather than the implementation.

For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of
the
implementation. "COUNT(DISTINCT) returns wrong results when the column
contains
nulls" is the user-visible problem.
-->
The equivalence fuzz tests never generated NULLs. So they could not
catch NULL-ordering bugs in `SortProperties` propagation such as apache#11596
and apache#24208.


## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here, but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
- Generated columns contain NULLs at configurable rates (`NULL_PCTS =
[0.0, 0.1, 0.5]`), crossed with every seed; the schema is nullable only
when the rate is non-zero.
- `create_random_schema` picks random `SortOptions` for each ordering
key.
- Requirements with random `SortOptions` are checked only for soundness:
if `ordering_satisfy` accepts an ordering, the data must be sorted by
it. The converse is not checked, since one batch can be sorted by
coincidence (e.g. NULLs of independent columns landing in the same
rows). For the same reason, `contains_overflowable_arithmetic` is
renamed to `contains_conservative_ordering_op` and also covers
comparisons and `AND`/`OR`.

## What is the testing strategy for this PR?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

Briefly describe how this PR is tested, and point to the specific tests
you added. For example: 'This new feature is covered by the
`sqllogictest` cases added in `foo.slt`'.

If this PR does not add tests, explain why. For example, if the change
is already covered by existing tests, please mention it.

You should also check the `codecov` bot reply on this PR to confirm the
changed code is exercised.
-->


Test-only change. Verified locally that reverting the fixes from apache#24276
and apache#24206 makes `test_ordering_satisfy_with_equivalence_complex_random`
fail, and that it passes on main:

```bash
git revert -n ceaef16 4042812
cargo test -p datafusion --test fuzz --features extended_tests equivalence
#  test_ordering_satisfy_with_equivalence_complex_random fails
```

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.

If there are any breaking changes to public APIs, please add the `api
change` label.
-->
No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt) v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Propagation of ordered SortProperties should consider nulls_first

4 participants