Repository navigation
fix: consider nulls_first when propagating ordered SortProperties - #24206
Conversation
|
The old query plan can lead to this sort error In datafusion-cli Get Order C was silently violated |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #24206 +/- ##
==========================================
- Coverage 81.05% 81.02% -0.03%
==========================================
Files 1106 1106
Lines 382287 384267 +1980
Branches 382287 384267 +1980
==========================================
+ Hits 309851 311364 +1513
- Misses 54121 54560 +439
- Partials 18315 18343 +28 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| OPTIONS ('format.has_header' 'true'); | ||
|
|
||
| query TT | ||
| EXPLAIN SELECT CAST((inc_col>desc_col) as integer) as c from annotated_data_finite_nulls_last order by c; |
There was a problem hiding this comment.
Do we have sqltest like the example in the description covered, like null + numeric -> null, then we expect to see the null first -> unordered
There was a problem hiding this comment.
Thanks for comment. The add case is added.
i found that the + slt would produce an identical plan with and without this patch, and wouldn't guard the change. The add/sub half of the fix is only observable in the unit tests; gt_or_gteq and and_or are the parts reachable end to end.
#23910 made BinaryExpr::arithmetic_sort_properties fall back to Unordered unless it can prove the arithmetic doesn't overflow.
let cannot_overflow = !range.is_unbounded()
&& !unsigned_subtraction_may_underflow(self.op, l_range, r_range, range);
if !wraps_in_domain && (self.fail_on_overflow || cannot_overflow) {
sort_properties
} else {
SortProperties::Unordered
}Here if endpoint null, so range.is_unbounded() holds for any col + col and the result is Unordered regardless of null placement. SortProperties::add's return value is discarded before nulls_first can matter. The SortExec still survives if declaring both columns ASC NULLS FIRST.
Before current patch:
DataFusion CLI v54.1.0
> set datafusion.execution.target_partitions = 1;
0 row(s) fetched.
Elapsed 0.001 seconds.
> COPY (VALUES (NULL, 1), (1, 2), (2, 4), (3, NULL))
TO 'test_files/scratch/order/mixed_null_placement.csv'
OPTIONS ('format.has_header' 'true');
+-------+
| count |
+-------+
| 4 |
+-------+
1 row(s) fetched.
Elapsed 0.002 seconds.
> CREATE EXTERNAL TABLE mixed_null_placement (
a BIGINT,
b BIGINT
)
STORED AS CSV
WITH ORDER (a ASC NULLS FIRST)
WITH ORDER (b ASC NULLS LAST)
LOCATION 'test_files/scratch/order/mixed_null_placement.csv'
OPTIONS ('format.has_header' 'true');
0 row(s) fetched.
Elapsed 0.001 seconds.
> SELECT a, b, a + b AS s FROM mixed_null_placement ORDER BY s ASC NULLS FIRST;
+------+------+------+
| a | b | s |
+------+------+------+
| NULL | 1 | NULL |
| 3 | NULL | NULL |
| 1 | 2 | 3 |
| 2 | 4 | 6 |
+------+------+------+
4 row(s) fetched.
Elapsed 0.003 seconds.
> EXPLAIN SELECT a + b AS s FROM mixed_null_placement ORDER BY s;
+---------------+-------------------------------+
| plan_type | plan |
+---------------+-------------------------------+
| physical_plan | ┌───────────────────────────┐ |
| | │ SortExec │ |
| | │ -------------------- │ |
| | │ s@0 ASC NULLS LAST │ |
| | └─────────────┬─────────────┘ |
| | ┌─────────────┴─────────────┐ |
| | │ DataSourceExec │ |
| | │ -------------------- │ |
| | │ files: 1 │ |
| | │ format: csv │ |
| | └───────────────────────────┘ |
| | |
+---------------+-------------------------------+
1 row(s) fetched.
Elapsed 0.001 seconds.|
Thanks @Nagato-Yuzuru |
…ache#24206) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#11596. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> In the original implementation, cases where `null_first` is not same were not correctly handled and were still treated as `ordered`. Example: `a` sorted `ASC NULLS FIRST`, `b` sorted `ASC NULLS LAST`: | a | b | a + b | | ---- | ---- | ----- | | NULL | 1 | NULL | | 1 | 2 | 3 | | 2 | 4 | 6 | | 3 | NULL | NULL | Since `a + b` is `NULL` wherever either input is, the result has nulls at both ends. The previous code returned `Ordered(ASC, nulls_first: true)` for this case; it now returns `Unordered`. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> Unit Tests were added and issues were fixed. Additional processing was performed on `and_or` (see apache#11596 (comment) ) SortProperties::{add, sub, gt_or_gteq, and_or} now propagate an Ordered result only when both operands agree on nulls_first; otherwise they return Unordered. This is conservative. If one input were known to be non-null the old result could be valid. But SortProperties carries no nullability information, so Unordered is the only sound answer at this layer. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> No.
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> Part of apache#24243 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> The equivalence fuzz tests never generated NULLs. So they could not catch NULL-ordering bugs in `SortProperties` propagation such as apache#11596 and apache#24208. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> - Generated columns contain NULLs at configurable rates (`NULL_PCTS = [0.0, 0.1, 0.5]`), crossed with every seed; the schema is nullable only when the rate is non-zero. - `create_random_schema` picks random `SortOptions` for each ordering key. - Requirements with random `SortOptions` are checked only for soundness: if `ordering_satisfy` accepts an ordering, the data must be sorted by it. The converse is not checked, since one batch can be sorted by coincidence (e.g. NULLs of independent columns landing in the same rows). For the same reason, `contains_overflowable_arithmetic` is renamed to `contains_conservative_ordering_op` and also covers comparisons and `AND`/`OR`. ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> Test-only change. Verified locally that reverting the fixes from apache#24276 and apache#24206 makes `test_ordering_satisfy_with_equivalence_complex_random` fail, and that it passes on main: ```bash git revert -n ceaef16 4042812 cargo test -p datafusion --test fuzz --features extended_tests equivalence # test_ordering_satisfy_with_equivalence_complex_random fails ``` ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> No
Which issue does this PR close?
SortPropertiesshould considernulls_first#11596.Rationale for this change
In the original implementation, cases where
null_firstis not same were not correctly handled and were still treated asordered.Example:
asortedASC NULLS FIRST,bsortedASC NULLS LAST:Since
a + bisNULLwherever either input is, the result has nulls at both ends. The previous code returnedOrdered(ASC, nulls_first: true)for this case; it now returnsUnordered.What changes are included in this PR?
Unit Tests were added and issues were fixed. Additional processing was performed on
and_or(see #11596 (comment) )SortProperties::{add, sub, gt_or_gteq, and_or} now propagate an Ordered result only when both operands agree on nulls_first; otherwise they return Unordered.
This is conservative. If one input were known to be non-null the old result could be valid. But SortProperties carries no nullability information, so Unordered is the only sound answer at this layer.
Are these changes tested?
Yes.
Are there any user-facing changes?
No.