You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 62d3725
Browse filesBrowse the repository at this point in the historyBrowse files
## Which issue does this PR close?
- Closesapache#24173
## Rationale for this change
`GlobalLimitExec`/`LocalLimitExec` `required_ordering` is not
serialized, so a roundtrip loses the only record that a pushed-down
`LIMIT` is order-sensitive. Re-running `LimitPushdown` on the decoded
plan then sets `preserve_order: false` on the scan, which is free to
read files out of order — `ORDER BY ... LIMIT` can return the wrong
rows.
## What changes are included in this PR?
- Add `required_ordering` to `GlobalLimitExecNode` (field 4) and
`LocalLimitExecNode` (field 3); empty list means `None`
- New
`optional_ordering_try_to_proto`/`optional_ordering_try_from_proto`
helpers in `physical-expr-common`, also reused by
`SymmetricHashJoinExec` serde which hand-rolled the same pattern
- Preserve `required_ordering` in both execs' `with_new_children`
## Are these changes tested?
Yes new tests that would fail on main if not fixed
## Are there any user-facing changes?
No
Copy file name to clipboardExpand all lines: datafusion/core/tests/physical_optimizer/enforce_sorting.rs
+52-1Lines changed: 52 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -34,7 +34,7 @@ use arrow::datatypes::{DataType, SchemaRef};
34
34
use datafusion_common::config::{ConfigOptions,CsvOptions};
35
35
use datafusion_common::tree_node::{TreeNode,TransformedResult};
36
36
use datafusion_common::{create_array,DataFusionError,NullEquality,Result,TableReference};
37
-
use datafusion_datasource::file_scan_config::FileScanConfigBuilder;
37
+
use datafusion_datasource::file_scan_config::{FileScanConfig,FileScanConfigBuilder};
38
38
use datafusion_datasource::source::DataSourceExec;
39
39
use datafusion_expr_common::operator::Operator;
40
40
use datafusion_expr::{JoinType,SortExpr};
@@ -57,6 +57,8 @@ use datafusion_physical_optimizer::sanity_checker::SanityCheckPlan;
57
57
use datafusion_physical_optimizer::enforce_sorting::replace_with_order_preserving_variants::{replace_with_order_preserving_variants,OrderPreservationContext};
58
58
use datafusion_physical_optimizer::enforce_sorting::sort_pushdown::{SortPushDown, assign_initial_requirements, pushdown_sorts};
59
59
use datafusion_physical_optimizer::ensure_requirements::EnsureRequirements;
60
+
use datafusion_physical_optimizer::limit_pushdown::LimitPushdown;
61
+
use datafusion_physical_optimizer::projection_pushdown::ProjectionPushdown;
60
62
use datafusion_physical_optimizer::output_requirements::OutputRequirementExec;
61
63
use datafusion_physical_optimizer::PhysicalOptimizerRule;
0 commit comments