You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1c3232c
Browse filesBrowse the repository at this point in the historyBrowse files
ScalarUdfImpl::strictly_order_preserving: Allow expression to report whether they keep the same ordering of the input (#23807)
## Which issue does this PR close?
- Closes#23798
Related to:
- #16217
## Rationale for this change
To be able to keep the same sorting order allowing for more
optimizations
Now comet own `cast` implementation can be recognized as not modifying
sort order in the same cases that datafusion cast does.
## What changes are included in this PR?
added `strictly_order_preserving` property to `ExprProperties` + varius
other places
and replaced the hard coded logic for cast (`substitute_cast_ordering`)
about keeping input order with more generic approach that now any
expression can implement and have the same advantage of sort elimination
also marked from_unixtime as keeping ordering to show a case of this
optimization
## Are these changes tested?
yes
## Are there any user-facing changes?
yes, breaking change, added `strictly_order_preserving` property to
`ExprProperties` and to `FFI_ExprProperties`
this property means that given expression `f` and 2 values from the
input column `a` and `b` the following variants are kept:
1. `a.cmp(b) == f(a).cmp(f(b))`
2. nulls maps to nulls
Example of satisfying expression:
`cast(col_a as BIGINT)` where `col_a` is `INT` it is keeping the
properties
Example of not satisfying:
`floor` - floor can not
`array_repeat(my_col, 2)` which might look like at first glance as
keeping the property as well but in fact it does not.
the reason is that `array_repeat(null, 2)` will output list of 2 nulls
which breaks the 2nd property that nulls must be kept as nulls
how to migrate:
Option 1 - keeping the old behavior (safest but least performant)
set `strictly_order_preserving` to false
Option 2 - Using the new optimization that this opens up:
set `strictly_order_preserving` only if the expression keep both
variants
Copy file name to clipboardExpand all lines: datafusion/core/tests/physical_optimizer/enforce_sorting.rs
+154-3Lines changed: 154 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -33,7 +33,7 @@ use arrow::compute::{SortOptions};
33
33
use arrow::datatypes::{DataType,SchemaRef};
34
34
use datafusion_common::config::{ConfigOptions,CsvOptions};
35
35
use datafusion_common::tree_node::{TreeNode,TransformedResult};
36
-
use datafusion_common::{create_array,NullEquality,Result,TableReference};
36
+
use datafusion_common::{create_array,DataFusionError,NullEquality,Result,TableReference};
37
37
use datafusion_datasource::file_scan_config::FileScanConfigBuilder;
38
38
use datafusion_datasource::source::DataSourceExec;
39
39
use datafusion_expr_common::operator::Operator;
@@ -49,7 +49,7 @@ use datafusion_physical_plan::limit::{GlobalLimitExec, LocalLimitExec};
49
49
use datafusion_physical_plan::repartition::RepartitionExec;
50
50
use datafusion_physical_plan::sorts::sort_preserving_merge::SortPreservingMergeExec;
51
51
use datafusion_physical_plan::sorts::sort::SortExec;
52
-
use datafusion_physical_plan::{displayable, get_plan_string,ExecutionPlan};
52
+
use datafusion_physical_plan::{displayable, get_plan_string,ExecutionPlan,ExecutionPlanProperties};
53
53
use datafusion::datasource::physical_plan::CsvSource;
54
54
use datafusion::datasource::listing::PartitionedFile;
55
55
use datafusion_physical_optimizer::enforce_sorting::{PlanWithCorrespondingCoalescePartitions,PlanWithCorrespondingSort, parallelize_sorts, ensure_sorting};
@@ -60,12 +60,15 @@ use datafusion_physical_optimizer::ensure_requirements::EnsureRequirements;
60
60
use datafusion_physical_optimizer::output_requirements::OutputRequirementExec;
61
61
use datafusion_physical_optimizer::PhysicalOptimizerRule;
62
62
use datafusion::prelude::*;
63
-
use arrow::array::{record_batch,ArrayRef,Int32Array,RecordBatch};
63
+
use arrow::array::{record_batch,Array,ArrayRef,Int32Array,RecordBatch};
64
64
use arrow::datatypes::{Field};
65
65
use arrow_schema::Schema;
66
66
use datafusion_execution::TaskContext;
67
67
use datafusion_catalog::streaming::StreamingTable;
68
68
69
+
use datafusion_expr_common::columnar_value::ColumnarValue;
70
+
use datafusion_physical_expr::projection::ProjectionExpr;
71
+
use datafusion_physical_plan::projection::ProjectionExec;
69
72
use futures::StreamExt;
70
73
use insta::{Settings, assert_snapshot};
71
74
@@ -3255,3 +3258,151 @@ async fn test_does_not_push_fetch_sort_through_projection_over_union() -> Result
3255
3258
3256
3259
Ok(())
3257
3260
}
3261
+
3262
+
/// A pass-through wrapper around a column: just assert that column does not contain any nulls
0 commit comments