Stop copying LogicalPlan and Exprs in ReplaceDistinctWithAggregate#10460
Merged
alamb merged 5 commits intoapache:mainfrom May 13, 2024
Merged
Stop copying LogicalPlan and Exprs in ReplaceDistinctWithAggregate#10460alamb merged 5 commits intoapache:mainfrom
ReplaceDistinctWithAggregate#10460alamb merged 5 commits intoapache:mainfrom
Conversation
Signed-off-by: cailue <[email protected]>
Signed-off-by: 蔡略 <[email protected]>
ClSlaid
commented
May 11, 2024
Comment on lines
+109
to
+110
| let aggr_expr = normalize_cols(aggr_expr, input.as_ref())?; | ||
| let group_expr = normalize_cols(on_expr, input.as_ref())?; |
Comment on lines
92
to
107
| let aggr_expr = select_expr | ||
| .iter() | ||
| .map(|e| { | ||
| Expr::AggregateFunction(AggregateFunction::new( | ||
| AggregateFunctionFunc::FirstValue, | ||
| vec![e.clone()], | ||
| false, | ||
| None, | ||
| sort_expr.clone(), | ||
| None, | ||
| )) | ||
| }) | ||
| .collect::<Vec<Expr>>(); | ||
| let aggr_expr = vec![Expr::AggregateFunction(AggregateFunction::new( | ||
| AggregateFunctionFunc::FirstValue, | ||
| select_expr, | ||
| false, | ||
| None, | ||
| sort_expr.clone(), | ||
| None, | ||
| ))]; |
Contributor
Author
There was a problem hiding this comment.
I don't know if they are semantically identical.
Instructions welcomed.
Contributor
Author
There was a problem hiding this comment.
CI failed, i guess the answer is no. :(
Contributor
There was a problem hiding this comment.
The difference is that the select_expr.iter()... makes a vec with the same number of elements as select_expr
To avoid a clone, perhaps you can use into_iter instead of iter, like
let aggr_expr = select_expr
.into_iter() // <---- Use into_iter() here
.map(|e| {
Expr::AggregateFunction(AggregateFunction::new(
AggregateFunctionFunc::FirstValue,
vec![e], // <--- to avoid a clone here
false,
None,
sort_expr.clone(),
None,
))
})
.collect::<Vec<Expr>>();
Contributor
Author
There was a problem hiding this comment.
The difference is that the
select_expr.iter()...makes avecwith the same number of elements asselect_exprTo avoid a clone, perhaps you can use
into_iterinstead ofiter, likelet aggr_expr = select_expr .into_iter() // <---- Use into_iter() here .map(|e| { Expr::AggregateFunction(AggregateFunction::new( AggregateFunctionFunc::FirstValue, vec![e], // <--- to avoid a clone here false, None, sort_expr.clone(), None, )) }) .collect::<Vec<Expr>>();
What bothers me most is that I cannot avoid cloning sort_expr. However, it looks like the only way to make it correct.
Signed-off-by: 蔡略 <[email protected]>
alamb
changed the title
ReplaceDistinctWithAggregate
This was referenced May 12, 2024
alamb
approved these changes
May 13, 2024
Contributor
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
findepi
pushed a commit
to findepi/datafusion
that referenced
this pull request
Jul 16, 2024
…pache#10460) * patch: implement rewrite for RDWA Signed-off-by: cailue <[email protected]> * refactor: rewrite replace_distinct_aggregate Signed-off-by: 蔡略 <[email protected]> * patch: recorrect aggr_expr Signed-off-by: 蔡略 <[email protected]> * Update datafusion/optimizer/src/replace_distinct_aggregate.rs --------- Signed-off-by: cailue <[email protected]> Signed-off-by: 蔡略 <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #10293.
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?