You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Potential performance regression for TPCH q18 #13188
select
c_name,
c_custkey,
o_orderkey,
o_orderdate,
o_totalprice,
sum(l_quantity)
from
customer,
orders,
lineitem
where
o_orderkey in (
select
l_orderkey
from
lineitem
group by
l_orderkey havingsum(l_quantity) >300
)
and c_custkey = o_custkey
and o_orderkey = l_orderkey
group by
c_name,
c_custkey,
o_orderkey,
o_orderdate,
o_totalprice
order by
o_totalprice desc,
o_orderdate;
To Reproduce
To reproduce
Make data
# make the data and get to the correct locationcd datafusion/benchmarks
./bench.sh data tpch
cd data/tpch_sf1
debug_assert!(gc_string.data_buffers().len() <= 1);// buffer count can be 0 if the `ideal_buffer_size` is 0
Arc::new(gc_string)
}else{
I noticed when writing up a small example using this code path locally.
When taking a look at the append_option source code it implements a copy.
I've identified where the performance fault is--but I'm unsure what next steps to take to try and alleviate it. I was going to maybe modify the coalescer to try and "change the RecordBatch to stringview if the datatype is Utf8View" as the input occurs around here (if that makes sense):
#11628 might have some ideas. I think @XiangpengHao has also been thinking of some way to do this too
Basically one thing you might be able to try is to switch from using take to directly building the output (with a StringViewBuilder or something equvialent)
But as you are hinting at this is a non trivial thing to do
Other than apache/arrow-rs#6692.
If we create filter version of GroupColumn that accumulate the filtered array into array builder for each column, output if the batch size reach target (i.e. 8192). We can then aggregate those small batches in filter exec and produce a single large output for next step. Does this sound a possible improvement?
I think the downside is again we need to implement builder per type like GroupColumn and the improvement is unclear
If we create filter version of GroupColumn that accumulate the filtered array into array builder for each column, output if the batch size reach target (i.e. 8192)
Would this be a specialization of a filter of the aggregates like for SELECT SUM(x) FROM ... HAVING SUM(x) > 5 ?
If we create filter version of GroupColumn that accumulate the filtered array into array builder for each column, output if the batch size reach target (i.e. 8192)
Would this be a specialization of a filter of the aggregates like for SELECT SUM(x) FROM ... HAVING SUM(x) > 5 ?
Probably, but I think it has potential to reduce copied too #11628. It aggregates at the first place similar to coalescer's role and no gc required since we aggregate the value without necessary buffer copied.
If someone else would like to take this please let me know. I may not be able to look at this again until the upcoming weekend.
Feel free to work on it since this kind of task is usually not that trivial and everyone may come out a different solution (less chance of duplicated work)
Activity
alamb commentedon Oct 30, 2024
This may be fixed by fixing #11628
devanbenz commentedon Nov 3, 2024
take
jayzhan211 commentedon Nov 5, 2024
I will take a look on this issue too, how is your progress on this @devanbenz ?
devanbenz commentedon Nov 5, 2024
Looking at the following code path:
datafusion/datafusion/physical-plan/src/coalesce/mod.rs
Lines 246 to 263 in 9005585
I noticed when writing up a small example using this code path locally.
When taking a look at the
append_optionsource code it implements a copy.I've identified where the performance fault is--but I'm unsure what next steps to take to try and alleviate it. I was going to maybe modify the coalescer to try and "change the RecordBatch to stringview if the datatype is
Utf8View" as the input occurs around here (if that makes sense):datafusion/datafusion/physical-plan/src/coalesce_batches.rs
Lines 297 to 307 in 9005585
I don't have a meaningful plan yet just have been doing some exploratory work as of right now while benchmarking locally.
alamb commentedon Nov 5, 2024
#11628 might have some ideas. I think @XiangpengHao has also been thinking of some way to do this too
Basically one thing you might be able to try is to switch from using
taketo directly building the output (with a StringViewBuilder or something equvialent)But as you are hinting at this is a non trivial thing to do
alamb commentedon Nov 5, 2024
I finally filed a ticket upstream in arrow trying to explain what I have been thinking about:
jayzhan211 commentedon Nov 7, 2024
Other than apache/arrow-rs#6692.
If we create filter version of
GroupColumnthat accumulate the filtered array into array builder for each column, output if the batch size reach target (i.e. 8192). We can then aggregate those small batches in filter exec and produce a single large output for next step. Does this sound a possible improvement?I think the downside is again we need to implement builder per type like
GroupColumnand the improvement is unclearalamb commentedon Nov 7, 2024
Would this be a specialization of a filter of the aggregates like for
SELECT SUM(x) FROM ... HAVING SUM(x) > 5?jayzhan211 commentedon Nov 8, 2024
Probably, but I think it has potential to reduce copied too #11628. It aggregates at the first place similar to coalescer's role and no gc required since we aggregate the value without necessary buffer copied.
devanbenz commentedon Nov 20, 2024
If someone else would like to take this please let me know. I may not be able to look at this again until the upcoming weekend.
jayzhan211 commentedon Nov 20, 2024
Feel free to work on it since this kind of task is usually not that trivial and everyone may come out a different solution (less chance of duplicated work)
alamb commentedon Nov 3, 2026
I don't think this is still trackng anything actionable. If we see something let's open a new PR