Repository navigation
Improve performance for roundrobin shuffle writes for nested schemas #5397
Description
What is the problem the feature request solves?
Noticed dramatic slowdown for tasks with shuffle writes for RR repartition comparing to Spark, 2-3x to be more precise.
Specifically for the nested schemas, there is some inefficiency when native shuffle deals with wide/complex fields. its invisible for TPC-* benchmarks because of historical flat schema. But in real user world this is pretty much downside.
Describe the potential solution
No response
Additional context
No response
Activity
comphead commentedon Aug 20, 2026
MIght be related to obtain correct shuffle metrics #5370
comphead commentedon Aug 21, 2026
Spark
Comet
With vectorized shuffle
[-]Improve performance for shuffle writes for nested schemas[/-][+]Improve performance for roundrobin shuffle writes for nested schemas[/+]comphead commentedon Aug 22, 2026
I'm experimenting with
comphead commentedon Aug 22, 2026
with vectorized shuffle pretty good results
UPD: This was another dataset experiment, and originally Comet took 21 mins to complete, now 10. It is still experimenting
comphead commentedon Aug 24, 2026
xxhash-rust experiment, takes much longer