You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Consider row-count skew when skipping aggregate repartitioning #27268
Skipping aggregate repartitioning can preserve a heavily skewed input layout, leaving most of the grouping work on one worker even when more execution partitions are available.
In Jay's benchmark on #27135, one of four files held 97% of 16M rows. With preserve_file_partitions enabled and mostly unique groups, the SinglePartitioned plan took 4.11 times as long as the shuffle plan at target_partitions=8. The two-argument date_bin showed the same behavior.
Consider per-partition row counts when deciding whether to skip aggregate repartitioning, so a shuffle can still be used when its load-balancing benefit outweighs its cost. Related to #19269, which describes the same trade-off for hash-subset partitioning.
Activity