Experiment: immediate-mode shuffle #3855

Activity

1. andygrove

   self-assigned this

   on Mar 31, 2026
2. andygrove

   added

   enhancementNew feature or requestNew feature or request

   on Mar 31, 2026
3. andygrove commented on Mar 31, 2026

   

   andygrove

   on Mar 31, 2026

   MemberAuthor

   More actions

   The main issue I am running into with this approach is higher peak memory due to allocator churn due to interleave creating so many small arrays. This approach uses 2x the peak memory of the buffered approach.
4. andygrove

   added this to the 0.15.0 milestone on Mar 31, 2026
5. github-actions

   added

   requires-triage

   on Mar 31, 2026
6. andygrove commented on Mar 31, 2026

   

   andygrove

   on Mar 31, 2026

   MemberAuthor

   More actions

   Gluten follows this approach:

   • Pre-allocate one set of column buffers per partition
   • Scatter-write rows from each input batch directly into partition buffers using raw pointers
   • Accumulate across multiple input batches until buffers fill
   • Evict (flush) full buffers and reuse the allocation
   • Only resize when the estimated size changes beyond a threshold

   This avoids a lot of intermediate allocations.
7. wForget commented on Apr 3, 2026

   

   wForget

   on Apr 3, 2026

   Member

   More actions

   If I understand correctly, the current proposal is similar to gluten's hash shuffle writer (default behavior).

   • buffered-mode shuffle is similar to gluten's sort shuffle writer: insert batch -> add batches to a global buffer -> spilt by pid and evict to partition writer
   • immediate-mode shuffle is similar to gluten's hash shuffle writer: insert batch -> spilt and push to partition buffers -> evict partition buffer to parititon writer

   According to the performance report at apache/gluten#6475, the performance of the two shuffle modes may be related to the number of partitions.

   👍React with 👍1Reacted by Andy Grove
8. andygrove

   added

   priority:lowMinor issues, test failures, tooling, cosmeticMinor issues, test failures, tooling, cosmetic

   and removed

   requires-triage

   on Apr 18, 2026
9. andygrove

   modified the milestones: 0.15.0, 0.16.0 on Apr 25, 2026
10. andygrove commented on Apr 25, 2026

    

    andygrove

    on Apr 25, 2026

    MemberAuthor

    More actions

    I am closing this as completed, since we did this experiment. Although it has benefits in some cases, it does not scale well with large number of output partitions due to the need to maintain builders or batch coalesces per output partition.
11. andygrove

    closed this as completedon Apr 25, 2026
12. andygrove

    moved this to In progress in Comet Developmenton May 6, 2026
13. andygrove

    added this to Comet Developmenton May 6, 2026
14. github-project-automation

    moved this from In progress to Done in Comet Developmenton May 6, 2026