Skip to content

Add selection vector repartitioning #15420

Open

Description

@Dandandan

Is your feature request related to a problem or challenge?

Add a mode that outputs selection vectors (for now let's use dense boolean arrays so it can be added to RecordBatch) in RepartitionExec. The array outputs true for each row that has hash % total_partition == current_partition (and false if not).

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Activity

  1. acking-you commented on Mar 25, 2025

    @acking-you
    Contributor

    After reviewing the related issues, I'm very excited about these features. I'd like to keep a close eye on their implementation, as I feel I can learn a lot from them 🥰

  2. goldmedal commented on Mar 25, 2025

    @goldmedal
    Contributor

    The array outputs true for each row that has hash % partition == 0 (and false if not).

    I don't understand why the formula is hash % partition == 0? IMO, hash % total_partition is the number of the portion it belongs. Maybe the formula should be hash % total_partition == current_partition?

    Given the following data:

    col1 | col2 | ... | hash % total_partition
    -------------------------
    data | data | ... | 2
    data | data | ... | 1
    data | data | ... | 2
    data | data | ... | 0
    

    The 0 partition will get

    col1 | col2 | ... | selection
    -------------------------
    data | data | ... | false
    data | data | ... | false
    data | data | ... | false
    data | data | ... | true
    

    The 1 partition will get

    col1 | col2 | ... | selection
    -------------------------
    data | data | ... | false
    data | data | ... | true
    data | data | ... | false
    data | data | ... | false
    

    The 2 partition will get

    col1 | col2 | ... | selection
    -------------------------
    data | data | ... | true
    data | data | ... | false
    data | data | ... | true
    data | data | ... | false
    

    Then, the following plan can aggregate or join the record which selection is true.
    Does it make sense?

  3. goldmedal commented on Mar 25, 2025

    @goldmedal
    Contributor

    I'm not sure, but I created a draft #15423 for my idea.

  4. Dandandan commented on Mar 25, 2025

    @Dandandan
    ContributorAuthor

    Oh you're right @goldmedal , the formula is hash % total_partition == current_partition

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions