Introduce selection vector repartitioning by goldmedal · Pull Request #15423 · apache/datafusion

goldmedal · 2025-03-25T17:47:18Z

Which issue does this PR close?

Closes Add selection vector repartitioning #15420

Rationale for this change

It's a pre-work of #15382 and #15383.

What changes are included in this PR?

Add a config to control the hash partitioning mode: datafusion.optimizer.prefer_hash_selection_vector_partitioning

        /// When set to true, the physical plan optimizer will prefer HashSelectionVectorPartitioning for RepartitionExec
        /// over HashPartitioning. HashSelectionVectorPartitioning can work without data copying.
        pub prefer_hash_selection_vector_partitioning_agg: bool, default = false

Add a new physical hash partitioning to change the behavior of RepartitionExec

    /// Allocate rows based on a hash of one or more expressions and the specified number
    /// of partitions with a selection vector column.
    ///
    /// The column is a boolean column called `__selection` that is used to filter out rows
    /// that should not be included in the partition. `true` means the row should be included
    /// and `false` means the row should be excluded.
    HashSelectionVector(Vec<Arc<dyn PhysicalExpr>>, usize),

Add HashPartitionMode to decide the hash partition behavior when planning physical plans.

/// The mode of hash partitioning
#[derive(Debug, Copy, Clone, PartialEq, Eq)]
pub enum HashPartitionMode {
    /// Hash partitioning with a selection vector. See [Partitioning::HashSelectionVector]
    SelectionVector,
    /// The default hash partitioning
    HashPartitioned,
}

Are these changes tested?

Are there any user-facing changes?

Dandandan · 2025-03-26T09:26:48Z

This starts to look nice @goldmedal

goldmedal · 2025-03-27T16:41:58Z

After rebasing to the main, the hash join no longer uses RepartitionExec. Remove the test by 3a151dd

goldmedal · 2025-03-27T16:57:42Z

#15339 It looks like the join plan is being changed.

goldmedal · 2025-03-27T17:55:20Z

also c.c. @zebsme

Dandandan · 2025-03-28T09:55:38Z

#15339 It looks like the join plan is being changed.

You should be able to get the test back by also setting datafusion.optimizer.hash_join_single_partition_threshold to 0 / a low value.

goldmedal · 2025-03-28T13:46:43Z

You should be able to get the test back by also setting datafusion.optimizer.hash_join_single_partition_threshold to 0 / a low value.

Thanks. It works. I also added the test for the normal hash partition for RepartitionExec.

goldmedal · 2025-03-29T13:06:00Z

I'm working on HashAggregate goldmedal#3 based on this PR.
I found we shouldn't use only one config, prefer_hash_selection_vector_partitioning, to control whether to use the selection vector or not. For testing and benchmark convenience, it's better to split it into two configs, one for hash join and another for HashAggregate.

zebsme

Thanks for your contribution! @goldmedal

Dandandan · 2025-03-29T17:33:21Z

I'm working on HashAggregate goldmedal#3 based on this PR. I found we shouldn't use only one config, prefer_hash_selection_vector_partitioning, to control whether to use the selection vector or not. For testing and benchmark convenience, it's better to split it into two configs, one for hash join and another for HashAggregate.

Makes sense to me 👍

goldmedal · 2025-03-29T18:49:37Z

+# TODO: The selection vector partitioning should be used for the hash join.
+# After fix https://github.com/apache/datafusion/issues/15382


I didn't implement the planner for the hash join to avoid making this PR huge and complex. I think #15382 will implement the required parts.

berkaysynnada · 2025-03-29T22:18:34Z

Isn't it always better partitioning on this selection vectors in case of hash-rep 🤔 What is the reason of keeping the old strategy ?

2010YOUY01 · 2025-03-30T03:18:50Z

Isn't it always better partitioning on this selection vectors in case of hash-rep 🤔 What is the reason of keeping the old strategy ?

I think to support this selection vector, the executors need to be updated to interpret an additional metadata column. However, since executors are part of the public interface that some downstream projects might depend on directly, it's better to ensure backward compatibility.

goldmedal · 2025-03-30T07:08:05Z

I think to support this selection vector, the executors need to be updated to interpret an additional metadata column. However, since executors are part of the public interface that some downstream projects might depend on directly, it's better to ensure backward compatibility.

I agree with @2010YOUY01. It would be a breaking change for RepartitionExec.
After this PR, we still have the following-up works for #15382 and #15383. Maybe we can use the selection vector by default, but I also prefer to keep the original strategy for backward compatibility for a while.

goldmedal · 2025-03-30T07:46:11Z

The CI failure isn't related to this PR. It could be fixed by #15493

github-actions · 2025-07-10T02:13:12Z

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

Rich-T-kid · 2026-09-17T03:10:32Z

run benchmarks

adriangbot · 2026-09-17T03:12:40Z

Benchmark for this request failed before finishing (Kubernetes reason: BackoffLimitExceeded).

Benchmarks requested: clickbench_partitioned

Runner log (last 40 lines)

2026-09-17T03:12:09.523486Z  INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/15423, benchmarks=clickbench_partitioned
2026-09-17T03:12:09.588055Z  INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
2026-09-17T03:12:09.588176Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
2026-09-17T03:12:14.597471Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/15423/head:feat/selection-vector-partition", "main"], cwd="/workspace/datafusion-branch"
2026-09-17T03:12:24.600269Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "feat/selection-vector-partition"], cwd="/workspace/datafusion-branch"
2026-09-17T03:12:29.601910Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
2026-09-17T03:12:34.604244Z ERROR runner benchmark failed error=git merge-base: git merge-base HEAD origin/main exited with code 1
stdout:

stderr:

Kubernetes message

Job has reached the specified backoff limit

File an issue against this benchmark runner

adriangbot · 2026-09-17T03:13:38Z

Benchmark for this request failed before finishing (Kubernetes reason: BackoffLimitExceeded).

Benchmarks requested: tpcds

Runner log (last 40 lines)

2026-09-17T03:13:03.335733Z  INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/15423, benchmarks=tpcds
2026-09-17T03:13:03.415173Z  INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
2026-09-17T03:13:03.415310Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
2026-09-17T03:13:08.423940Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/15423/head:feat/selection-vector-partition", "main"], cwd="/workspace/datafusion-branch"
2026-09-17T03:13:23.431204Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "feat/selection-vector-partition"], cwd="/workspace/datafusion-branch"
2026-09-17T03:13:28.433219Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
2026-09-17T03:13:33.435599Z ERROR runner benchmark failed error=git merge-base: git merge-base HEAD origin/main exited with code 1
stdout:

stderr:

Kubernetes message

Job has reached the specified backoff limit

File an issue against this benchmark runner

adriangbot · 2026-09-17T03:13:51Z

Benchmark for this request failed before finishing (Kubernetes reason: BackoffLimitExceeded).

Benchmarks requested: tpch

Runner log (last 40 lines)

2026-09-17T03:13:20.575817Z  INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/15423, benchmarks=tpch
2026-09-17T03:13:20.635718Z  INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
2026-09-17T03:13:20.635865Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
2026-09-17T03:13:25.645713Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/15423/head:feat/selection-vector-partition", "main"], cwd="/workspace/datafusion-branch"
2026-09-17T03:13:35.648746Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "feat/selection-vector-partition"], cwd="/workspace/datafusion-branch"
2026-09-17T03:13:40.650540Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
2026-09-17T03:13:45.652499Z ERROR runner benchmark failed error=git merge-base: git merge-base HEAD origin/main exited with code 1
stdout:

stderr:

Kubernetes message

Job has reached the specified backoff limit

File an issue against this benchmark runner

goldmedal mentioned this pull request Mar 25, 2025

Add selection vector repartitioning #15420

Open

Dandandan reviewed Mar 26, 2025

View reviewed changes

Comment thread datafusion/physical-plan/src/repartition/mod.rs Resolved

Dandandan reviewed Mar 26, 2025

View reviewed changes

Comment thread datafusion/physical-plan/src/repartition/mod.rs Resolved

github-actions Bot added physical-expr Changes to the physical-expr crates optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) common Related to common crate proto Related to proto crate labels Mar 26, 2025

goldmedal force-pushed the feat/selection-vector-partition branch from 87a84f9 to 3a151dd Compare March 27, 2025 16:39

github-actions Bot added the documentation Improvements or additions to documentation label Mar 27, 2025

goldmedal marked this pull request as ready for review March 27, 2025 17:54

goldmedal requested a review from Dandandan March 27, 2025 17:54

This was referenced Mar 29, 2025

Implement selection vector partitioning mode for HashAggregate goldmedal/datafusion#3

Closed

Support zero copy hash repartitioning for Hash Aggregate #15383

Open

zebsme reviewed Mar 29, 2025

View reviewed changes

Comment thread datafusion/physical-plan/src/repartition/mod.rs Resolved

Comment thread datafusion/physical-plan/src/repartition/mod.rs Resolved

goldmedal marked this pull request as draft March 29, 2025 17:42

goldmedal commented Mar 29, 2025

View reviewed changes

goldmedal added 21 commits May 10, 2025 21:44

address review comment

b489378

fix compile and ehance doc

0f9f398

rename config

2b93d80

fix fmt

bad2e15

fix test

bc70653

fix sqllogictests

764cce7

fix fmt

c01119c

optimze repartition

057660f

fix fmt

40caf0c

fix clippy and fmt

84ebdc9

use scalar instead array

ebdab1f

fix clippy

401cb9c

fix fmt

8d7d038

rename selection_vector to selection_bitmap

e3c7021

fix sqllogictest

c7e4ac4

add sanity check for filtered batch

5c46c6b

fix doc

be34338

fix sanity check and add test

d11f0cc

fix clippy

4c83644

fix test and fmt

f76eba1

update config

0a0055d

goldmedal force-pushed the feat/selection-vector-partition branch from ab5a0e3 to 0a0055d Compare May 10, 2025 13:50

goldmedal marked this pull request as ready for review May 10, 2025 15:10

github-actions Bot added the Stale PR has not had any activity for some time label Jul 10, 2025

github-actions Bot closed this Jul 17, 2025

		# TODO: The selection vector partitioning should be used for the hash join.
		# After fix https://github.com/apache/datafusion/issues/15382

Conversation

goldmedal commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

Uh oh!

Dandandan commented Mar 26, 2025

Uh oh!

goldmedal commented Mar 27, 2025

Uh oh!

goldmedal commented Mar 27, 2025

Uh oh!

goldmedal commented Mar 27, 2025

Uh oh!

Dandandan commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

goldmedal commented Mar 28, 2025

Uh oh!

goldmedal commented Mar 29, 2025

Uh oh!

zebsme left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Dandandan commented Mar 29, 2025

Uh oh!

goldmedal Mar 29, 2025

Choose a reason for hiding this comment

Uh oh!

berkaysynnada commented Mar 29, 2025

Uh oh!

2010YOUY01 commented Mar 30, 2025

Uh oh!

goldmedal commented Mar 30, 2025

Uh oh!

goldmedal commented Mar 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 10, 2025

Uh oh!

Rich-T-kid commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

goldmedal commented Mar 25, 2025 •

edited

Loading

Dandandan commented Mar 28, 2025 •

edited

Loading

goldmedal commented Mar 30, 2025 •

edited

Loading