Skip to content

Commit 954552d

Browse files
adriangbclaude
andauthored
fix: reject probe rows routed to a canceled hash join partition (apache#27092)
## Which issue does this PR close? - This PR does not close an issue. It follows up on apache#21666, which added canceled partitions to the hash join dynamic filter to fix apache#21625. ## Rationale for this change A `Partitioned` hash join cancels one of its partitions when a parent operator drops that partition's stream before it reports its build side. For example, a parent join partition whose own build side is empty drops its probe-side stream unpolled. The dynamic filter keeps every probe row routed to a canceled partition, because that partition's build keys are unknown. Keeping those rows buys nothing. A stream starts to probe only after its report is delivered, so a canceled stream has emitted no row and never will. The shared probe input still reads the rows routed to it, and `RepartitionExec` then discards them. Keeping them also costs something: - A canceled partition forces the routed `CASE` fallback instead of the merged lookup. After apache#27091 it is the only thing that does. - Under `NullEqualsNull`, a canceled partition makes every live partition keep probe-side NULL rows. ## What changes are included in this PR? Three commits: 1. `test:` Two dynamic filter tests checked range routing for `-0.0` and for composite nullable keys through a canceled partition's `CASE` branch. The commit moves both cases into `RangeRouter` tests, before the next commit removes that branch. 2. `fix:` A canceled partition counts as empty, so the filter rejects the rows routed to it. The commit rewrites the tests that asserted the old behavior. 3. `refactor:` The `CASE` fallback has no input left. The commit removes it, the per-partition filters it used, and the test assertions that a filter is not a `CASE`. When exactly one partition holds rows, the filter is still that partition's own IN list or hash lookup, so the probe scan keeps bloom filter pruning. 4. `fix:` Restores the branch for a lone non-empty partition, which the refactor first dropped. Adds a test for it and for a canceled partition next to live ones, and removes the remaining `CASE` wording. This is +155 / −345 lines. ## What is the testing strategy for this PR? - The new `RangeRouter` tests keep the routing cases that the removed tests covered. - `partitioned_canceled_and_empty_partitions_reject_every_row` and `partitioned_canceled_partition_does_not_widen_filter` cover the new behavior. - `test_partitioned_dynamic_filter_reports_empty_canceled_partitions`, the regression test for apache#21625, still passes. - All `datafusion-physical-plan` tests, all sqllogictests, and `core_integration` pass. I also checked the behavior on a query that cancels partitions. In `SELECT x.k FROM (SELECT big.k FROM mid JOIN big ON mid.k = big.k) x WHERE x.k IN (SELECT k FROM tiny)` with 4 partitions, the parent `RightSemi` join cancels two child partitions. The result is the same with and without this PR. ## Are there any user-facing changes? No. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent ce7ce07 commit 954552d

4 files changed

Lines changed: 162 additions & 350 deletions

File tree

‎datafusion/physical-plan/src/joins/hash_join/exec.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10542,8 +10542,8 @@ mod tests {
1054210542
/// Here the build-side logical NULL lives in exactly one build partition, so
1054310543
/// the `keys_have_null` flag has to survive the per-partition aggregation in
1054410544
/// `build_partitioned_filter` and widen the *routed* filter as a whole: the
10545-
/// probe NULL is routed to a single `CASE` branch, and OR-ing `IS NULL` into
10546-
/// that one branch would still drop it for every other route.
10545+
/// probe NULL is routed to a single partition, and OR-ing `IS NULL` into
10546+
/// that one partition's check would still drop it for every other route.
1054710547
#[tokio::test]
1054810548
async fn test_partitioned_null_equal_dynamic_filter_keeps_probe_nulls_for_build_logical_null()
1054910549
-> Result<()> {

‎datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -59,9 +59,8 @@ impl SeededRandomState {
5959
/// This expression computes the hash of join key columns using a specific RandomState.
6060
/// It returns a UInt64Array containing the hash values.
6161
///
62-
/// This is used for:
63-
/// - Computing routing hashes (with RepartitionExec's 0,0,0,0 seeds)
64-
/// - Computing lookup hashes (with HashJoin's 'J','O','I','N' seeds)
62+
/// The hash join does not build this expression. It exists so that
63+
/// `datafusion-proto` can still decode serialized plans that contain it.
6564
pub struct HashExpr {
6665
/// Columns to hash
6766
on_columns: Vec<PhysicalExprRef>,
@@ -354,8 +353,9 @@ impl HashTableLookupExpr {
354353
/// * `pruning_bitmap` - key-range summary covering every partition's keys
355354
/// for pruning only, or `None`
356355
///
357-
/// A single non-empty map is probed directly without routing, so it may be
358-
/// any [`Map`] and `router` is ignored. Otherwise `router` is required.
356+
/// When `maps` has exactly one entry, that map is probed directly without
357+
/// routing and `router` is ignored. Otherwise `router` is required, even if
358+
/// only one entry is non-empty.
359359
///
360360
/// # Public Only for Internal Use:
361361
/// `datafusion-proto` tests require this constructor, but it is not part of

0 commit comments

Comments
 (0)