Repository navigation
Commit 954552d
fix: reject probe rows routed to a canceled hash join partition (apache#27092)
## Which issue does this PR close?
- This PR does not close an issue. It follows up on
apache#21666, which added canceled
partitions to the hash join dynamic filter to fix
apache#21625.
## Rationale for this change
A `Partitioned` hash join cancels one of its partitions when a parent
operator drops that partition's stream before it reports its build side.
For example, a parent join partition whose own build side is empty drops
its probe-side stream unpolled. The dynamic filter keeps every probe row
routed to a canceled partition, because that partition's build keys are
unknown.
Keeping those rows buys nothing. A stream starts to probe only after its
report is delivered, so a canceled stream has emitted no row and never
will. The shared probe input still reads the rows routed to it, and
`RepartitionExec` then discards them. Keeping them also costs something:
- A canceled partition forces the routed `CASE` fallback instead of the
merged lookup. After apache#27091 it
is the only thing that does.
- Under `NullEqualsNull`, a canceled partition makes every live
partition keep probe-side NULL rows.
## What changes are included in this PR?
Three commits:
1. `test:` Two dynamic filter tests checked range routing for `-0.0` and
for composite nullable keys through a canceled partition's `CASE`
branch. The commit moves both cases into `RangeRouter` tests, before the
next commit removes that branch.
2. `fix:` A canceled partition counts as empty, so the filter rejects
the rows routed to it. The commit rewrites the tests that asserted the
old behavior.
3. `refactor:` The `CASE` fallback has no input left. The commit removes
it, the per-partition filters it used, and the test assertions that a
filter is not a `CASE`. When exactly one partition holds rows, the
filter is still that partition's own IN list or hash lookup, so the
probe scan keeps bloom filter pruning.
4. `fix:` Restores the branch for a lone non-empty partition, which the
refactor first dropped. Adds a test for it and for a canceled partition
next to live ones, and removes the remaining `CASE` wording.
This is +155 / −345 lines.
## What is the testing strategy for this PR?
- The new `RangeRouter` tests keep the routing cases that the removed
tests covered.
- `partitioned_canceled_and_empty_partitions_reject_every_row` and
`partitioned_canceled_partition_does_not_widen_filter` cover the new
behavior.
- `test_partitioned_dynamic_filter_reports_empty_canceled_partitions`,
the regression test for
apache#21625, still passes.
- All `datafusion-physical-plan` tests, all sqllogictests, and
`core_integration` pass.
I also checked the behavior on a query that cancels partitions. In
`SELECT x.k FROM (SELECT big.k FROM mid JOIN big ON mid.k = big.k) x
WHERE x.k IN (SELECT k FROM tiny)` with 4 partitions, the parent
`RightSemi` join cancels two child partitions. The result is the same
with and without this PR.
## Are there any user-facing changes?
No.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent ce7ce07 commit 954552d
4 files changed
Lines changed: 162 additions & 350 deletions
File tree
- datafusion/physical-plan/src
- joins/hash_join
- repartition
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10542 | 10542 | | |
10543 | 10543 | | |
10544 | 10544 | | |
10545 | | - | |
10546 | | - | |
| 10545 | + | |
| 10546 | + | |
10547 | 10547 | | |
10548 | 10548 | | |
10549 | 10549 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
62 | | - | |
63 | | - | |
64 | | - | |
| 62 | + | |
| 63 | + | |
65 | 64 | | |
66 | 65 | | |
67 | 66 | | |
| |||
354 | 353 | | |
355 | 354 | | |
356 | 355 | | |
357 | | - | |
358 | | - | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
359 | 359 | | |
360 | 360 | | |
361 | 361 | | |
| |||
0 commit comments