Repository navigation
Commit 5747e87
authored
fix: null-aware anti join could emit rows after a sibling probe partition saw NULL (#25076)
## Which issue does this PR close?
<!-- No issue filed yet; found while auditing the hash join final stage.
Happy to open one first if preferred. -->
- Closes #.
## Rationale for this change
`x NOT IN (subquery)` is planned as a null-aware `LeftAnti` hash join in
`CollectLeft` mode with several probe partitions. If any probe partition
sees
a NULL key, the predicate is UNKNOWN for every build row and the join
must
return nothing.
The final stage of `HashJoinStream` read the shared "probe side saw
NULL"
flag *before* decrementing the probe-partition counter. That allowed
this
interleaving:
1. partition B finishes its probe input and reads the flag as `false`;
2. partition A processes a batch with a NULL key, sets the flag,
finishes and
decrements the counter;
3. partition B decrements the counter, becomes the last partition, and
emits
the unmatched build rows.
The result then contains rows that `NOT IN` must suppress. The counter
was
also decremented with `Ordering::Relaxed`, so nothing ordered the flag
stores
of a finishing partition before the reads of the last one, even when the
reads happened after the decrement.
## What changes are included in this PR?
- `JoinLeftData::report_probe_completed` now decrements with `AcqRel`
and
returns `Option<ProbeSideSummary>`: `Some` only for the last partition,
with
the shared flags read after the decrement. The two `AtomicBool` flags
are
private; the probe phase stores through `record_probe_batch` and the
early
exit reads a `probe_side_has_null_hint` that is documented as
best-effort.
The final stage therefore cannot read the flags before its own
decrement,
which guards the fix structurally rather than by a test that would have
to
force a few-instruction race window.
- `process_unmatched_build_batch` decrements first and hands the summary
to
the null-aware helpers. The NULL-probe rule of `LeftAnti` moved into
`null_aware_left_anti_final_indices`, so all its final-stage rules are
in
one place.
- No change for joins that are not null-aware beyond the `AcqRel`
ordering
on the counter.
## What is the testing strategy for this PR?
Two new tests in `hash_join/exec.rs`,
`test_null_aware_anti_join_probe_null_in_other_partition` and
`test_null_aware_left_mark_probe_null_in_other_partition`, run a
`CollectLeft`
join with a two-partition probe side where only partition 0 has a NULL
key,
draining the partitions in both orders. They cover the cross-partition
flag
propagation that had no test. The race itself cannot be reproduced
deterministically without a test hook, which is why the fix makes the
wrong
call order unrepresentable instead. Existing null-aware join tests and
the
join unit test suite pass unchanged.
## Are there any user-facing changes?
Wrong results from `NOT IN` under this interleaving are fixed. No API
changes;
all touched items are `pub(super)`.1 parent 2abf69d commit 5747e87
7 files changed
Lines changed: 761 additions & 53 deletions
File tree
- datafusion/physical-plan
- src/joins/hash_join
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
175 | 175 | | |
176 | 176 | | |
177 | 177 | | |
| 178 | + | |
178 | 179 | | |
179 | 180 | | |
180 | 181 | | |
| |||
322 | 323 | | |
323 | 324 | | |
324 | 325 | | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
325 | 331 | | |
326 | 332 | | |
327 | 333 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
105 | 105 | | |
106 | 106 | | |
107 | 107 | | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
108 | 127 | | |
109 | 128 | | |
110 | 129 | | |
| |||
0 commit comments