Skip to content

Commit 5747e87

Browse files
authored
fix: null-aware anti join could emit rows after a sibling probe partition saw NULL (#25076)
## Which issue does this PR close? <!-- No issue filed yet; found while auditing the hash join final stage. Happy to open one first if preferred. --> - Closes #. ## Rationale for this change `x NOT IN (subquery)` is planned as a null-aware `LeftAnti` hash join in `CollectLeft` mode with several probe partitions. If any probe partition sees a NULL key, the predicate is UNKNOWN for every build row and the join must return nothing. The final stage of `HashJoinStream` read the shared "probe side saw NULL" flag *before* decrementing the probe-partition counter. That allowed this interleaving: 1. partition B finishes its probe input and reads the flag as `false`; 2. partition A processes a batch with a NULL key, sets the flag, finishes and decrements the counter; 3. partition B decrements the counter, becomes the last partition, and emits the unmatched build rows. The result then contains rows that `NOT IN` must suppress. The counter was also decremented with `Ordering::Relaxed`, so nothing ordered the flag stores of a finishing partition before the reads of the last one, even when the reads happened after the decrement. ## What changes are included in this PR? - `JoinLeftData::report_probe_completed` now decrements with `AcqRel` and returns `Option<ProbeSideSummary>`: `Some` only for the last partition, with the shared flags read after the decrement. The two `AtomicBool` flags are private; the probe phase stores through `record_probe_batch` and the early exit reads a `probe_side_has_null_hint` that is documented as best-effort. The final stage therefore cannot read the flags before its own decrement, which guards the fix structurally rather than by a test that would have to force a few-instruction race window. - `process_unmatched_build_batch` decrements first and hands the summary to the null-aware helpers. The NULL-probe rule of `LeftAnti` moved into `null_aware_left_anti_final_indices`, so all its final-stage rules are in one place. - No change for joins that are not null-aware beyond the `AcqRel` ordering on the counter. ## What is the testing strategy for this PR? Two new tests in `hash_join/exec.rs`, `test_null_aware_anti_join_probe_null_in_other_partition` and `test_null_aware_left_mark_probe_null_in_other_partition`, run a `CollectLeft` join with a two-partition probe side where only partition 0 has a NULL key, draining the partitions in both orders. They cover the cross-partition flag propagation that had no test. The race itself cannot be reproduced deterministically without a test hook, which is why the fix makes the wrong call order unrepresentable instead. Existing null-aware join tests and the join unit test suite pass unchanged. ## Are there any user-facing changes? Wrong results from `NOT IN` under this interleaving are fixed. No API changes; all touched items are `pub(super)`.
1 parent 2abf69d commit 5747e87

7 files changed

Lines changed: 761 additions & 53 deletions

File tree

‎Cargo.lock‎

Lines changed: 48 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -175,6 +175,7 @@ itertools = "0.15"
175175
itoa = "1.0"
176176
liblzma = { version = "0.4.6", features = ["static"] }
177177
log = "^0.4"
178+
loom = "0.7"
178179
memchr = "2.8.1"
179180
num-traits = { version = "0.2" }
180181
object_store = { version = "0.13.2", default-features = false }
@@ -322,6 +323,11 @@ unexpected_cfgs = { level = "warn", check-cfg = [
322323
'cfg(datafusion_coop, values("tokio", "tokio_fallback", "per_stream"))',
323324
"cfg(coverage)",
324325
"cfg(coverage_nightly)",
326+
# Enables the loom model-checked concurrency tests; see
327+
# datafusion/physical-plan/src/joins/hash_join/probe_completion.rs.
328+
# Namespaced rather than a bare `loom`, which tokio also reads and which
329+
# would switch tokio itself into its loom build.
330+
"cfg(datafusion_loom)",
325331
] }
326332
unused_qualifications = "deny"
327333

‎datafusion/physical-plan/Cargo.toml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,25 @@ tokio = { workspace = true, features = [
105105
"parking_lot",
106106
] }
107107

108+
# Model checker for the hash join probe-completion protocol; see
109+
# joins/hash_join/probe_completion.rs.
110+
#
111+
# Gated on `--cfg datafusion_loom` rather than declared a plain dev-dependency: loom pulls
112+
# in tracing-subscriber, which turns on `regex-automata`'s DFA features for
113+
# every crate in the workspace build. That is not inert. It changes which
114+
# engine the shared `regex` picks and costs enough extra stack to overflow
115+
# unrelated deep-recursion tests (`sql::unparser::test_tpch_unparser_roundtrip`
116+
# aborts in CI). Keeping loom behind its own cfg leaves the ordinary build
117+
# untouched.
118+
#
119+
# Run the model checks with:
120+
# RUSTFLAGS="--cfg datafusion_loom" cargo test -p datafusion-physical-plan --lib loom_tests
121+
#
122+
# The cfg is namespaced because a bare `loom` is also read by tokio, which
123+
# would switch tokio into its own loom build and drop `tokio::fs`.
124+
[target.'cfg(datafusion_loom)'.dev-dependencies]
125+
loom = { workspace = true }
126+
108127
[[bench]]
109128
harness = false
110129
name = "partial_ordering"

0 commit comments

Comments
 (0)