refactor: simplify null-aware and join-type branching in hash join - #24957

…tion saw NULL

RE-IMPLEMENTATION, not a faithful cherry-pick. apache#25076
(5747e87) does not apply here: this fork's base
predates the null-aware branching refactor (apache#24957, `NullAwareMode`) and the
chunked final emission (apache#25028, `EmitUnmatchedBuildRowsState`), and
`git show 5747e87 | git apply --check` fails on `exec.rs`, `stream.rs` and the
three manifests. The semantics below are the upstream ones; the shape is this
fork's.

The bug: the final stage read the shared "probe side saw NULL" flag and only
then decremented the probe-partition counter, so a partition could read
`false`, let a sibling record a NULL and finish, then become the last partition
itself and emit build rows that `NOT IN` must suppress. The counter also
decremented with `Relaxed`, so nothing ordered a sibling's store before the
last partition's load.

Changes:

- New `hash_join/probe_completion.rs`, ported nearly verbatim: `ProbeCompletion`
  owns the probe-partition counter and the two shared flags,
  `report_completed()` decrements with `AcqRel` and returns a
  `ProbeSideSummary` only to the last partition, so the final stage cannot read
  the flags before its own decrement. Its sequential unit tests and the
  `--cfg datafusion_loom` model-checked tests come along with the workspace
  `loom` dependency and the `cfg(datafusion_loom)` check-cfg registration.
- `JoinLeftData` holds one `ProbeCompletion` instead of
  `probe_threads_counter`/`probe_side_non_empty`/`probe_side_has_null`. The
  probe phase records through `record_probe_batch()` and keeps a best-effort
  `probe_side_has_null_hint()` early exit.
- `process_unmatched_build_batch` decrements first and hands the resulting
  summary to the new `null_aware_left_anti_final_indices`, which holds the
  `LeftAnti` rules: a probe NULL seen by any partition suppresses every build
  row; otherwise a NULL build key is dropped unless the probe side was empty.
- This fork has no null-aware `LeftMark`, so the mark half of the upstream PR
  has no counterpart here.

Tests (`hash_join/exec.rs`):
- `test_null_aware_anti_join_probe_null_in_other_partition`: two probe
  partitions, only partition 0 sees the NULL, drained in both finishing orders;
  no rows may be emitted.
- `test_null_aware_anti_join_probe_logical_null_in_other_partition`: the same
  with a dictionary key whose NULL is logical only.
- `test_null_aware_anti_join_empty_probe_keeps_null_build_key`: pins
  `NULL NOT IN (empty) = TRUE` (added here; upstream covers it via slt only).

(backport of apache#25076, 5747e87)

Signed-off-by: discord9 <[email protected]>
(cherry picked from commit 3fb9a3084e39781b90d5dbffa01f2664f64dcc2d)

…results

RE-IMPLEMENTATION, not a faithful cherry-pick. apache#25339
(95bb0a0) builds on the correlated null-aware
machinery of apache#21585/apache#24957 (correlation scope maps, `NullAwareMode`,
`null_indices_bitmap`, shared null-mark buffers), none of which exists here;
`git show 95bb0a0 | git apply --check` fails on `exec.rs`, `stream.rs` and
`join_selection.rs`, and it also edits benchmark files this fork does not have.

The bug: a correlated `NOT IN` whose correlation cannot become an equi-join key
leaves a residual join filter, and the executor ignored that filter when
deciding whether a NULL on the subquery side makes `NOT IN` UNKNOWN, so a NULL
the filter excludes still poisoned every outer row.

Same semantics for the shapes this fork can plan (`on[0]` is the `NOT IN` value
key and the residual correlation lives in the join filter; the null-aware
validation still requires a single join key, where upstream also accepted
`on[1..]` scope keys):

- `JoinLeftData` gains the shared `null_indices_bitmap` of build rows whose
  `NOT IN` is UNKNOWN, allocated with the build side for correlated joins.
- A correlated null-aware join (one with a join filter) no longer lets a probe
  NULL poison the whole join: `mark_null_candidates_for_probe_batch` records,
  per build row, whether a NULL that the residual filter keeps makes `NOT IN`
  UNKNOWN - case 1, a NULL build key with a probe row that passes the filter;
  case 2, a probe row with a NULL key that passes the filter. Without scope
  keys both cases are cross products, already-marked build rows are skipped,
  and the filter is not re-evaluated for them.
- The final stage drops the marked build rows instead of consulting the
  probe-side flags, so outer rows whose subquery is empty keep
  `NOT IN (empty) = TRUE`.
- The uncorrelated probe-side NULL check now uses logical nulls too, so
  dictionary keys are covered on both paths.

Tests:
- `hash_join/exec.rs`: `test_null_aware_left_anti_filter_only` (a `LeftAnti`
  with a second-column `>` filter and no scope keys; only the rows with an
  empty subquery result are emitted), from the upstream PR.
- `sqllogictest/test_files/null_aware_anti_join.slt`: new correlated-residual
  section - the query from the upstream issue, an `EXPLAIN` assertion that the
  shape stays `LeftAnti ... filter=..., null_aware` on a single equi key, and
  the case where the residual leaves the subquery empty for every outer row.

(backport of apache#25339, 95bb0a0)

Signed-off-by: discord9 <[email protected]>
(cherry picked from commit 163cc1d5fb8a8565825a13ebfc82c010bc8750fa)