Skip to content
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
docs: say that the null-aware pairing drops most pairs of marked rows
A skipped check of the build row marks, and the lookup per probe row with
marks on probe rows, let some pairs of a marked row reach the join
filter. Such a pair only sets a mark that is already set.
  • Loading branch information
adriangb committed Oct 26, 2026
commit b8f1a0c43c7326599db2e1b86cdcd1c75af4ee1d
15 changes: 9 additions & 6 deletions datafusion/physical-plan/src/joins/hash_join/not_in.rs
Original file line number Diff line number Diff line change
Expand Up @@ -754,10 +754,11 @@ impl<'a> UnknownMarks<'a> {
/// The value tuples and then the join filter, if any, decide which candidates
/// count.
///
/// A row stays UNKNOWN once it is marked, so pairs that decide a row already
/// known to be UNKNOWN are skipped, and the join filter is not evaluated for
/// them. Without scope keys this also ends the pairing as soon as every row
/// that can be decided is UNKNOWN. See [`UnmarkedPairs`].
/// A row stays UNKNOWN once it is marked. So the pairing drops most pairs that
/// decide a row already known to be UNKNOWN before the join filter is
/// evaluated. Without scope keys this also ends the pairing as soon as every
/// row that can be decided is UNKNOWN. See [`UnmarkedPairs`], and
/// [`UnmarkedPairs::lookup_unmarked`] for the checks it skips.
#[expect(clippy::too_many_arguments)]
fn mark_null_candidates_for_probe_batch(
left_data: &JoinLeftData,
Expand Down Expand Up @@ -989,8 +990,10 @@ pub(super) struct LookupBuffers {
/// chunks of at most `batch_size` pairs.
///
/// `f` marks rows (see [`UnknownMarks`]), and a marked row cannot become
/// unmarked. So the pairing drops the pairs that decide a marked row before
/// it sends them, and drops the marked rows again after each chunk.
/// unmarked. So the pairing drops most pairs that decide a marked row before
/// it sends them, and drops the marked rows again after each chunk. Some
/// pairs of a marked row still reach `f` (see [`Self::lookup_unmarked`]).
/// Such a pair only sets a mark that is already set.
struct UnmarkedPairs<'a, F> {
marks: UnknownMarks<'a>,
batch_size: usize,
Expand Down
Loading