Skip to content

Null-aware NOT IN hash join in Partitioned mode returns rows when the subquery contains NULL #26496

Closed

Description

@jayzhan211

Describe the bug

An uncorrelated null-aware anti join (build.c1 NOT IN (probe.c2), JoinType::LeftAnti with null_aware = true) built with PartitionMode::Partitioned returns rows even when the probe side contains a NULL. SQL semantics say that if the subquery produces a NULL, x NOT IN (...) is never true, so the result must be empty. The same join in PartitionMode::CollectLeft returns the correct empty result.

The cause is that in Partitioned mode every output partition builds its own build side, and the "probe side saw a NULL" fact is recorded on that per-partition build data (hash_join/exec.rs around line 1935, hash_join/stream.rs around line 1366). A NULL probe key is hash-routed to one partition, so it only suppresses the build rows of that partition. The others still emit their unmatched rows.

NullAwareMode::try_new already rejects Partitioned for the correlated LeftAnti and LeftMark forms, but accepts it for the uncorrelated LeftAnti form.

To Reproduce

The DataFusion SQL planner never produces this plan: JoinSelection and the physical planner both pin null-aware joins to CollectLeft. It is reachable from plans built by hand or deserialized from protobuf (HashJoinExecNode decodes the partition mode as written).

Build side c1 = 1..8, probe side c2 = {1, NULL}, both sides RepartitionExec(Hash([c], 4)), HashJoinExec with JoinType::LeftAnti, PartitionMode::Partitioned, null_aware = true, NullEqualsNothing.

Expected behavior

No rows (as CollectLeft returns).

Actual behavior

[3, 6, 8] — the build rows hashed to partitions that did not receive the NULL probe row.

Additional context

A fix that rejects this shape at plan construction, consistent with the existing rejection for the correlated forms, is ready; I'll link the PR.

Activity

  1. cool9850311 commented on Oct 19, 2026

    @cool9850311
    Contributor

    take

  2. added a commit that references this issue on Oct 22, 2026
    509725f
  3. added a commit that references this issue on Oct 23, 2026
    0ab1fd2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions