Skip to content

REGRESSION: Multi-key null-aware join silently defaults to one NOT IN value key #26569

Closed

Description

@jayzhan211

Describe the bug

null_aware_value_keys (added in #19857) says how many of a null-aware join's keys are NOT IN value keys; the rest are correlation (scope) keys. It defaults to 1 on the physical HashJoinExecBuilder and on the logical Join (datafusion/physical-plan/src/joins/hash_join/exec.rs:530, datafusion/expr/src/logical_plan/plan.rs:4745, builder.rs:1285).

So a caller that sets only with_null_aware(true) on a join with more than one key gets "one value key plus correlation keys" without asking for it. For (a, b) NOT IN (SELECT x, y ...), the outer row (1, NULL) against the subquery row (1, 5) is UNKNOWN and must be dropped. With the count left at 1, b is treated as a correlation key, and the row is kept.

In 55.2.0 the same plan was rejected at planning time ("null_aware anti join only supports single column join key"), so the mistake used to be loud and is now silent. One caller that builds null-aware joins this way is Ballista's adaptive planner, which calls HashJoinExecBuilder::new(..).with_null_aware(..) and never sets the count (dynamic_join.rs). I derived the example above from the code and haven't run it in Ballista.

The in-repo producers (decorrelation, eliminate_join, extract_equijoin_predicate) set the count explicitly, so SQL planned by DataFusion itself is not affected.

Expected behavior

A multi-key null-aware join with no value-key count fails at planning time and asks for with_null_aware_value_keys, as 55.2.0 did. A single-key join keeps working without it.

Additional context

Both fields are new since 55, so changing the default doesn't break a released API. It would be good to settle this before branch-56 is cut.

Activity

  1. siddubakka commented on Oct 22, 2026

    @siddubakka
    Contributor

    take

  2. alamb commented on Oct 28, 2026

    @alamb
    Contributor

    I added this to the list of 56 issues

  3. alamb commented on Nov 3, 2026

    @alamb
    Contributor

    I am going through the list of items on the 56 release list and this is listed

    Is this something you still think we should get in @jayzhan211 ?

  4. changed the title [-]Multi-key null-aware join silently defaults to one NOT IN value key[/-] [+]REGRESSION: Multi-key null-aware join silently defaults to one NOT IN value key[/+] on Nov 3, 2026
  5. added
    regressionSomething that used to work no longer does
    on Nov 3, 2026
  6. viirya commented on Nov 4, 2026

    @viirya
    Member

    I'm looking at this.

  7. self-assigned this
    on Nov 4, 2026
  8. added a commit that references this issue on Nov 6, 2026
    22aedf0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingregressionSomething that used to work no longer does

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions