Repository navigation
Commit 28f8113
authored
test: JoinScan NOT IN null-safety regress (requires patched datafusion-proto) (#5006)
## Summary
Adds `joinscan_notin_null_safety.sql` -- a multi-table regress test that
exercises the JoinScan path's `NOT IN` handling against a NULL-bearing
inner column. Asserts the result is 0 rows (matching SQL `NOT IN`
three-valued logic -- any NULL in the inner key column poisons NOT IN
under WHERE) and that the EXPLAIN does not show a swapped `RightAnti`
(which would lose null-aware semantics).
This PR is a regress test only. The pg_search-side null_aware code that
makes the test return the correct answer is supplied by sister PR #5005
-- that PR adds the `null_aware` field on `JoinNode`, lifts `NOT IN`
SubPlans to `JoinType::Anti` with `null_aware=true` in
`wrap_with_semi_anti`, and routes through
`LogicalPlanBuilder::join_detailed_with_options(.., null_aware=true)` in
the translator.
## Why this regress test needs a patched `datafusion-proto`
DataFusion fully implements null-aware NOT IN -- `LogicalPlan::Join` has
a `null_aware: bool` field, the optimizer routes NOT IN with nullable
join keys through it, the physical planner threads it down to
`HashJoinExec`, and `HashJoinExec` tracks `probe_side_has_null` to emit
zero rows when needed. All of this was added in
[apache/datafusion#19635](apache/datafusion#19635)
and ships in DataFusion 53.1.0.
But that PR missed one symmetric proto change: `null_aware` was added to
the **physical** `HashJoinExecNode` proto in `datafusion-proto` but
**not** to the **logical** `JoinNode` proto. The encoder destructures
`LogicalPlan::Join` with `..` and silently drops `null_aware` during
serialization.
```protobuf
// datafusion-proto-53.1.0/proto/datafusion.proto:244 (line is the JoinNode for the LogicalPlan)
message JoinNode {
LogicalPlanNode left = 1;
LogicalPlanNode right = 2;
// ... 6 more fields ...
// null_aware MISSING -- should be `bool null_aware = 9;`
}
```
```rust
// datafusion-proto-53.1.0/src/logical_plan/mod.rs:1316
LogicalPlan::Join(Join {
left, right, on, filter, join_type, join_constraint, null_equality,
.. // <- `null_aware` falls into "the rest" and is silently dropped
}) => { /* serialize without null_aware */ }
```
**pg_search's JoinScan path serializes its `LogicalPlan` to bytes** for
Postgres' parallel-execution leader/worker IPC (see
`pg_search/src/scan/codec.rs::serialize_logical_plan`), so the
round-trip drops the flag and reverts to the silent-wrong-answer
behavior. The agg-on-join path (covered by #5005) does NOT serialize the
LogicalPlan -- it executes in-process -- and is unaffected.
Empirically demonstrated:
| Configuration | Rows returned |
|---|---|
| Pure PG (correct ground truth) | 0 |
| pg_search JoinScan ON, no proto patch | 89 (wrong -- `LeftAnti`
semantics, not `NOT IN`) |
| pg_search JoinScan ON, patched datafusion-proto | 0 (correct) |
## What the test covers
The query is intentionally minimal -- single outer table, single inner
list table with a NULL-bombed exclude set, single-column equi-key --
chosen to exercise:
1. JoinScan accepts the lifted `NOT IN` SubPlan (via #5005's
`wrap_with_semi_anti` change).
2. JoinScan's translator routes `null_aware=true` through
`LogicalPlanBuilder::join_detailed_with_options(.., null_aware=true)`.
3. JoinScan serializes the LogicalPlan via
`datafusion_proto::bytes::logical_plan_to_bytes_with_extension_codec`
for IPC.
4. The patched `datafusion-proto` preserves `null_aware` through
round-trip.
5. DataFusion's physical `HashJoinExec` runs with `null_aware=true`,
refuses the `LeftAnti`->`RightAnti` swap (guarded in `join_selection`),
and returns 0 rows when the probe side has NULL.
## Dependencies (must land before this PR's CI passes)
1. **#5005** -- adds the pg_search-side `null_aware` plumbing and
`wrap_with_semi_anti` lift.
2. **A published patched `datafusion-proto`** -- for example a
`paradedb/datafusion-proto` fork containing the missing `null_aware`
field on `LogicalPlan::JoinNode` plus the symmetric `to_proto` /
`from_proto` code. Pinned via `[patch.crates-io]` in the workspace
`Cargo.toml`.
3. **An upstream issue / PR at apache/datafusion** as a follow-up to
#19635 -- to add the missing field to the logical Join proto. Filed as
[apache/datafusion#22065](apache/datafusion#22065).
Once merged and released, the `[patch.crates-io]` entry can be dropped.
The proto patch itself is small (~10 lines): add `bool null_aware = 9`
to `message JoinNode`, propagate `null_aware` through `to_proto`'s
destructure, restore via `Join::try_new(.., null_aware: true)` in
`from_proto` when set. Local fork at `/tmp/datafusion-fork` works
empirically end-to-end; needs to be hosted somewhere reachable for CI.
## What CI will look like
| State | CI result |
|---|---|
| #5005 not merged, no proto patch | Test returns `89` (wrong); CI
fails. |
| #5005 merged, no proto patch | Test returns `89` (wrong); CI fails. |
| #5005 merged, proto patch in `Cargo.toml` `[patch.crates-io]` | Test
returns `0` (correct); CI passes. |
This PR is in the third state on the author's machine. To merge to main,
the `[patch.crates-io]` `Cargo.toml` change needs to ship as a follow-up
commit on this branch once the fork is published.
Refs #4911 -- JoinScan NOT IN portion. The agg-on-join NOT IN portion is
in #5005.1 parent fb90b2b commit 28f8113
2 files changed
Lines changed: 240 additions & 0 deletions
File tree
- pg_search/tests/pg_regress
- expected
- sql
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
0 commit comments