Repository navigation
Commit d040501
fix: Derive Substrait intersection nullability from every input (#25091)
## Which issue does this PR close?
- Closes #25042.
## Rationale for this change
The Substrait consumer derived all three intersection schemas from the
primary
input alone, so a field the intersection makes required stayed nullable
in the
logical output schema.
The [Set Operation rules] give intersections a different rule: for the
multiset
intersections a field is required as soon as any input requires it, and
for
`INTERSECTION_PRIMARY` it is nullable only when it is nullable in the
primary
input and in at least one secondary input.
`from_set_rel` builds intersections with
`LogicalPlanBuilder::intersect`, which
compiles to a left semi join and therefore keeps the left input's
nullability.
[Set Operation rules]:
https://substrait.io/relations/logical_relations/#set-operation
## What changes are included in this PR?
Intersections now go through a small helper that gives the result
`left AND right` nullability per field.
The join matches nulls with nulls (`NullEquality::NullEqualsNull`) on
every
field, so a left row holding a null in some field only survives when the
right
input holds a null there too — a field is nullable in the result only
when both
inputs make it nullable. Applied to each step, that single rule
reproduces both
spec rules:
- the multiset intersections chain pairwise, so the result is required
when any
input requires it;
- for `INTERSECTION_PRIMARY` the right side is the union of the
secondary
inputs, whose field is nullable exactly when some secondary makes it
nullable,
which yields "nullable in the primary and in at least one secondary".
How the narrowing is built matters, because it has to hold in the
physical plan
too, not only in the logical schema. When the right input requires a
field that
the left input leaves nullable, the intersection is built as an inner
join
(nulls equal nulls) against the distinct right rows, and that field is
read from
the right side:
- matched rows hold equal values, so the result rows are unchanged;
- the field is non-nullable because its source column is, and the
logical and
the physical planner both derive that from the input schema, so the
logical
schema, the physical plan and the batches agree without any schema
override;
- joining against distinct right rows keeps each left row at most once,
as the
semi join does (and the left side is still made distinct unless
`is_all`).
A field is only read from the right side when its type matches the
left's; its
metadata does not matter, since metadata is no part of nullability. The
result
should still describe the left input, so where the inputs' metadata
disagree the
left's wins: a column read from the right is aliased with the left
field's
metadata, and an inner join's schema lets the left input's schema
metadata win
(in the logical and in the physical plan alike). Keys that only the
right input
carries, on a field or on the schema, are merged into the result.
When nothing needs narrowing, the plan is exactly the one
`LogicalPlanBuilder::intersect` builds today, so the common all-nullable
case is
untouched. Unions and the `MINUS` operations are not affected.
## What is the testing strategy for this PR?
New test `intersect_nullability` in
`datafusion/substrait/tests/cases/logical_plans.rs`,
with three plans added under `tests/testdata/test_plans/`. They
intersect three
tables carrying the same six columns (`?` marks nullable, `~` marks
`NULLABILITY_UNSPECIFIED`, which the consumer reads as nullable):
```
primary a? b? c? d? e? f?
secondary a b c? d? e~ f~
secondary a b? c d? e? f
```
| Operation | Result |
| --- | --- |
| `INTERSECTION_PRIMARY` | `a, b?, c?, d?, e?, f?` |
| `INTERSECTION_MULTISET` | `a, b, c, d?, e?, f` |
| `INTERSECTION_MULTISET_ALL` | `a, b, c, d?, e?, f` |
Columns `e` and `f` pin the unspecified-nullability boundary: if it were
read as
required, `e` would come out required for the multiset intersections and
`f`
for `INTERSECTION_PRIMARY`.
The tables are given rows, including nulls, and for each plan the test
checks
that the logical schema, the physical plan's schema and the schema of
every
collected batch are equal, and checks the result rows. It fails on
`main`
(`a?` where `a` is expected), and the physical-schema check also fails
against
a narrowing done with a logical-only `Projection`.
Each plan also runs a second time over tables with differing schema and
field
metadata: the secondary tables share their metadata, which differs from
the
primary's and adds a key the primary lacks. That run checks that the
primary's
metadata wins on the keys the tables disagree on, and that the logical
schema,
the physical plan's schema and the collected batches' schemas agree. It
fails
without the metadata handling (`a?` where `a` is expected).
The secondary tables share their metadata on purpose: a `Union` of
inputs with
differing field metadata can report different metadata in its logical
and in its
physical schema. That is independent of this change (plain SQL,
`SELECT a FROM t1 UNION ALL SELECT a FROM t2`, shows it on a checkout
without
it), and I have not root-caused it.
With the spec's full input pattern from the issue (`R` = required,
`N` = nullable), both the logical and the physical schema now match the
expected column:
```
primary: R R R R N N N N
secondary 1: R R N N R R N N
secondary 2: R N R N R N R N
INTERSECTION_PRIMARY R R R R R N N N
INTERSECTION_MULTISET R R R R R R R N
INTERSECTION_MULTISET_ALL R R R R R R R N
```
The existing `datafusion-substrait` suite passes unchanged, including
the
intersection roundtrip tests, and a Substrait round trip of the
sqllogictest
files that use `INTERSECT` gives the same output as `main`.
## Are there any user-facing changes?
Intersections consumed from Substrait now report a narrower,
spec-conforming
nullability. No public API changes.
One note for reviewers: the same narrowing would apply to SQL
`INTERSECT`, since
`LogicalPlanBuilder::intersect` keeps the left nullability for every
caller. I
kept this change inside the Substrait consumer to match the scope of the
issue
and to avoid changing SQL plans in the same PR. If you would rather see
the rule
live in `LogicalPlanBuilder`, I am happy to move it.
---------
Co-authored-by: Claude Sonnet 5 <[email protected]>1 parent d1790f2 commit d040501
5 files changed
Lines changed: 955 additions & 9 deletions
File tree
- datafusion/substrait
- src/logical_plan/consumer/rel
- tests
- cases
- testdata/test_plans
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
20 | | - | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
21 | 25 | | |
22 | 26 | | |
23 | 27 | | |
| |||
31 | 35 | | |
32 | 36 | | |
33 | 37 | | |
34 | | - | |
| 38 | + | |
35 | 39 | | |
36 | 40 | | |
37 | 41 | | |
| |||
77 | 81 | | |
78 | 82 | | |
79 | 83 | | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
| 84 | + | |
85 | 85 | | |
86 | 86 | | |
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
90 | 215 | | |
91 | 216 | | |
92 | 217 | | |
| |||
0 commit comments