Skip to content

Commit 3c11ef9

Browse files
Document that the intersection's schema-level metadata fix does not survive arbitrary optimizer rewrites, and test what actually holds
Restores the left input's schema-level metadata (not just field-level, already fixed) on the projection intersect_rel returns, matching LogicalPlanBuilder::intersect. But LogicalPlan::recompute_schema rebuilds a Join's or a Projection's schema-level metadata from its (possibly rewritten) children whenever the optimizer decides anything in the plan changed, discarding that fix - confirmed even for the simplest two-input case, no union or join filter involved. There is no supported way from this crate to opt a join's or a projection's schema-level metadata out of that recomputation; field-level metadata is unaffected, since it travels with the Field objects through any rebuild. Updated the regression test to check the complete schema metadata map (not just one key) on the plan as returned, documented as a right-after-conversion guarantee, and to compare the physical plan and batches against the optimized logical schema instead of the pre-optimization one, since that is what physical planning actually consumes. Field-level metadata and nullability are asserted at every stage and are unaffected by this limitation.
1 parent 2a8d8f5 commit 3c11ef9

2 files changed

Lines changed: 132 additions & 28 deletions

File tree

‎datafusion/substrait/src/logical_plan/consumer/rel/set_rel.rs‎

Lines changed: 91 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,10 @@
1717

1818
use crate::logical_plan::consumer::SubstraitConsumer;
1919
use datafusion::arrow::datatypes::Field;
20-
use datafusion::common::{JoinType, NullEquality, not_impl_err, substrait_err};
20+
use datafusion::common::{DFSchema, JoinType, NullEquality, not_impl_err, substrait_err};
2121
use datafusion::logical_expr::{
22-
Cast, Expr, LogicalPlan, LogicalPlanBuilder, requalify_sides_if_needed,
22+
Cast, Expr, LogicalPlan, LogicalPlanBuilder, Projection, projection_schema,
23+
requalify_sides_if_needed,
2324
};
2425
use std::sync::Arc;
2526
use substrait::proto::set_rel::SetOp;
@@ -113,15 +114,43 @@ async fn intersect_rels(
113114
/// semi join does.
114115
///
115116
/// Differing metadata does not change which path is taken, as it is no part of
116-
/// nullability. The result should describe the left input, exactly as
117-
/// [`LogicalPlanBuilder::intersect`] does: a column read from the right is cast
118-
/// to an explicit target field carrying the left field's metadata, and an
119-
/// explicit-field cast target replaces the source's metadata outright rather
120-
/// than merging into it, so a key only the right input carries is dropped
121-
/// rather than surviving into the result. A column read from the left is
122-
/// already exactly the left input's own field, in the logical and in the
123-
/// physical plan alike, so it needs no such treatment. The physical plan and
124-
/// the batches carry the same metadata as the logical plan.
117+
/// nullability. The result describes the left input's metadata exactly at the
118+
/// field level, and does so at the schema level for the plan this function
119+
/// returns, though not necessarily after later rewrites; see the two notes
120+
/// below.
121+
///
122+
/// At the field level: a column read from the right is cast to an explicit
123+
/// target field carrying the left field's metadata, and an explicit-field
124+
/// cast target replaces the source's metadata outright rather than merging
125+
/// into it, so a key only the right input's field carries is dropped rather
126+
/// than surviving into the result. A column read from the left is already
127+
/// exactly the left input's own field, so it needs no such treatment. This is
128+
/// robust: a `Field`'s own metadata travels with it through any later
129+
/// rewrite, since a rebuilt schema is still assembled from the same `Field`s.
130+
///
131+
/// At the schema level: [`build_join_schema`](datafusion::logical_expr::build_join_schema)
132+
/// chains the two inputs' schema metadata maps (right's entries, then
133+
/// left's), so a conflicting key resolves to the left's value but a key
134+
/// present only on the right's schema survives, and [`projection_schema`]
135+
/// copies that merged map verbatim onto a projection built directly on top
136+
/// of the join. The projection this function returns is therefore built with
137+
/// an explicit schema instead, whose metadata is the left input's own schema
138+
/// metadata, unmodified - correct for the plan as returned here, and for a
139+
/// caller that reads its schema without executing it.
140+
///
141+
/// That schema-level fix is not preserved through arbitrary later rewrites,
142+
/// unlike the field-level one. `LogicalPlan::recompute_schema` rebuilds a
143+
/// `Join`'s or a `Projection`'s schema-level metadata from its (possibly
144+
/// rewritten) children's *current* schema-level metadata, discarding
145+
/// whatever this function set. DataFusion's optimizer calls it whenever it
146+
/// judges anything in the plan changed, including something unrelated
147+
/// elsewhere in the same query, so by the time this plan reaches physical
148+
/// planning its schema-level metadata may again be the join's merged one
149+
/// rather than the left input's. This holds even for the simplest two-input,
150+
/// non-correlated case, with no join filter or `INTERSECTION_PRIMARY` union
151+
/// involved. There is currently no supported way from this crate to opt a
152+
/// join's or a projection's schema-level metadata out of that
153+
/// recomputation.
125154
///
126155
/// [Set Operation rules]: https://substrait.io/relations/logical_relations/#set-operation
127156
fn intersect_rel(
@@ -193,15 +222,57 @@ fn intersect_rel(
193222
})
194223
.collect::<Vec<_>>();
195224

196-
left.join_detailed(
197-
right,
198-
JoinType::Inner,
199-
(left_columns, right_columns),
200-
None,
201-
NullEquality::NullEqualsNull,
202-
)?
203-
.project(exprs)?
204-
.build()
225+
// Captured before `left` is consumed by `join_detailed` below: this is
226+
// the metadata the projection's schema must end up with, not the join's.
227+
let left_schema_metadata = left.schema().metadata().clone();
228+
229+
let joined = left
230+
.join_detailed(
231+
right,
232+
JoinType::Inner,
233+
(left_columns, right_columns),
234+
None,
235+
NullEquality::NullEqualsNull,
236+
)?
237+
.build()?;
238+
239+
// `projection_schema` would copy the join's own schema metadata, which
240+
// (per `build_join_schema`) can carry a key that only the right side's
241+
// schema has. Build the fields the normal way, then swap in the left
242+
// input's schema metadata so the projection's schema, like its fields,
243+
// describes the left input exactly.
244+
//
245+
// This is correct for the plan `intersect_rel` returns, but it is not
246+
// preserved through arbitrary later rewrites: `LogicalPlan::Join` and
247+
// `LogicalPlan::Projection` both recompute their schema's metadata from
248+
// their (possibly rewritten) children whenever `recompute_schema` runs on
249+
// them (see its match arms in `datafusion/expr/src/logical_plan/plan.rs`),
250+
// which happens whenever the surrounding optimizer decides anything in
251+
// the plan changed - even something unrelated elsewhere in the same
252+
// query. There is no supported way from this crate to make a join's or a
253+
// projection's schema-level metadata (as opposed to per-field metadata,
254+
// which is carried on the `Field`s themselves and does survive) opt out
255+
// of that recomputation. See the discussion on this PR for confirmation
256+
// this is a general limitation, not specific to this construction.
257+
let projected_schema = projection_schema(&joined, &exprs)?;
258+
let schema = Arc::new(
259+
DFSchema::new_with_metadata(
260+
projected_schema
261+
.iter()
262+
.map(|(qualifier, field)| (qualifier.cloned(), Arc::clone(field)))
263+
.collect(),
264+
left_schema_metadata,
265+
)?
266+
.with_functional_dependencies(
267+
projected_schema.functional_dependencies().clone(),
268+
)?,
269+
);
270+
271+
Ok(LogicalPlan::Projection(Projection::try_new_with_schema(
272+
exprs,
273+
Arc::new(joined),
274+
schema,
275+
)?))
205276
}
206277

207278
async fn except_rels(

‎datafusion/substrait/tests/cases/logical_plans.rs‎

Lines changed: 41 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -394,13 +394,26 @@ mod tests {
394394
if tagged {
395395
// Where the inputs disagree, the primary table's metadata wins,
396396
// also on the columns read from a secondary input. Comparing
397-
// the complete map, not just the "column" key, catches a
397+
// the complete map, not just the "table" key, catches a
398398
// secondary-only key ("only_in_secondary") leaking through:
399-
// the primary table never carries that key, so its presence
400-
// would fail this exact-equality check.
399+
// the primary table's own schema metadata never carries that
400+
// key, so its presence would fail this exact-equality check.
401+
//
402+
// This holds for the plan as `from_substrait_plan` returns it,
403+
// right after conversion. It is a schema-level (not per-field)
404+
// property, and DataFusion's optimizer can recompute a join's
405+
// or a projection's schema-level metadata from its children
406+
// whenever it decides anything in the plan changed - even
407+
// something unrelated - which can reintroduce
408+
// "only_in_secondary". See `intersect_rel`'s doc comment. The
409+
// per-field metadata checked below is not affected by that and
410+
// is asserted again on the optimized, physical and batch
411+
// schemas.
412+
let expected_schema_metadata =
413+
HashMap::from([("table".to_string(), "data".to_string())]);
401414
assert_eq!(
402-
plan.schema().metadata().get("table").map(String::as_str),
403-
Some("data"),
415+
plan.schema().metadata(),
416+
&expected_schema_metadata,
404417
"schema metadata of {file}"
405418
);
406419
for field in plan.schema().fields() {
@@ -418,9 +431,29 @@ mod tests {
418431
}
419432

420433
// The physical plan and the batches it produces must carry the same
421-
// schema as the logical plan, not the left input's nullability
422-
let logical_schema = Arc::clone(plan.schema().inner());
423-
let df = DataFrame::new(ctx.state(), plan);
434+
// schema as the *optimized* logical plan: physical planning runs on
435+
// the optimizer's output, not on `plan` as `from_substrait_plan`
436+
// returned it, and (per the doc comment above) the optimizer is not
437+
// guaranteed to preserve schema-level metadata that this function
438+
// set. Field-level metadata and nullability are unaffected and are
439+
// checked here too, so a regression in either still fails this.
440+
let optimized = ctx.state().optimize(&plan)?;
441+
let logical_schema = Arc::clone(optimized.schema().inner());
442+
if tagged {
443+
for field in logical_schema.fields() {
444+
let expected_metadata = HashMap::from([(
445+
"column".to_string(),
446+
format!("data.{}", field.name()),
447+
)]);
448+
assert_eq!(
449+
field.metadata(),
450+
&expected_metadata,
451+
"optimized metadata of column {} of {file}",
452+
field.name()
453+
);
454+
}
455+
}
456+
let df = DataFrame::new(ctx.state(), optimized);
424457
let physical_plan = df.clone().create_physical_plan().await?;
425458
assert_eq!(
426459
physical_plan.schema(),

0 commit comments

Comments
 (0)