Skip to content

Commit 2506ce4

Browse files
wirybeaverkosiew
authored andcommitted
fix: preserve MERGE target qualifier bindings (apache#24429)
## Which issue does this PR close? - Part of apache#20746. - Follow-up to apache#22988. ## Rationale for this change apache#22988 deliberately rejected two valid MERGE forms to avoid silently changing expression meaning. ### Limitation 1: target-correlated subquery with an aliased target ```sql MERGE INTO target AS t USING source AS s ON EXISTS (SELECT 1 FROM source AS x WHERE x.id = t.id) WHEN MATCHED THEN DELETE; ``` `t.id` inside the subquery becomes `outer_ref(t.id)`. The old top-level canonicalizer only rewrote `Expr::Column(t.id)` to `target.id`, leaving the outer reference inconsistent with the schema later rebuilt from `DmlStatement.table_name`. ### Limitation 2: source qualifier equals the target table name ```sql MERGE INTO target AS t USING source AS target ON t.id = target.id WHEN MATCHED THEN DELETE; ``` Canonicalizing target `t.id` to `target.id` collapses both operands onto the source qualifier and can turn the condition into `target.id = target.id`. A recursive string rewrite is not safe: qualifiers are scope-local, so an inner relation can legally shadow `t`. It also cannot solve the second limitation because both relations would still have the same qualifier after rewriting. ## What changes are included in this PR? ### Solution Keep provider identity and the SQL-visible target qualifier as separate plan state: ```text SQL: MERGE INTO target AS t USING source AS target SQL binding +-------------------+ | | target table source relation provider: target qualifier: target qualifier: t | | +---------+---------+ | MERGE expression schema +-----------+-----------+ | t.* | target.* | | index 0.. | index N.. | +-----------+-----------+ | TableProvider::merge_into Logical / protobuf representation DmlStatement.table_name / DmlNode.table_name = target (provider identity) MergeIntoOp.target_qualifier / MergeIntoOpNode.target_qualifier = t (SQL-visible binding) ``` Planning now proceeds as follows: 1. Resolve `target` to the target provider, while retaining alias `t` as the visible qualifier. 2. Store `t` on `MergeIntoOp`; do not canonicalize target columns or recursively rewrite subquery plans. 3. Build one target-first expression schema (`t.*`, then source fields) and use it consistently in SQL planning, analyzer/optimizer rules, physical planning, and programmatic plans. 4. Reject only a source qualifier that collides with the *visible* target qualifier in the outer MERGE scope. A source qualifier matching the real table name remains valid when the target is aliased. 5. Pass the preserved schema and expressions to `TableProvider::merge_into`, where target and source columns resolve to distinct physical indices. ### Protobuf change and 55.0/56.0 compatibility `MergeIntoOpNode` gains optional `target_qualifier` field 3. This field is necessary because `DmlNode.table_name` contains provider identity and cannot also represent alias `t`; without it, a proto round trip loses the binding needed to resolve MERGE expressions. Compatibility is directional: - **55.0 payload -> 56.0 reader: supported.** The field is absent, so the new reader falls back to `DmlNode.table_name`, matching 55.0's canonicalized representation. - **56.0 payload -> 55.0 reader: unsupported.** A 55.0 reader ignores the unknown field but alias-preserving expressions still require it, so expressions can fail resolution or be misbound. ### Rust API compatibility DataFusion 55.0 released `MergeIntoOp` with public struct-literal construction: `MergeIntoOp { on, clauses }`. This PR makes the struct non-exhaustive, adds private target-qualifier state, and requires `MergeIntoOp::new(target_qualifier, on, clauses)`. Therefore existing 55.0 downstream struct literals will not compile unchanged against 56.0. This breaking change is documented in the 56.0 upgrade guide and is appropriate for the next major release. The PR also: - removes alias canonicalization and the recursive target-correlation guard; - documents that providers may receive residual subqueries; and - adds coverage for direct correlations, nested shadowing, lateral and LIMIT scopes, qualifier collisions, quoted/qualified identifiers, proto fallback/round trips, and physical column indices. ## Are these changes tested? - `cargo fmt --all` - `cargo clippy --all-targets --all-features -- -D warnings` - `./ci/scripts/doc_prettier_check.sh --write --allow-dirty` - `RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption` ## Are there any user-facing changes? Yes. Both valid MERGE forms above now plan successfully and reach `TableProvider::merge_into`. The Rust and protobuf compatibility constraints for upgrading from 55.0 to 56.0 are documented above. --------- Co-authored-by: kosiew <[email protected]>
1 parent bdd29f5 commit 2506ce4

20 files changed

Lines changed: 607 additions & 300 deletions

File tree

‎datafusion/core/src/physical_planner.rs‎

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -870,11 +870,9 @@ impl DefaultPhysicalPlanner {
870870
e.context(format!("MERGE INTO operation on table '{table_name}'"))
871871
})?;
872872
let input_exec = children.one()?;
873-
let target_schema = DFSchema::try_from_qualified_schema(
874-
table_name.clone(),
875-
&target.schema(),
876-
)?;
877-
let merge_schema = Arc::new(target_schema.join(input.schema())?);
873+
let merge_schema = Arc::new(
874+
merge_op.expression_schema(&target.schema(), input.schema())?,
875+
);
878876
provider
879877
.merge_into(
880878
session_state,
@@ -3642,8 +3640,8 @@ mod tests {
36423640
ctx.register_table("source", source)?;
36433641

36443642
ctx.sql(
3645-
"MERGE INTO target AS t USING source AS s ON t.id = s.id \
3646-
WHEN MATCHED AND t.id > s.id THEN DELETE",
3643+
"MERGE INTO target AS t USING source AS target ON t.id = target.id \
3644+
WHEN MATCHED AND t.id > target.id THEN DELETE",
36473645
)
36483646
.await?
36493647
.create_physical_plan()
@@ -3654,11 +3652,11 @@ mod tests {
36543652
captured.as_ref().expect("merge_into should be called");
36553653
assert_eq!(*clause_count, 1);
36563654
assert_eq!(
3657-
merge_schema.index_of_column(&Column::new(Some("target"), "id"))?,
3655+
merge_schema.index_of_column(&Column::new(Some("t"), "id"))?,
36583656
0
36593657
);
36603658
assert_eq!(
3661-
merge_schema.index_of_column(&Column::new(Some("s"), "id"))?,
3659+
merge_schema.index_of_column(&Column::new(Some("target"), "id"))?,
36623660
1
36633661
);
36643662
assert_contains!(physical_on, "index: 0");

‎datafusion/core/tests/sql/sql_api.rs‎

Lines changed: 79 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,10 @@
1616
// under the License.
1717

1818
use datafusion::prelude::*;
19-
use datafusion_common::assert_contains;
19+
use datafusion_common::tree_node::{TreeNode, TreeNodeRecursion};
20+
use datafusion_common::{TableReference, assert_contains};
21+
use datafusion_expr::dml::MergeIntoOp;
22+
use datafusion_expr::{Expr, LogicalPlan, WriteOp};
2023

2124
use tempfile::TempDir;
2225

@@ -215,93 +218,93 @@ async fn merge_into_context() -> SessionContext {
215218
ctx
216219
}
217220

218-
async fn assert_merge_sql_error(ctx: &SessionContext, sql: &str, expected: &str) {
219-
let err = ctx.sql(sql).await.unwrap_err();
220-
assert_contains!(err.strip_backtrace(), expected);
221+
async fn merge_operation(ctx: &SessionContext, sql: &str) -> Box<MergeIntoOp> {
222+
let plan = ctx.state().create_logical_plan(sql).await.unwrap();
223+
let LogicalPlan::Dml(dml) = plan else {
224+
panic!("expected MERGE DML")
225+
};
226+
let WriteOp::MergeInto(merge_op) = dml.op else {
227+
panic!("expected MERGE operation")
228+
};
229+
merge_op
221230
}
222231

223-
async fn assert_merge_physical_error(ctx: &SessionContext, sql: &str, expected: &str) {
224-
let err = ctx
225-
.sql(sql)
226-
.await
227-
.unwrap()
228-
.create_physical_plan()
229-
.await
230-
.unwrap_err();
231-
assert_contains!(err.strip_backtrace(), expected);
232-
}
233-
234-
#[tokio::test]
235-
async fn merge_into_rejects_source_alias_colliding_with_target_name() {
236-
// Canonicalizing `t.id` to `target.id` must not collapse it onto a source
237-
// that also uses `target` as its qualifier.
238-
let ctx = merge_into_context().await;
239-
240-
for target_ref in ["target", "public.target", "datafusion.public.target"] {
241-
assert_merge_sql_error(
242-
&ctx,
243-
&format!(
244-
"MERGE INTO {target_ref} AS t USING source AS target \
245-
ON t.id = target.id WHEN MATCHED THEN DELETE"
246-
),
247-
&format!(
248-
"MERGE source may not use the target table name '{target_ref}' \
249-
as a qualifier"
250-
),
251-
)
252-
.await;
253-
}
232+
fn has_outer_reference_to(expr: &Expr, qualifier: &TableReference) -> bool {
233+
let mut found = false;
234+
expr.apply(|expr| {
235+
let outer_refs = match expr {
236+
Expr::Exists(exists) => Some(&exists.subquery.outer_ref_columns),
237+
Expr::InSubquery(in_subquery) => {
238+
Some(&in_subquery.subquery.outer_ref_columns)
239+
}
240+
Expr::SetComparison(set_comparison) => {
241+
Some(&set_comparison.subquery.outer_ref_columns)
242+
}
243+
Expr::ScalarSubquery(subquery) => Some(&subquery.outer_ref_columns),
244+
_ => None,
245+
};
246+
found = outer_refs.is_some_and(|outer_refs| {
247+
outer_refs.iter().any(|expr| {
248+
matches!(
249+
expr,
250+
Expr::OuterReferenceColumn(_, column)
251+
if column.relation.as_ref() == Some(qualifier)
252+
)
253+
})
254+
});
255+
Ok(if found {
256+
TreeNodeRecursion::Stop
257+
} else {
258+
TreeNodeRecursion::Continue
259+
})
260+
})
261+
.unwrap();
262+
found
254263
}
255264

256265
#[tokio::test]
257-
async fn merge_into_rejects_subqueries_correlated_to_target_alias() {
266+
async fn merge_into_preserves_target_alias_in_correlated_subquery() {
258267
let ctx = merge_into_context().await;
259-
assert_merge_sql_error(
260-
&ctx,
261-
"MERGE INTO target AS t USING source AS s \
268+
let direct_exists = "MERGE INTO target AS t USING source AS s \
262269
ON EXISTS (SELECT 1 FROM source AS x WHERE x.id = t.id) \
263-
WHEN MATCHED THEN DELETE",
264-
"MERGE subqueries correlated to target alias 't' are not supported",
265-
)
266-
.await;
270+
WHEN MATCHED THEN DELETE";
271+
let direct_in = "MERGE INTO target AS t USING source AS s \
272+
ON t.id IN (SELECT x.id FROM source AS x WHERE x.id = t.id) \
273+
WHEN MATCHED THEN DELETE";
274+
let direct_any = "MERGE INTO target AS t USING source AS s \
275+
ON t.id = ANY (SELECT x.id FROM source AS x WHERE x.id = t.id) \
276+
WHEN MATCHED THEN DELETE";
277+
let direct_all = "MERGE INTO target AS t USING source AS s \
278+
ON t.id = ALL (SELECT x.id FROM source AS x WHERE x.id = t.id) \
279+
WHEN MATCHED THEN DELETE";
280+
let direct_scalar = "MERGE INTO target AS t USING source AS s \
281+
ON t.id = (SELECT max(x.id) FROM source AS x WHERE x.id = t.id) \
282+
WHEN MATCHED THEN DELETE";
267283

268-
// Source-correlated and uncorrelated subqueries remain supported through
269-
// logical optimization.
270284
for sql in [
271-
"MERGE INTO target AS t USING source AS s \
272-
ON EXISTS (SELECT 1 FROM source AS x WHERE x.id = s.id) \
273-
WHEN MATCHED THEN DELETE",
274-
"MERGE INTO target AS t USING source AS s \
275-
ON t.id = ANY (SELECT id FROM source) \
276-
WHEN MATCHED THEN DELETE",
285+
direct_exists,
286+
direct_in,
287+
direct_any,
288+
direct_all,
289+
direct_scalar,
277290
] {
278-
assert_merge_physical_error(&ctx, sql, "MERGE INTO not supported for Base table")
279-
.await;
291+
let merge_op = merge_operation(&ctx, sql).await;
292+
assert_eq!(merge_op.target_qualifier(), &TableReference::bare("t"));
293+
assert!(has_outer_reference_to(
294+
&merge_op.on,
295+
&TableReference::bare("t")
296+
));
280297
}
281-
}
282298

283-
#[tokio::test]
284-
async fn merge_into_requires_boolean_conditions() {
285-
let ctx = merge_into_context().await;
286-
287-
for (sql, expected) in [
288-
(
289-
"MERGE INTO target USING source ON 1 WHEN MATCHED THEN DELETE",
290-
"MERGE ON condition must be boolean type, but got Int64",
291-
),
292-
(
293-
"MERGE INTO target USING source ON true \
294-
WHEN MATCHED AND 1 THEN DELETE",
295-
"MERGE WHEN condition must be boolean type, but got Int64",
296-
),
297-
(
298-
"MERGE INTO target USING source ON NULL \
299-
WHEN MATCHED AND NULL THEN DELETE",
300-
"MERGE INTO not supported for Base table",
301-
),
302-
] {
303-
assert_merge_physical_error(&ctx, sql, expected).await;
304-
}
299+
let shadowed_correlation = "MERGE INTO target AS t USING source AS s \
300+
ON EXISTS (SELECT 1 FROM source AS t \
301+
WHERE EXISTS (SELECT 1 FROM source AS x WHERE x.id = t.id)) \
302+
WHEN MATCHED THEN DELETE";
303+
let merge_op = merge_operation(&ctx, shadowed_correlation).await;
304+
assert!(!has_outer_reference_to(
305+
&merge_op.on,
306+
&TableReference::bare("t")
307+
));
305308
}
306309

307310
#[tokio::test]

0 commit comments

Comments
 (0)