Repository navigation
Commit 2506ce4
fix: preserve MERGE target qualifier bindings (apache#24429)
## Which issue does this PR close?
- Part of apache#20746.
- Follow-up to apache#22988.
## Rationale for this change
apache#22988 deliberately rejected two valid MERGE forms to avoid silently
changing expression meaning.
### Limitation 1: target-correlated subquery with an aliased target
```sql
MERGE INTO target AS t
USING source AS s
ON EXISTS (SELECT 1 FROM source AS x WHERE x.id = t.id)
WHEN MATCHED THEN DELETE;
```
`t.id` inside the subquery becomes `outer_ref(t.id)`. The old top-level
canonicalizer only rewrote `Expr::Column(t.id)` to `target.id`, leaving
the outer reference inconsistent with the schema later rebuilt from
`DmlStatement.table_name`.
### Limitation 2: source qualifier equals the target table name
```sql
MERGE INTO target AS t
USING source AS target
ON t.id = target.id
WHEN MATCHED THEN DELETE;
```
Canonicalizing target `t.id` to `target.id` collapses both operands onto
the source qualifier and can turn the condition into `target.id =
target.id`.
A recursive string rewrite is not safe: qualifiers are scope-local, so
an inner relation can legally shadow `t`. It also cannot solve the
second limitation because both relations would still have the same
qualifier after rewriting.
## What changes are included in this PR?
### Solution
Keep provider identity and the SQL-visible target qualifier as separate
plan state:
```text
SQL: MERGE INTO target AS t USING source AS target
SQL binding
+-------------------+
| |
target table source relation
provider: target qualifier: target
qualifier: t
| |
+---------+---------+
|
MERGE expression schema
+-----------+-----------+
| t.* | target.* |
| index 0.. | index N.. |
+-----------+-----------+
|
TableProvider::merge_into
Logical / protobuf representation
DmlStatement.table_name / DmlNode.table_name
= target (provider identity)
MergeIntoOp.target_qualifier /
MergeIntoOpNode.target_qualifier
= t (SQL-visible binding)
```
Planning now proceeds as follows:
1. Resolve `target` to the target provider, while retaining alias `t` as
the visible qualifier.
2. Store `t` on `MergeIntoOp`; do not canonicalize target columns or
recursively rewrite subquery plans.
3. Build one target-first expression schema (`t.*`, then source fields)
and use it consistently in SQL planning, analyzer/optimizer rules,
physical planning, and programmatic plans.
4. Reject only a source qualifier that collides with the *visible*
target qualifier in the outer MERGE scope. A source qualifier matching
the real table name remains valid when the target is aliased.
5. Pass the preserved schema and expressions to
`TableProvider::merge_into`, where target and source columns resolve to
distinct physical indices.
### Protobuf change and 55.0/56.0 compatibility
`MergeIntoOpNode` gains optional `target_qualifier` field 3. This field
is necessary because `DmlNode.table_name` contains provider identity and
cannot also represent alias `t`; without it, a proto round trip loses
the binding needed to resolve MERGE expressions.
Compatibility is directional:
- **55.0 payload -> 56.0 reader: supported.** The field is absent, so
the new reader falls back to `DmlNode.table_name`, matching 55.0's
canonicalized representation.
- **56.0 payload -> 55.0 reader: unsupported.** A 55.0 reader ignores
the unknown field but alias-preserving expressions still require it, so
expressions can fail resolution or be misbound.
### Rust API compatibility
DataFusion 55.0 released `MergeIntoOp` with public struct-literal
construction: `MergeIntoOp { on, clauses }`. This PR makes the struct
non-exhaustive, adds private target-qualifier state, and requires
`MergeIntoOp::new(target_qualifier, on, clauses)`. Therefore existing
55.0 downstream struct literals will not compile unchanged against 56.0.
This breaking change is documented in the 56.0 upgrade guide and is
appropriate for the next major release.
The PR also:
- removes alias canonicalization and the recursive target-correlation
guard;
- documents that providers may receive residual subqueries; and
- adds coverage for direct correlations, nested shadowing, lateral and
LIMIT scopes, qualifier collisions, quoted/qualified identifiers, proto
fallback/round trips, and physical column indices.
## Are these changes tested?
- `cargo fmt --all`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `./ci/scripts/doc_prettier_check.sh --write --allow-dirty`
- `RUST_BACKTRACE=1 cargo test --profile ci --exclude
datafusion-examples --exclude datafusion-benchmarks --exclude
datafusion-cli --workspace --lib --tests --bins --features
avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`
## Are there any user-facing changes?
Yes. Both valid MERGE forms above now plan successfully and reach
`TableProvider::merge_into`. The Rust and protobuf compatibility
constraints for upgrading from 55.0 to 56.0 are documented above.
---------
Co-authored-by: kosiew <[email protected]>1 parent bdd29f5 commit 2506ce4
20 files changed
Lines changed: 607 additions & 300 deletions
File tree
- datafusion
- core
- src
- tests/sql
- expr/src/logical_plan
- optimizer/src
- analyzer
- simplify_expressions
- proto-models
- proto
- src/generated
- proto
- src/logical_plan
- tests/cases
- session/src
- sqllogictest/test_files
- sql
- src
- tests
- docs/source/library-user-guide/upgrading
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
870 | 870 | | |
871 | 871 | | |
872 | 872 | | |
873 | | - | |
874 | | - | |
875 | | - | |
876 | | - | |
877 | | - | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
878 | 876 | | |
879 | 877 | | |
880 | 878 | | |
| |||
3642 | 3640 | | |
3643 | 3641 | | |
3644 | 3642 | | |
3645 | | - | |
3646 | | - | |
| 3643 | + | |
| 3644 | + | |
3647 | 3645 | | |
3648 | 3646 | | |
3649 | 3647 | | |
| |||
3654 | 3652 | | |
3655 | 3653 | | |
3656 | 3654 | | |
3657 | | - | |
| 3655 | + | |
3658 | 3656 | | |
3659 | 3657 | | |
3660 | 3658 | | |
3661 | | - | |
| 3659 | + | |
3662 | 3660 | | |
3663 | 3661 | | |
3664 | 3662 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
20 | 23 | | |
21 | 24 | | |
22 | 25 | | |
| |||
215 | 218 | | |
216 | 219 | | |
217 | 220 | | |
218 | | - | |
219 | | - | |
220 | | - | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
221 | 230 | | |
222 | 231 | | |
223 | | - | |
224 | | - | |
225 | | - | |
226 | | - | |
227 | | - | |
228 | | - | |
229 | | - | |
230 | | - | |
231 | | - | |
232 | | - | |
233 | | - | |
234 | | - | |
235 | | - | |
236 | | - | |
237 | | - | |
238 | | - | |
239 | | - | |
240 | | - | |
241 | | - | |
242 | | - | |
243 | | - | |
244 | | - | |
245 | | - | |
246 | | - | |
247 | | - | |
248 | | - | |
249 | | - | |
250 | | - | |
251 | | - | |
252 | | - | |
253 | | - | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
254 | 263 | | |
255 | 264 | | |
256 | 265 | | |
257 | | - | |
| 266 | + | |
258 | 267 | | |
259 | | - | |
260 | | - | |
261 | | - | |
| 268 | + | |
262 | 269 | | |
263 | | - | |
264 | | - | |
265 | | - | |
266 | | - | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
267 | 283 | | |
268 | | - | |
269 | | - | |
270 | 284 | | |
271 | | - | |
272 | | - | |
273 | | - | |
274 | | - | |
275 | | - | |
276 | | - | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
277 | 290 | | |
278 | | - | |
279 | | - | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
280 | 297 | | |
281 | | - | |
282 | 298 | | |
283 | | - | |
284 | | - | |
285 | | - | |
286 | | - | |
287 | | - | |
288 | | - | |
289 | | - | |
290 | | - | |
291 | | - | |
292 | | - | |
293 | | - | |
294 | | - | |
295 | | - | |
296 | | - | |
297 | | - | |
298 | | - | |
299 | | - | |
300 | | - | |
301 | | - | |
302 | | - | |
303 | | - | |
304 | | - | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
305 | 308 | | |
306 | 309 | | |
307 | 310 | | |
| |||
0 commit comments