Repository navigation
Commit 3f4a37f
## Which issue does this PR close?
- Closes #26162.
## Rationale for this change
In #26162, running `ORDER BY b, a` over a multi-partition subquery with
`ORDER BY b LIMIT n` produces incorrect ordering on `a`.
Pushing `[b, a]` down removed the outer `SortExec`, but SPM only merges
on `[b]`. As a result, `a` was not sorted in the final output.
In the discussion, @jayzhan211 suggested two possible approaches:
declining pushdown for SPM with fetch, or rewriting SPM's expressions
when pushing through it.
I chose `Ok(None)` (declining pushdown) because:
- The subquery did not explicitly request the parent's sort. When a
limit is present, forcing that sort on the entire dataset across
partitions before a limit is unnecessary work.
- Since `pushdown_requirement_to_children` only adds requirements to
children, I think a logic to rewrite SPM adds extra complexity.
## What changes are included in this PR?
- Return `Ok(None)` from `pushdown_requirement_to_children` when `plan`
is `SortPreservingMergeExec` and has a `fetch`.
- Add an SLT regression query in `limit.slt`.
## What is the testing strategy for this PR?
- query on a partitioned table in `limit.slt` reproducing #26162.
## Are there any user-facing changes?
Yes, bug fix: queries like `ORDER BY b, a` over a multi-partition
subquery with `ORDER BY b LIMIT n` now return correctly ordered results
on `a`. No API changes.
---------
Co-authored-by: Jay Zhan <[email protected]>
1 parent 042727c commit 3f4a37f
3 files changed
Lines changed: 102 additions & 34 deletions
File tree
- datafusion
- core/tests/physical_optimizer
- physical-optimizer/src/ensure_requirements/enforce_sorting
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
522 | 522 | | |
523 | 523 | | |
524 | 524 | | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
525 | 551 | | |
526 | 552 | | |
527 | 553 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
9 | | - | |
10 | | - | |
11 | | - | |
| 8 | + | |
12 | 9 | | |
13 | 10 | | |
14 | 11 | | |
| |||
524 | 521 | | |
525 | 522 | | |
526 | 523 | | |
527 | | - | |
528 | | - | |
529 | | - | |
530 | | - | |
531 | | - | |
532 | | - | |
533 | | - | |
534 | | - | |
535 | | - | |
536 | | - | |
537 | | - | |
538 | | - | |
539 | | - | |
540 | | - | |
541 | | - | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
542 | 528 | | |
543 | 529 | | |
544 | 530 | | |
| |||
593 | 579 | | |
594 | 580 | | |
595 | 581 | | |
596 | | - | |
597 | | - | |
598 | | - | |
599 | | - | |
600 | | - | |
601 | | - | |
602 | | - | |
603 | | - | |
604 | | - | |
605 | | - | |
606 | | - | |
607 | | - | |
608 | | - | |
609 | | - | |
610 | | - | |
611 | 582 | | |
612 | 583 | | |
613 | 584 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1331 | 1331 | | |
1332 | 1332 | | |
1333 | 1333 | | |
| 1334 | + | |
| 1335 | + | |
| 1336 | + | |
| 1337 | + | |
| 1338 | + | |
| 1339 | + | |
| 1340 | + | |
| 1341 | + | |
| 1342 | + | |
| 1343 | + | |
| 1344 | + | |
| 1345 | + | |
| 1346 | + | |
| 1347 | + | |
| 1348 | + | |
| 1349 | + | |
| 1350 | + | |
| 1351 | + | |
| 1352 | + | |
| 1353 | + | |
| 1354 | + | |
| 1355 | + | |
| 1356 | + | |
| 1357 | + | |
| 1358 | + | |
| 1359 | + | |
| 1360 | + | |
| 1361 | + | |
| 1362 | + | |
| 1363 | + | |
| 1364 | + | |
| 1365 | + | |
| 1366 | + | |
| 1367 | + | |
| 1368 | + | |
| 1369 | + | |
| 1370 | + | |
| 1371 | + | |
| 1372 | + | |
| 1373 | + | |
| 1374 | + | |
| 1375 | + | |
| 1376 | + | |
| 1377 | + | |
| 1378 | + | |
| 1379 | + | |
| 1380 | + | |
| 1381 | + | |
| 1382 | + | |
| 1383 | + | |
| 1384 | + | |
| 1385 | + | |
| 1386 | + | |
| 1387 | + | |
| 1388 | + | |
| 1389 | + | |
| 1390 | + | |
| 1391 | + | |
| 1392 | + | |
| 1393 | + | |
| 1394 | + | |
| 1395 | + | |
| 1396 | + | |
| 1397 | + | |
| 1398 | + | |
| 1399 | + | |
| 1400 | + | |
| 1401 | + | |
| 1402 | + | |
| 1403 | + | |
| 1404 | + | |
0 commit comments