Repository navigation
Commit 16ace4f
authored
## Which issue does this PR close?
- Closes #.
## Rationale for this change
`ORDER BY` applied on top of a `LIMIT ... OFFSET` subquery returns too
few rows.
```sql
SELECT * FROM (SELECT a FROM t1 LIMIT 4 OFFSET 3) ORDER BY a;
```
`EnforceSorting` pushes the sort below the `GlobalLimitExec` and
converts it into a
TopK. The TopK's fetch was seeded from `GlobalLimitExec::fetch()`, which
is the
limit's *output* row count and ignores `skip`. So the TopK kept only 4
rows, the
limit then skipped 3 of them, and the query returned a single row
instead of 4.
The same seeding happens in two places in `sort_pushdown.rs`: when a
root's direct
child is a `GlobalLimitExec` (`assign_initial_requirements`) and when
the pushdown
descends through one (`pushdown_sorts_helper`). Both had the bug.
## What changes are included in this PR?
- Add an `input_fetch` helper in `sort_pushdown.rs` that returns `skip +
fetch` for
`GlobalLimitExec` and plain `fetch()` for every other operator.
- Use it at both sites where the pushed-down fetch is derived from the
plan node, so
the TopK below a `GlobalLimitExec` retains enough rows for the limit to
skip and
still return `fetch` rows.
## What is the testing strategy for this PR?
- `limit.slt`: new `EXPLAIN` + result test for `LIMIT 4 OFFSET 3` under
`ORDER BY`,
showing `TopK(fetch=7)` below `GlobalLimitExec: skip=3, fetch=4` and the
correct
4 rows.
- `ensure_requirements.rs`: two unit tests, one per code path
(`test_sort_pushed_below_limit_with_skip_keeps_skip_rows` and
`test_parent_ordering_over_limit_with_skip_keeps_skip_rows`), asserting
the
pushed sort has `fetch=15` for `skip=5, fetch=10`.
## Are there any user-facing changes?
Queries with a sort above `LIMIT ... OFFSET` now return the correct
number of rows.
No API changes.
1 parent b987501 commit 16ace4f
3 files changed
Lines changed: 134 additions & 2 deletions
File tree
- datafusion
- core/tests/physical_optimizer
- physical-optimizer/src/ensure_requirements/enforce_sorting
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1531 | 1531 | | |
1532 | 1532 | | |
1533 | 1533 | | |
| 1534 | + | |
| 1535 | + | |
| 1536 | + | |
| 1537 | + | |
| 1538 | + | |
| 1539 | + | |
| 1540 | + | |
| 1541 | + | |
| 1542 | + | |
| 1543 | + | |
| 1544 | + | |
| 1545 | + | |
| 1546 | + | |
| 1547 | + | |
| 1548 | + | |
| 1549 | + | |
| 1550 | + | |
| 1551 | + | |
| 1552 | + | |
| 1553 | + | |
| 1554 | + | |
| 1555 | + | |
| 1556 | + | |
| 1557 | + | |
| 1558 | + | |
| 1559 | + | |
| 1560 | + | |
| 1561 | + | |
| 1562 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
82 | 114 | | |
83 | 115 | | |
84 | 116 | | |
| |||
364 | 396 | | |
365 | 397 | | |
366 | 398 | | |
367 | | - | |
| 399 | + | |
368 | 400 | | |
369 | 401 | | |
370 | 402 | | |
| |||
379 | 411 | | |
380 | 412 | | |
381 | 413 | | |
382 | | - | |
| 414 | + | |
383 | 415 | | |
384 | 416 | | |
385 | 417 | | |
| |||
1204 | 1236 | | |
1205 | 1237 | | |
1206 | 1238 | | |
| 1239 | + | |
1207 | 1240 | | |
1208 | 1241 | | |
1209 | 1242 | | |
| |||
1214 | 1247 | | |
1215 | 1248 | | |
1216 | 1249 | | |
| 1250 | + | |
| 1251 | + | |
| 1252 | + | |
| 1253 | + | |
| 1254 | + | |
| 1255 | + | |
| 1256 | + | |
| 1257 | + | |
| 1258 | + | |
| 1259 | + | |
| 1260 | + | |
| 1261 | + | |
| 1262 | + | |
| 1263 | + | |
| 1264 | + | |
| 1265 | + | |
| 1266 | + | |
| 1267 | + | |
| 1268 | + | |
| 1269 | + | |
| 1270 | + | |
| 1271 | + | |
| 1272 | + | |
| 1273 | + | |
| 1274 | + | |
| 1275 | + | |
| 1276 | + | |
| 1277 | + | |
| 1278 | + | |
| 1279 | + | |
| 1280 | + | |
| 1281 | + | |
| 1282 | + | |
| 1283 | + | |
| 1284 | + | |
| 1285 | + | |
| 1286 | + | |
| 1287 | + | |
| 1288 | + | |
| 1289 | + | |
| 1290 | + | |
| 1291 | + | |
| 1292 | + | |
| 1293 | + | |
| 1294 | + | |
| 1295 | + | |
| 1296 | + | |
1217 | 1297 | | |
1218 | 1298 | | |
1219 | 1299 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
339 | 339 | | |
340 | 340 | | |
341 | 341 | | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
342 | 365 | | |
343 | 366 | | |
344 | 367 | | |
| |||
0 commit comments