Repository navigation
Commit 39be5f2
authored
fix(substrait): Preserve pushed-down table scan offsets and limits (apache#26035)
## Which issue does this PR close?
- Closes apache#26034.
## Rationale for this change
Substrait roundtripping loses pushed-down TableScan offsets and limits
when no parent Limit exists. A Projection between the scan and a parent
Limit also bypasses the existing offset compensation, changing query
results.
## What changes are included in this PR?
- Encode a scan's skip/fetch in a FetchRel wrapping its ReadRel.
- Serialize each parent Limit's own offset without adding the scan
offset again.
- Use checked conversions for scan offsets and fetch counts; values
above `i64::MAX` return an explicit unsupported-range error instead of
being encoded as negative values.
- Add execution-based roundtrip coverage for 96 combinations of scan
offsets, fetch counts, optional projections, and parent limits,
including zero limits and offsets beyond the input.
## What is the testing strategy for this PR?
Verified the offset-only regression against upstream
6047791: the original scan returns [2,
3, 4, 5], while the roundtripped scan incorrectly returns all six rows.
The new regression test passes with the fix. An additional regression
test verifies that the pushed-down fetch from `SELECT * FROM t LIMIT
9223372036854775807 OFFSET 1` is rejected with an explicit
unsupported-range error.
Passed:
- cargo fmt --all
- cargo clippy --all-targets --all-features -- -D warnings
- All Substrait integration tests: 227 passed, 6 ignored.
- Required extended workspace tests: 12,388 passed, 0 failed, 8 ignored;
all 526 SQLLogicTests files completed.
- Contributor lint suite through formatting, its feature-specific Clippy
check, TOML formatting, license headers, typos, documentation formatting
and generation, workflow checks, large-file checks, Markdown links, and
security audit.
- Rust documentation check.
No relevant local Substrait producer benchmark is defined.
## Are there any user-facing changes?
Substrait roundtrips preserve pushed-down table-scan offsets and limits,
including across projections and nested limits. No public API changes.1 parent 46bff84 commit 39be5f2
5 files changed
Lines changed: 157 additions & 25 deletions
File tree
- datafusion/substrait
- src/logical_plan/producer/rel
- tests/cases
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
| 57 | + | |
57 | 58 | | |
58 | 59 | | |
59 | 60 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
20 | | - | |
| 20 | + | |
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
52 | 36 | | |
53 | 37 | | |
54 | 38 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
20 | 22 | | |
21 | | - | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
25 | 27 | | |
26 | 28 | | |
27 | 29 | | |
28 | 30 | | |
29 | | - | |
| 31 | + | |
30 | 32 | | |
31 | 33 | | |
32 | 34 | | |
| |||
61 | 63 | | |
62 | 64 | | |
63 | 65 | | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
64 | 72 | | |
65 | 73 | | |
66 | 74 | | |
| |||
95 | 103 | | |
96 | 104 | | |
97 | 105 | | |
98 | | - | |
| 106 | + | |
99 | 107 | | |
100 | 108 | | |
101 | 109 | | |
| |||
108 | 116 | | |
109 | 117 | | |
110 | 118 | | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
111 | 143 | | |
112 | 144 | | |
113 | 145 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
320 | 320 | | |
321 | 321 | | |
322 | 322 | | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
323 | 437 | | |
324 | 438 | | |
325 | 439 | | |
| |||
0 commit comments