Repository navigation
Commit 6286f5e
authored
fix: saturate window frame lookahead instead of overflowing on huge FOLLOWING bounds (apache#26662)
## Which issue does this PR close?
- Closes apache#26658.
## Rationale for this change
A `ROWS` window frame with a huge `FOLLOWING` bound (for example
`18446744073709551615 FOLLOWING`) combined with `ORDER BY ... LIMIT`
panics with `attempt to add with overflow` in debug builds. In release
builds the addition wraps, so `LimitPushPastWindows` could give the
`SortExec` below the window a fetch that is far too small and drop rows
the window needs.
After fixing the optimizer rule, the same query panicked a second time
in `WindowFrameContext::calculate_range_rows` (`window_state.rs`), which
computes the frame end as `idx + n + 1` with the same unchecked bound.
So both places need fixing for the query in the issue to return its
result.
## What changes are included in this PR?
- `limit_pushdown_past_window.rs`: use `saturating_add` when growing the
limit by the window lookahead (`fetch + lookahead`, `lookahead +
max_rel`) and when adding `skip` to a `GlobalLimitExec` fetch. A
saturated fetch is still correct, it just means the sort keeps all rows.
This matches how `input_fetch` in `sort_pushdown.rs` already handles the
same kind of sum.
- `window_state.rs`: use `saturating_add` for the `FOLLOWING n` start
and end of a `ROWS` frame. The result is already clamped to the
partition length, so saturating gives the right range.
This touches the same lines in `limit_pushdown_past_window.rs` as the
refactor in apache#26660, which leaves these sums to apache#26658. The change here
is small, so whichever lands second should be easy to rebase.
## What is the testing strategy for this PR?
- New cases in `window_limits.slt`: the query from the issue (`sum(x)
OVER (... ROWS BETWEEN CURRENT ROW AND 18446744073709551615 FOLLOWING)
... LIMIT 2`), and a `lead(x, 1)` variant that adds its own lookahead on
top of the huge frame bound. Without this change the first case panics
at `limit_pushdown_past_window.rs:198`; with only the optimizer fix it
panics at `window_state.rs:287`.
- Locally: all `window*.slt` sqllogictests, `cargo test -p
datafusion-expr --lib window_state`, `cargo test -p
datafusion-physical-optimizer --lib limit_pushdown_past_window`, `cargo
fmt --all` and `cargo clippy -p datafusion-expr -p
datafusion-physical-optimizer --all-targets --all-features -- -D
warnings` pass.
## Are there any user-facing changes?
No API changes. Queries with very large `ROWS ... FOLLOWING` bounds now
return results instead of panicking (or, in release builds, possibly
returning too few rows).
---
This PR was prepared with AI assistance. I reviewed the change and ran
the tests listed above.1 parent 914fc1b commit 6286f5e
3 files changed
Lines changed: 68 additions & 6 deletions
File tree
- datafusion
- expr/src
- physical-optimizer/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
263 | 263 | | |
264 | 264 | | |
265 | 265 | | |
| 266 | + | |
266 | 267 | | |
267 | | - | |
| 268 | + | |
268 | 269 | | |
269 | 270 | | |
270 | 271 | | |
| |||
289 | 290 | | |
290 | 291 | | |
291 | 292 | | |
292 | | - | |
| 293 | + | |
293 | 294 | | |
294 | 295 | | |
295 | 296 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
177 | 177 | | |
178 | 178 | | |
179 | 179 | | |
180 | | - | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
181 | 184 | | |
182 | 185 | | |
183 | 186 | | |
| |||
194 | 197 | | |
195 | 198 | | |
196 | 199 | | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
197 | 206 | | |
198 | | - | |
199 | | - | |
| 207 | + | |
| 208 | + | |
200 | 209 | | |
201 | 210 | | |
202 | 211 | | |
203 | 212 | | |
204 | 213 | | |
205 | 214 | | |
206 | | - | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
207 | 220 | | |
208 | 221 | | |
209 | 222 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
862 | 862 | | |
863 | 863 | | |
864 | 864 | | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
0 commit comments