Repository navigation
Commit 3470bd0
fix: wait for the whole NULL peer group before emitting RANGE
## Which issue does this PR close?
- Closes apache#25859.
## Rationale for this change
Window aggregates over a `RANGE ... N PRECEDING` frame return results
that depend on how the input is split into batches when the ORDER BY
value is NULL. For a NULL row the frame is its group of NULL peers, and
when that group spans batches the row is computed before the rest of the
group arrives. At default settings a query over 20,000 rows with 10,000
NULL keys returns a count of 6384 instead of 10000 for 6,384 of the NULL
rows. Details and repros are in the issue.
## What changes are included in this PR?
- `WindowFrame::new_bounds`: a `RANGE` frame is causal only when it ends
at `UNBOUNDED PRECEDING`. With an offset end bound, the frame of a NULL
or NaN row ends at its last peer, which can be a later row. `GROUPS`
keeps the current rule, since NULL peers form an ordinary group there.
- `is_end_bound_safe_for_range`: a `PRECEDING` end bound is handled like
`CURRENT ROW`, so the row waits until an input row past its peer group
arrives (`is_row_ahead`) or the partition ends. An `N PRECEDING` frame
only reaches the end of the buffer when the row's frame ends at its last
peer, so the check only holds back NULL and NaN rows.
Rows with other ORDER BY values behave as before: their `N PRECEDING`
frame ends before the current row, so it never reaches the end of the
buffer and results are still produced as the input streams. RANGE frames
planned from SQL were already non-causal (their bounds are strings until
type coercion), so their plans don't change; the second change is what
fixes them. The first change fixes frames built from typed bounds
(reversed frames, the DataFrame API, proto), where built-in window
functions such as `nth_value` were affected too.
## What is the testing strategy for this PR?
New cases at the end of `window.slt`, with `batch_size = 2` so the peers
span batches: `sum`/`count`/`max` over `5 PRECEDING AND 1 PRECEDING` and
`UNBOUNDED PRECEDING AND 1 PRECEDING`, NULLS FIRST and DESC orderings, a
`FOLLOWING` frame that the planner reverses to reuse another window's
sort (covering `nth_value`), and a group of NaN keys. Expected values
match DuckDB 1.5.5 and Postgres 17, and all four queries fail without
the fix.
I also ran a randomized check outside the test suite: random tables with
NULL keys, random RANGE/GROUPS/ROWS frames and functions whose result
doesn't depend on tie order, compared across batch sizes 1, 2, 3, 7 and
8192 and against DuckDB. Main had 31 batch-dependent results in the
first 1,000 queries; this branch had none in 7,500, and none in variants
with NaN keys and with an unbounded ordered source (Linear mode,
PARTITION BY). The whole sqllogictest suite and the window fuzz tests
(`--features extended_tests`) pass.
## Are there any user-facing changes?
Queries with NULL or NaN values in the ORDER BY column of a `RANGE ... N
PRECEDING` window now return the same results regardless of batch size.
`WindowFrame::is_causal()` now returns false for RANGE frames with an
offset end bound. For frames built from typed bounds this can drop an
ordering that a set-monotonic aggregate used to add to the window
output, so a plan may need an extra sort; SQL-planned frames were
already non-causal, and no sqllogictest plan changed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>N PRECEDING window results (apache#25860)1 parent 5b5edc7 commit 3470bd0
3 files changed
Lines changed: 88 additions & 10 deletions
File tree
- datafusion
- expr/src
- physical-expr/src/window
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
224 | 224 | | |
225 | 225 | | |
226 | 226 | | |
227 | | - | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
228 | 234 | | |
229 | 235 | | |
230 | 236 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
507 | 507 | | |
508 | 508 | | |
509 | 509 | | |
510 | | - | |
511 | | - | |
512 | | - | |
513 | | - | |
514 | | - | |
515 | | - | |
516 | | - | |
517 | | - | |
518 | | - | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
519 | 514 | | |
520 | 515 | | |
521 | 516 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7345 | 7345 | | |
7346 | 7346 | | |
7347 | 7347 | | |
| 7348 | + | |
| 7349 | + | |
| 7350 | + | |
| 7351 | + | |
| 7352 | + | |
| 7353 | + | |
| 7354 | + | |
| 7355 | + | |
| 7356 | + | |
| 7357 | + | |
| 7358 | + | |
| 7359 | + | |
| 7360 | + | |
| 7361 | + | |
| 7362 | + | |
| 7363 | + | |
| 7364 | + | |
| 7365 | + | |
| 7366 | + | |
| 7367 | + | |
| 7368 | + | |
| 7369 | + | |
| 7370 | + | |
| 7371 | + | |
| 7372 | + | |
| 7373 | + | |
| 7374 | + | |
| 7375 | + | |
| 7376 | + | |
| 7377 | + | |
| 7378 | + | |
| 7379 | + | |
| 7380 | + | |
| 7381 | + | |
| 7382 | + | |
| 7383 | + | |
| 7384 | + | |
| 7385 | + | |
| 7386 | + | |
| 7387 | + | |
| 7388 | + | |
| 7389 | + | |
| 7390 | + | |
| 7391 | + | |
| 7392 | + | |
| 7393 | + | |
| 7394 | + | |
| 7395 | + | |
| 7396 | + | |
| 7397 | + | |
| 7398 | + | |
| 7399 | + | |
| 7400 | + | |
| 7401 | + | |
| 7402 | + | |
| 7403 | + | |
| 7404 | + | |
| 7405 | + | |
| 7406 | + | |
| 7407 | + | |
| 7408 | + | |
| 7409 | + | |
| 7410 | + | |
| 7411 | + | |
| 7412 | + | |
| 7413 | + | |
| 7414 | + | |
| 7415 | + | |
| 7416 | + | |
| 7417 | + | |
| 7418 | + | |
| 7419 | + | |
| 7420 | + | |
| 7421 | + | |
| 7422 | + | |
| 7423 | + | |
| 7424 | + | |
0 commit comments