Skip to content

Wrong results: ORDER BY on MAX/MIN over a sliding window frame is skipped #27422

Open

Description

@jayzhan211

Describe the bug

Sorting on MAX or MIN over a sliding window frame (e.g. ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) can return rows in the wrong order: the physical optimizer thinks the window output is already sorted and removes the final sort. It happens when the window is ordered DESC for MAX or ASC for MIN, when the frame skips the current row, or when NULL inputs sort first. Reproduced on main at ac1a63d; the rule has been there since 46.0.0.

To Reproduce

CREATE TABLE u(x INT NOT NULL) AS VALUES (3), (1), (5);
-- case A
SELECT x, m FROM (SELECT x, MAX(x) OVER (ORDER BY x DESC ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS m FROM u) ORDER BY m ASC NULLS LAST;
SQL for cases B, C, D
CREATE TABLE u(x INT NOT NULL) AS VALUES (3), (1), (5);
CREATE TABLE t(x INT) AS VALUES (3), (NULL), (1), (5);
-- B
SELECT x, m FROM (SELECT x, MIN(x) OVER (ORDER BY x ASC ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS m FROM u) ORDER BY m DESC NULLS LAST;
-- C
SELECT x, m FROM (SELECT x, MAX(x) OVER (ORDER BY x ASC ROWS BETWEEN 2 PRECEDING AND 1 PRECEDING) AS m FROM u) ORDER BY m ASC NULLS LAST;
-- D
SELECT x, m FROM (SELECT x, MAX(x) OVER (ORDER BY x ASC NULLS FIRST ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS m FROM t) ORDER BY m ASC NULLS LAST;
Case What differs m returned m expected
A MAX, window ordered x DESC 5, 5, 3 3, 5, 5
B MIN, window ordered x ASC 1, 1, 3 3, 1, 1
C frame skips the current row NULL, 1, 3 1, 3, NULL
D NULL input sorted first NULL, 1, 3, 5 1, 3, 5, NULL

MAX with the window ordered x ASC and a frame that includes the current row returns the right order.

Expected behavior

Rows come back in the order of the final ORDER BY (the "expected" column).

Additional context

window_equivalence_properties (datafusion/physical-plan/src/windows/mod.rs:509-570) declares an output ordering for set-monotonic aggregates over sliding frames, and that ordering is what lets the final sort be removed. Two parts of it are wrong:

  • Direction. Without PARTITION BY it declares MAX ascending and MIN descending (increasing && (asc || no_partitioning), line 560). But MAX over input ordered x DESC, like MIN over x ASC, takes the oldest row in the frame, so the result follows the input's order (A, B).
  • NULLs. It always declares NULLS LAST (lines 563, 568). A frame with no non-NULL input gives NULL: at the start when the frame skips the current row (C) or NULL inputs sort first (D), and at the end when enough NULL inputs sort last. With 2 PRECEDING AND 1 PRECEDING over 1, 3, NULL, NULL, NULL the window returns NULL, 1, 3, 3, NULL, so no NULL placement is safe there.

The rule came in #14813 and is unchanged from 46.0.0 through branch-56. #27267 is the same NULL mistake in the rules for frames that only grow or only shrink, a separate code path.

Anyone is welcome to take this; please ping me for review.

Activity

  1. self-assigned this
    on Nov 14, 2026
  2. blaginin commented on Nov 14, 2026

    @blaginin
    Member

    Also came across something similar, in my case it even lost some data...

    DataFusion CLI v57.0.0-dev
    > SET datafusion.execution.target_partitions = 1;
    
    CREATE TABLE t(g INT NOT NULL, ts INT NOT NULL)
    AS VALUES (1, 1), (1, 2), (2, 3), (2, 4);
    
    SELECT COUNT(*) OVER (
        PARTITION BY g
        ORDER BY ts
        ROWS BETWEEN 1 PRECEDING AND CURRENT ROW
    ) AS c
    FROM t
    ORDER BY c ASC
    LIMIT 2;
    0 row(s) fetched.
    Elapsed 0.013 seconds.
    
    0 row(s) fetched.
    Elapsed 0.020 seconds.
    
    +----+
    | c  |
    +----+
    | 1  |
    | 2  |
    +----+
    2 row(s) fetched.
    Elapsed 0.013 seconds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions