[Bug] FLOAT/DOUBLE stats pruning drops NaN rows for >, >= #1159

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

main (4a947ec)

Describe the bug

The writer leaves NaN out of FLOAT/DOUBLE min/max without counting it as null. Range filters still prune by max, so a NaN row gets dropped. NaN is above every number in Java, and in the row filter too.

With two files [(0, 0.0), (1, NaN)] and [(2, 2.0)]:

Putting all rows in one file gives [1, 2]. Java GreaterThan(f, 1.0).test(NaN) is true (2.1.0). NOT BETWEEN and NOT(<), NOT(<=), NOT(BETWEEN) have the same problem. #926 fixed it for NOT(=).

How to reproduce

with_filter(greater_than("f", 1.0)) on the two files above.

Additional context

Partitioning by f has a similar gap (the NaN partition is dropped for f > 1.0 and f = NaN), but that's about NaN ordering, which #1055 put off, so I left it out.

Are you willing to submit a PR?

• I'm willing to submit a PR!