Repository navigation
Commit c312b1c
committed
fix: fall back to Spark for RANK and DENSE_RANK limits over nested floating-point keys
The native WindowGroupLimit finds RANK and DENSE_RANK ties by comparing
row-encoded ORDER BY keys byte for byte. Scalar float keys are normalized
first, but a float nested in an array or struct keeps its raw bits, so
[-0.0] and [0.0] got different ranks and the cutoff dropped rows that Spark
keeps as ties. In 1.0.0 the limit ran in Spark, so this is a regression in
1.1.0.
Fall back for RANK and DENSE_RANK when an order key contains a nested
FLOAT or DOUBLE. ROW_NUMBER never compares peers and stays native, and
Spark already normalizes floating-point partition keys.
Part of apache#5507.1 parent 9f68a41 commit c312b1c
3 files changed
Lines changed: 93 additions & 5 deletions
File tree
- docs/source/user-guide/latest/compatibility
- spark/src
- main/scala/org/apache/spark/sql/comet
- test/scala/org/apache/comet/exec
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
102 | 102 | | |
103 | 103 | | |
104 | 104 | | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
105 | 110 | | |
106 | 111 | | |
107 | 112 | | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
112 | 118 | | |
113 | 119 | | |
114 | 120 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
32 | | - | |
| 33 | + | |
33 | 34 | | |
34 | 35 | | |
35 | 36 | | |
| |||
99 | 100 | | |
100 | 101 | | |
101 | 102 | | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
102 | 130 | | |
103 | 131 | | |
104 | 132 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
210 | 210 | | |
211 | 211 | | |
212 | 212 | | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
213 | 267 | | |
214 | 268 | | |
215 | 269 | | |
| |||
0 commit comments