Repository navigation
Commit 7dd27a6
committed
fix(parquet): don't runtime-prune row groups while a page-index RowSelection is live (apache#24355)
The runtime row-group pruner rebuilds the decoder via
into_builder().with_row_groups(new_indices), which drops row groups without
slicing the carried page-index RowSelection to match. A dropped RG's selectors
are then applied to the next surviving RG, silently returning wrong results.
Disable the runtime pruner when a row selection is present, mirroring
reorder_by_statistics which already declines to reorder in that case. The proper
fix (slice the selection alongside the row groups) belongs in arrow-rs and is
tracked in apache#24358.
Adds an slt regression test: it fails on main (returns 50..54 instead of 0..4)
and passes with this change.1 parent 00eba79 commit 7dd27a6
2 files changed
Lines changed: 108 additions & 20 deletions
File tree
- datafusion
- datasource-parquet/src/opener
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1435 | 1435 | | |
1436 | 1436 | | |
1437 | 1437 | | |
1438 | | - | |
| 1438 | + | |
1439 | 1439 | | |
1440 | 1440 | | |
1441 | 1441 | | |
| |||
1464 | 1464 | | |
1465 | 1465 | | |
1466 | 1466 | | |
| 1467 | + | |
| 1468 | + | |
| 1469 | + | |
| 1470 | + | |
| 1471 | + | |
| 1472 | + | |
| 1473 | + | |
| 1474 | + | |
1467 | 1475 | | |
1468 | 1476 | | |
1469 | 1477 | | |
| |||
1482 | 1490 | | |
1483 | 1491 | | |
1484 | 1492 | | |
1485 | | - | |
| 1493 | + | |
1486 | 1494 | | |
1487 | 1495 | | |
1488 | 1496 | | |
| |||
1504 | 1512 | | |
1505 | 1513 | | |
1506 | 1514 | | |
1507 | | - | |
1508 | | - | |
1509 | | - | |
1510 | | - | |
1511 | | - | |
1512 | | - | |
1513 | | - | |
1514 | | - | |
1515 | | - | |
1516 | | - | |
1517 | | - | |
1518 | | - | |
1519 | | - | |
1520 | | - | |
1521 | | - | |
1522 | | - | |
1523 | | - | |
1524 | | - | |
| 1515 | + | |
| 1516 | + | |
| 1517 | + | |
| 1518 | + | |
| 1519 | + | |
| 1520 | + | |
| 1521 | + | |
| 1522 | + | |
| 1523 | + | |
| 1524 | + | |
| 1525 | + | |
| 1526 | + | |
| 1527 | + | |
| 1528 | + | |
| 1529 | + | |
| 1530 | + | |
| 1531 | + | |
| 1532 | + | |
| 1533 | + | |
| 1534 | + | |
| 1535 | + | |
| 1536 | + | |
| 1537 | + | |
1525 | 1538 | | |
1526 | 1539 | | |
1527 | 1540 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
194 | 194 | | |
195 | 195 | | |
196 | 196 | | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
0 commit comments