Repository navigation
Commit 0eecdfc
authored
refactor(parquet): strip empty row groups so PreparedAccessPlan stays well-formed (apache#24509)
## Which issue does this PR close?
- Closes apache#24287.
## Rationale for this change
Split out from apache#23696 (per @adriangb's [decomposition
proposal](apache#23696 (comment))).
This keeps `PreparedAccessPlan` well-formed: a row group whose
`RowSelection` selects zero rows is dropped so `with_row_groups(...)`
never names a row group the push decoder would immediately skip.
arrow-rs advances past an all-skipped row group inside `try_next_reader`
without handing back a reader, so keeping such a row group in the plan
would leave the plan's row-group list one entry longer than the readers
produced.
This is mostly **hardening** — the two in-tree flat-selection producers
already avoid landing an empty `Selection` (`page_filter.rs` maps an
all-skip selection to `Skip`; `try_new_from_overall_row_selection`
normalizes via `RowGroupAccessBuilder::into_access`), and
`RowGroupPruner` is gated off whenever a selection is live. There is,
however, **one reachable producer**: `ParquetAccessPlan::scan_selection`
intersects an existing `Selection` with a new one and can leave a row
group selecting nothing (its `rows_selected > 0` guard checks the
incoming selection, not the intersection). A well-formed plan here is
also a precondition for re-enabling runtime pruning under a live
selection (apache#24358) and for apache#23696.
## What changes are included in this PR?
- `access_plan.rs`: `PreparedAccessPlan::prepare` calls a new generic
`strip_empty_row_groups(row_group_indexes, row_selection,
row_group_meta_data)`. It walks the flat `RowSelection` once with
`OverallRowSelectionCursor` (the same single-pass approach
`try_new_from_overall_row_selection` uses, rather than a per-row-group
`split_off`) and keeps only row groups that still select a row.
- `sort.rs`: `test_prepared_access_plan_reverse_empty_selection` — an
all-skipped plan now strips to an empty plan.
No `fully_matched` coupling, no public API change, no new metric, no slt
churn.
## Are these changes tested?
Yes — unit tests in `access_plan.rs`:
- `test_strip_empty_row_groups_drops_only_empties` — mixed
empty/non-empty row groups.
- `test_strip_empty_row_groups_non_contiguous_indexes` — `vec![1, 3]`
(RGs 0/2 skipped), pinning that `row_group_meta_data` is indexed
absolutely.
- `test_strip_empty_row_groups_none_selection_unchanged` — `None` passes
through.
- `test_prepare_strips_row_group_emptied_by_intersecting_selections` —
the reachable `scan_selection` empty-intersection path, exercised
through `prepare()`.
## Are there any user-facing changes?
No.
## Follow-up
@adriangb also suggested a complementary one-line normalization in
`scan_selection` itself (empty intersection → `Skip`), which would keep
`ParquetAccessPlan` well-formed for its other consumers
(`row_group_indexes()`, metrics, `is_fully_matched`) too. Left out here
to keep this PR scoped to apache#24287; happy to do it as a small follow-up.1 parent dbdc627 commit 0eecdfc
2 files changed
Lines changed: 171 additions & 13 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
573 | 573 | | |
574 | 574 | | |
575 | 575 | | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
576 | 579 | | |
577 | 580 | | |
578 | 581 | | |
579 | 582 | | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
580 | 653 | | |
581 | 654 | | |
582 | 655 | | |
| |||
1430 | 1503 | | |
1431 | 1504 | | |
1432 | 1505 | | |
| 1506 | + | |
| 1507 | + | |
| 1508 | + | |
| 1509 | + | |
| 1510 | + | |
| 1511 | + | |
| 1512 | + | |
| 1513 | + | |
| 1514 | + | |
| 1515 | + | |
| 1516 | + | |
| 1517 | + | |
| 1518 | + | |
| 1519 | + | |
| 1520 | + | |
| 1521 | + | |
| 1522 | + | |
| 1523 | + | |
| 1524 | + | |
| 1525 | + | |
| 1526 | + | |
| 1527 | + | |
| 1528 | + | |
| 1529 | + | |
| 1530 | + | |
| 1531 | + | |
| 1532 | + | |
| 1533 | + | |
| 1534 | + | |
| 1535 | + | |
| 1536 | + | |
| 1537 | + | |
| 1538 | + | |
| 1539 | + | |
| 1540 | + | |
| 1541 | + | |
| 1542 | + | |
| 1543 | + | |
| 1544 | + | |
| 1545 | + | |
| 1546 | + | |
| 1547 | + | |
| 1548 | + | |
| 1549 | + | |
| 1550 | + | |
| 1551 | + | |
| 1552 | + | |
| 1553 | + | |
| 1554 | + | |
| 1555 | + | |
| 1556 | + | |
| 1557 | + | |
| 1558 | + | |
| 1559 | + | |
| 1560 | + | |
| 1561 | + | |
| 1562 | + | |
| 1563 | + | |
| 1564 | + | |
| 1565 | + | |
| 1566 | + | |
| 1567 | + | |
| 1568 | + | |
| 1569 | + | |
| 1570 | + | |
| 1571 | + | |
| 1572 | + | |
| 1573 | + | |
| 1574 | + | |
| 1575 | + | |
| 1576 | + | |
| 1577 | + | |
| 1578 | + | |
| 1579 | + | |
| 1580 | + | |
| 1581 | + | |
| 1582 | + | |
| 1583 | + | |
| 1584 | + | |
| 1585 | + | |
| 1586 | + | |
| 1587 | + | |
| 1588 | + | |
| 1589 | + | |
| 1590 | + | |
| 1591 | + | |
| 1592 | + | |
1433 | 1593 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
407 | 407 | | |
408 | 408 | | |
409 | 409 | | |
410 | | - | |
| 410 | + | |
| 411 | + | |
411 | 412 | | |
412 | 413 | | |
413 | 414 | | |
| |||
423 | 424 | | |
424 | 425 | | |
425 | 426 | | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
426 | 434 | | |
427 | 435 | | |
428 | 436 | | |
429 | | - | |
430 | | - | |
431 | | - | |
432 | | - | |
433 | | - | |
434 | | - | |
435 | | - | |
436 | | - | |
437 | | - | |
438 | | - | |
439 | | - | |
440 | | - | |
| 437 | + | |
| 438 | + | |
441 | 439 | | |
442 | 440 | | |
443 | 441 | | |
| |||
0 commit comments