Repository navigation
Commit a0330ea
perf: decode Parquet footers for statistics in parallel during planning (apache#26407)
## Which issue does this PR close?
- Related to apache#21968. There is no dedicated issue for this change.
## Rationale for this change
When a `ListingTable` over Parquet files collects statistics, planning
time grows with the number of files multiplied by the size of their
footers, and none of that work runs in parallel. On wide tables it
dominates planning.
`ListingTable::collect_files_for_scan` gathers per-file statistics with
`buffer_unordered(meta_fetch_concurrency)`. That makes the object store
requests concurrent, but every future is polled by the one planning
task. As a result, the footer decode and statistics conversion for each
file run one file at a time, on whichever thread is polling the planner.
In a profile of `wide_schema` Q02 (1024 columns, 256 files) with the
metadata and statistics caches disabled, the planning thread spent about
345 ms per query, which is roughly the whole query time. About three
quarters of that time was inside
`ParquetFormat::infer_stats_and_ordering`. With the default cache limits
the statistics cache cannot hold the statistics for this table, so the
same work happens on every query.
## What changes are included in this PR?
- `ParquetFormat::infer_stats_and_ordering` resolves its
session-dependent inputs (the decryption properties and the metadata
cache) on the calling task. When it runs inside a multi-thread Tokio
runtime, it then fetches and decodes the footer, and computes the
statistics and ordering, in a `SpawnedTask`. With the existing
`buffer_unordered`, up to `meta_fetch_concurrency` files are decoded in
parallel on the runtime's threads.
- Outside a multi-thread runtime (a current-thread runtime, or no
runtime at all, as in the wasm tests), it does the work on the calling
task as before, so the caller's `buffer_unordered` polls it together
with the other files. This is the same check `spawn_buffered` uses.
- The shared work moves into a private helper,
`fetch_stats_and_ordering`, which both paths call.
This follows the convention described in the crate docs: CPU-bound work
runs as tasks on the Tokio runtime, through `SpawnedTask`, rather than
in `spawn_blocking`. Dropping a `SpawnedTask` aborts it, so when
`get_files_with_limit` stops early, outstanding work is still cancelled.
`ParquetFormat::infer_schema` has the same serial pattern when a table
is created. It is left for a follow-up.
## What is the testing strategy for this PR?
The results are unchanged; only where the work runs has changed, so the
existing tests cover it. The `#[tokio::test]` unit tests run on a
current-thread runtime and take the inline path, while the sqllogictests
run on a multi-thread runtime and take the spawned path.
- `datafusion-datasource-parquet` lib tests
- `datafusion-catalog-listing` tests
- `datafusion` `parquet_integration` tests and the core
`datasource::listing` tests
- the Parquet sqllogictests
The performance change will be checked with the `wide_schema` benchmark.
## Are there any user-facing changes?
There are no API changes. Planning that collects Parquet statistics uses
more than one thread when it runs inside a multi-thread Tokio runtime,
and behaves as before otherwise.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_019C4oRqstvxH5CXTWnot8CD
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent 123747e commit a0330ea
1 file changed
Lines changed: 73 additions & 16 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| 41 | + | |
41 | 42 | | |
42 | 43 | | |
43 | 44 | | |
44 | 45 | | |
| 46 | + | |
45 | 47 | | |
46 | 48 | | |
47 | 49 | | |
| |||
388 | 390 | | |
389 | 391 | | |
390 | 392 | | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
391 | 419 | | |
392 | 420 | | |
393 | 421 | | |
| |||
562 | 590 | | |
563 | 591 | | |
564 | 592 | | |
565 | | - | |
566 | | - | |
567 | | - | |
568 | | - | |
569 | | - | |
570 | | - | |
571 | | - | |
572 | | - | |
573 | | - | |
574 | | - | |
575 | | - | |
576 | | - | |
577 | | - | |
578 | | - | |
579 | | - | |
580 | | - | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
581 | 638 | | |
582 | 639 | | |
583 | 640 | | |
| |||
0 commit comments