Skip to content

Commit a0330ea

Browse files
cetra3claude
andauthored
perf: decode Parquet footers for statistics in parallel during planning (apache#26407)
## Which issue does this PR close? - Related to apache#21968. There is no dedicated issue for this change. ## Rationale for this change When a `ListingTable` over Parquet files collects statistics, planning time grows with the number of files multiplied by the size of their footers, and none of that work runs in parallel. On wide tables it dominates planning. `ListingTable::collect_files_for_scan` gathers per-file statistics with `buffer_unordered(meta_fetch_concurrency)`. That makes the object store requests concurrent, but every future is polled by the one planning task. As a result, the footer decode and statistics conversion for each file run one file at a time, on whichever thread is polling the planner. In a profile of `wide_schema` Q02 (1024 columns, 256 files) with the metadata and statistics caches disabled, the planning thread spent about 345 ms per query, which is roughly the whole query time. About three quarters of that time was inside `ParquetFormat::infer_stats_and_ordering`. With the default cache limits the statistics cache cannot hold the statistics for this table, so the same work happens on every query. ## What changes are included in this PR? - `ParquetFormat::infer_stats_and_ordering` resolves its session-dependent inputs (the decryption properties and the metadata cache) on the calling task. When it runs inside a multi-thread Tokio runtime, it then fetches and decodes the footer, and computes the statistics and ordering, in a `SpawnedTask`. With the existing `buffer_unordered`, up to `meta_fetch_concurrency` files are decoded in parallel on the runtime's threads. - Outside a multi-thread runtime (a current-thread runtime, or no runtime at all, as in the wasm tests), it does the work on the calling task as before, so the caller's `buffer_unordered` polls it together with the other files. This is the same check `spawn_buffered` uses. - The shared work moves into a private helper, `fetch_stats_and_ordering`, which both paths call. This follows the convention described in the crate docs: CPU-bound work runs as tasks on the Tokio runtime, through `SpawnedTask`, rather than in `spawn_blocking`. Dropping a `SpawnedTask` aborts it, so when `get_files_with_limit` stops early, outstanding work is still cancelled. `ParquetFormat::infer_schema` has the same serial pattern when a table is created. It is left for a follow-up. ## What is the testing strategy for this PR? The results are unchanged; only where the work runs has changed, so the existing tests cover it. The `#[tokio::test]` unit tests run on a current-thread runtime and take the inline path, while the sqllogictests run on a multi-thread runtime and take the spawned path. - `datafusion-datasource-parquet` lib tests - `datafusion-catalog-listing` tests - `datafusion` `parquet_integration` tests and the core `datasource::listing` tests - the Parquet sqllogictests The performance change will be checked with the `wide_schema` benchmark. ## Are there any user-facing changes? There are no API changes. Planning that collects Parquet statistics uses more than one thread when it runs inside a multi-thread Tokio runtime, and behaves as before otherwise. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019C4oRqstvxH5CXTWnot8CD Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 123747e commit a0330ea

1 file changed

Lines changed: 73 additions & 16 deletions

File tree

‎datafusion/datasource-parquet/src/file_format.rs‎

Lines changed: 73 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -38,10 +38,12 @@ use datafusion_common::{
3838
DEFAULT_PARQUET_EXTENSION, DataFusionError, GetExt, Result, internal_datafusion_err,
3939
internal_err, not_impl_err,
4040
};
41+
use datafusion_common_runtime::SpawnedTask;
4142
use datafusion_datasource::file::FileSource;
4243
use datafusion_datasource::file_scan_config::{FileScanConfig, FileScanConfigBuilder};
4344
use datafusion_datasource::sink::DataSinkExec;
4445
use datafusion_datasource::write::get_writer_schema;
46+
use datafusion_execution::cache::cache_manager::FileMetadataCache;
4547
use datafusion_expr::dml::InsertOp;
4648
use datafusion_physical_expr_common::sort_expr::{LexOrdering, LexRequirement};
4749
use datafusion_physical_plan::ExecutionPlan;
@@ -388,6 +390,32 @@ async fn get_file_decryption_properties(
388390
Ok(None)
389391
}
390392

393+
/// Fetch the footer of `object` and compute its statistics and ordering for
394+
/// [`ParquetFormat::infer_stats_and_ordering`].
395+
async fn fetch_stats_and_ordering(
396+
store: &dyn ObjectStore,
397+
object: &ObjectMeta,
398+
metadata_size_hint: Option<usize>,
399+
file_decryption_properties: Option<Arc<FileDecryptionProperties>>,
400+
file_metadata_cache: Arc<FileMetadataCache>,
401+
table_schema: &SchemaRef,
402+
) -> Result<datafusion_datasource::file_format::FileMeta> {
403+
let metadata = DFParquetMetadata::new(store, object)
404+
.with_metadata_size_hint(metadata_size_hint)
405+
.with_decryption_properties(file_decryption_properties)
406+
.with_file_metadata_cache(Some(file_metadata_cache))
407+
.fetch_metadata()
408+
.await?;
409+
let statistics =
410+
DFParquetMetadata::statistics_from_parquet_metadata(&metadata, table_schema)?;
411+
let ordering =
412+
crate::metadata::ordering_from_parquet_metadata(&metadata, table_schema)?;
413+
Ok(
414+
datafusion_datasource::file_format::FileMeta::new(statistics)
415+
.with_ordering(ordering),
416+
)
417+
}
418+
391419
#[async_trait]
392420
impl FileFormat for ParquetFormat {
393421
fn get_ext(&self) -> String {
@@ -562,22 +590,51 @@ impl FileFormat for ParquetFormat {
562590
.await?;
563591
let file_metadata_cache =
564592
state.runtime_env().cache_manager.get_file_metadata_cache();
565-
let metadata = DFParquetMetadata::new(store, object)
566-
.with_metadata_size_hint(self.metadata_size_hint())
567-
.with_decryption_properties(file_decryption_properties)
568-
.with_file_metadata_cache(Some(file_metadata_cache))
569-
.fetch_metadata()
570-
.await?;
571-
let statistics = DFParquetMetadata::statistics_from_parquet_metadata(
572-
&metadata,
573-
&table_schema,
574-
)?;
575-
let ordering =
576-
crate::metadata::ordering_from_parquet_metadata(&metadata, &table_schema)?;
577-
Ok(
578-
datafusion_datasource::file_format::FileMeta::new(statistics)
579-
.with_ordering(ordering),
580-
)
593+
let metadata_size_hint = self.metadata_size_hint();
594+
595+
// Decoding the footer and converting its statistics is CPU bound and
596+
// grows with the number of columns. Callers such as `ListingTable` run
597+
// this for many files concurrently from a single planning task, so run
598+
// it in its own task to spread that work over the runtime's threads
599+
// instead of decoding every file on the thread polling the planner.
600+
// As in `spawn_buffered`, only spawn from a multi-thread tokio runtime.
601+
match tokio::runtime::Handle::try_current() {
602+
Ok(handle)
603+
if handle.runtime_flavor()
604+
== tokio::runtime::RuntimeFlavor::MultiThread =>
605+
{
606+
let store = Arc::clone(store);
607+
let object = object.clone();
608+
SpawnedTask::spawn(async move {
609+
fetch_stats_and_ordering(
610+
store.as_ref(),
611+
&object,
612+
metadata_size_hint,
613+
file_decryption_properties,
614+
file_metadata_cache,
615+
&table_schema,
616+
)
617+
.await
618+
})
619+
.join_unwind()
620+
.await
621+
.map_err(|e| DataFusionError::ExecutionJoin(Box::new(e)))?
622+
}
623+
// Otherwise do the work on this task, where the caller polls it
624+
// together with the other files (for example in `ListingTable`'s
625+
// `buffer_unordered`).
626+
_ => {
627+
fetch_stats_and_ordering(
628+
store.as_ref(),
629+
object,
630+
metadata_size_hint,
631+
file_decryption_properties,
632+
file_metadata_cache,
633+
&table_schema,
634+
)
635+
.await
636+
}
637+
}
581638
}
582639

583640
async fn create_physical_plan(

0 commit comments

Comments
 (0)