Add benchmarks for queries against wide schema parquet files

@HippoBaro

There is some great work going on in arrow-rs (apache/arrow-rs#9722) by @HippoBaro to make wide schemas (schemas with many columns) more performant.

We have similarly wide data and spend a lot of time in our queries loading parquet metadata. I spent some time investigating this today and came to the conclusion that not only is it expensive to load wide schemas, they also introduce overhead in the rest of the query. Below is a summary of the analysis (AI generated). This makes me think we need to tackle the problem from the DataFusion side as well, with the first step being adding a benchmark for wide schema queries.

TL;DR

A simple selective query over 339 Parquet files runs ~11× slower when the
files carry hundreds of columns the query never touches, even though every
unused column is correctly projected away and every unused row group is
correctly pruned.

metadata_load_time is the cleanest signal but only ~10% of the wall-clock
overhead. The rest sits in downstream per-file setup that scales with schema
width — most of it not explicitly bucketed by EXPLAIN ANALYZE.

Setup

Two copies of the same 339 Parquet files, same rows, same row groups.
wide: full production schema (681 unique schema paths, 163,926 column
chunks across the dataset).
narrow: only the 4 columns the query references (4 schema paths, 1,356
column chunks).
Engine: datafusion-cli 53.0.0, local disk, hot OS cache, 12-way scan
parallelism.
Query:
```
SELECT start_timestamp, trace_id
FROM '<path>/*.parquet'
WHERE service_name = 'api' AND span_name = 'event';
```
Returns 1,287 rows out of ~12.1 M scanned. Pruning behaviour is identical
on both datasets: 350→350 files matched on stats, 339→339 row groups
matched on stats, 339→320 row groups after bloom-filter pruning.

Wall time (3 hot runs each)

Dataset	run 1	run 2	run 3	median
narrow	0.053 s	0.046 s	0.049 s	0.049 s
wide	0.548 s	0.556 s	0.557 s	0.557 s

EXPLAIN ANALYZE — DataSourceExec metrics

Sums across 12 scan tasks. Same output_bytes=1266.2 MB, same final row
count, identical pruning.

Phase	narrow	wide	Δ
metadata_load_time	8.4 ms	580 ms	+572 ms
time_elapsed_opening	27 ms	161 ms	+134 ms
time_elapsed_scanning_until_data	308 ms	526 ms	+218 ms
time_elapsed_processing	307 ms	737 ms	+430 ms
time_elapsed_scanning_total	391 ms	755 ms	+364 ms
bloom_filter_eval_time	349 ms	284 ms	-65 ms
statistics_eval_time	4.6 ms	7.0 ms	+2.4 ms
page_index_eval_time	13 µs	14 µs	~0
FilterExec elapsed_compute	17.7 ms	17.3 ms	~0

Where the cost lands

Wall-clock delta is +508 ms going narrow → wide. With ~12-way parallelism,
dividing the cumulative metrics by 12 gives a rough wall-clock estimate per
phase:

Phase	Cumulative Δ	~Wall Δ
metadata_load_time	+572 ms	~48 ms
time_elapsed_opening	+134 ms	~11 ms
time_elapsed_processing (excl. metadata + opening)	~+296 ms	~25 ms
Unattributed (serial setup, expression binding against wide schema, scan-task dispatch)	—	~420 ms

Observations

metadata_load_time scales linearly with column-chunk count. ~3.5 µs
per chunk in both datasets (8 ms / 1,356 ≈ 580 ms / 163,926). The
per-chunk cost is layout-independent; what changes is how many chunks the
footer describes.
The wide schema keeps costing after metadata is loaded.
time_elapsed_processing more than doubles and time_elapsed_scanning_until_data
grows by 218 ms even though the projected columns are identical. That is
per-column reader-state construction, schema-descriptor traversal, and
predicate binding against a 681-element schema running per file before
the first useful byte flows.
Predicate evaluation phases are flat. bloom_filter_eval_time,
statistics_eval_time, page_index_eval_time, FilterExec.elapsed_compute
— none of them respond to schema width. Once row groups are selected, the
unused columns do not participate.
Most of the wall-clock gap is unattributed by EXPLAIN ANALYZE. The
bucketed phases sum to ~85 ms wall; the observed delta is 508 ms wall.
The remaining ~420 ms is serial coordinator/planner work that scales with
schema width and is not broken out in current metrics. On object storage
this is likely amplified further because every file open also pays a
network round-trip proportional to footer size.

Conclusion

The narrow result (~49 ms) is roughly what this query should cost. Anything
above that is the price of carrying schema descriptions for columns the query
never touched through the planner and scanner.

I was able to reproduce this using data derived from TPCH data. Essentially I copied the data row wise and column wise to produce 128 files with 1024 columns. Then ran a simple query against them. I was able to reproduce a 7x slowdown with criterion warming up (so metadata is cached) and a 30x slowdown on cold datafusin-cli runs.

I'm hoping this will be a good target for profiling + systematically fixing the places we are no efficient on wide schemas.

PR is ready for review #21970

@adriangb

Thank you @adriangb. I also have a similar setup, which I’ve used locally to identify potential improvement opportunities. I’m currently working my way up the stack, from Parquet to Arrow, and eventually DataFusion.

I do have some local DataFusion patches that make a big difference locally, but they’re untested and very much not ready for upstream yet 😕

Very happy to hear you're working your way up the stack!

I'm curious if these benchmarks align with your local setup / patches? It'd be useful if we can merge some high signal benchmarks as a first step. Then if you'd be open to at least put draft PRs up we can start looking into how to get them in shape to upstream 😄

@adriangb

Exploration was human-directed; the writeup below is AI-generated.
Driver: @adriangb. The accompanying changes live in two draft PRs
(not meant to land as-is — purely for visibility while the
investigation continues):

DataFusion: perf: resolve parquet leaf columns in O(1) during statistics collection #21987

arrow-rs: parquet: drop the leaf scan in parquet_column; add reusable ArrowReaderMetadata primitives arrow-rs#9882

Wide-schema parquet read perf — investigation

Setup

I generated the wide-schema benchmark suite (now committed on the
draft branch as benchmarks/sql_benchmarks/wide_schema/): two copies
of the same 256 parquet files / 50 k rows each — one with 8
columns, one with 1024 columns. Identical row counts, identical
file counts, identical bytes scanned. Q04 is the most stressful:

SELECT id, ts FROM events
WHERE category = 'c0' AND flag = 'f0' AND id % 1000 = 0;

Profiled with samply, all hotspots resolved with --unstable-presymbolicate.

Worst offenders identified

Default 50 MB metadata cache thrashes badly. Per-file
ParquetMetaData is ~1.5 MB (page index dominates), dataset total
~400 MB. The default fits ~30 of 256 files — enough to make
threads contend on the cache mutex, not enough to stop
re-parsing. Cache-size sweep showed 50 MB is the worst regime:
disabling the cache (0) is faster (~100 ms hot vs ~136 ms hot at
50 MB) because the lock contention disappears. Sizing at 400 MB+
(everything fits) doubles hot performance to ~55 ms.
statistics_from_parquet_metadata was O(N²) per file. It
iterated every logical field and called StatisticsConverter::try_new,
which itself did an O(N) scan in parquet_column (the comment
said "this could be made more efficient (#TBD)"). 1024² × 256 ≈
268 M ops/query just to set up stats.
ArrowReaderMetadata::try_new runs per file open. Walks every
parquet leaf to build the arrow Schema + dremel ParquetField.
~190 µs/file at 1024 cols. With 256 files / 12 threads = ~4 ms
wall per query.
Coercion forces a second arrow-schema rebuild. With
schema_force_view_types=true (default), apply_file_schema_type_coercions
returns a new schema ⇒ try_new runs again per file. Another
~190 µs/file.
apply_file_schema_type_coercions always built a 1024-entry
HashMap even when the early-return condition would discard it.
DefaultFilesMetadataCache::evict_entries called full
structural memory_size() walks on every put / remove / eviction.
collect_statistics=true (default) is the cold-path tax.
Eagerly fetches metadata + computes per-column stats for every
file at planning time, even when the query touches only 4 of 1024
columns. For Q04 the inferred stats are then unused
(files_ranges_pruned_statistics: 267 → 267).

Fixes shipped (on the draft branches)

arrow-rs (#9882)

SchemaDescriptor::root_to_first_leaf: precomputed at construction
⇒ parquet_column is now O(1).
StatisticsConverter::from_arrow_field: low-overhead constructor
taking a resolved (field, leaf_idx) pair.
ArrowReaderMetadata::from_field_levels: package precomputed parts
so callers can skip the per-leaf walk.
parquet_to_arrow_schema_and_field_levels: produces (Schema, FieldLevels) in one walk.
AsyncFileReader::get_arrow_reader_metadata: new trait method (default
delegates to try_new) so cache-aware readers can short-circuit.
load_async now goes through it.
Public accessors on ArrowReaderOptions so callers can decide
whether their cached arrow view applies.

DataFusion (#21987)

statistics_from_parquet_metadata: precompute logical → parquet
leaf indices once (O(N)) and use from_arrow_field in the loop.
O(N²)/file → O(N)/file.
CachedParquetMetaData carries a OnceLock<ArrowReaderMetadata>
(cache hit ⇒ Arc::clone, ~4 ns) plus a single-slot
Mutex<Option<(supplied_schema_ptr, ArrowReaderMetadata)>> for the
post-coercion build.
CachedParquetFileReader::get_arrow_reader_metadata overrides the
new trait method and serves both base and post-coercion ARMs from
cache.
prepare_filters made async so the post-coercion rebuild also
routes through the cache-aware reader.
apply_file_schema_type_coercions: cheap "any view/string?" first
pass (only build the lookup HashMap when needed); return None when
nothing actually changed.
DefaultFilesMetadataCache stores memory_size next to each entry —
no more re-walks on put / evict / remove.
New wide_schema_microbench covering try_new vs clone_cached,
apply_file_schema_type_coercions no-op, PruningPredicate::try_new,
StatisticsConverter::try_new vs from_arrow_field.

Results (Q04, 1024 cols × 256 files, profiling build)

Scenario	Before	After	Δ
narrow control, hot	~25 ms	~24 ms	~0
wide @50 M cache (default), cold	~1010 ms	~615 ms	−39%
wide @50 M cache (default), hot	~108 ms	~87 ms	−19%
wide @50 M, `collect_statistics=false`, cold	~1010 ms	~257 ms	−75%

For reference (warm-fitting cache):

wide @2 G cold: 830 → 510 ms (−39%)
wide @2 G hot: 47 → 38 ms (−19%)

Microbench numbers (from the new `wide_schema_microbench`)

Cols	`ArrowReaderMetadata::try_new`	`clone_cached`	Speedup
8	2.0 µs	4.4 ns	450×
64	12 µs	4.4 ns	2700×
256	47 µs	4.4 ns	10700×
1024	190 µs	4.4 ns	43000×

try_new is O(N) at ~190 ns/col; the cached clone path is O(1).
PruningPredicate::try_new against a 1-column predicate is already
~2-3 µs and does not scale with schema width — predicate-driven,
not schema-driven, so that part is already where we want it.

Open / next

The collect_statistics=true default eats most of the cold-path
budget. The structural fix is lazy / projection-aware
Statistics::column_statistics so the optimizer only pays for
the columns it actually inspects. That'd let the default stay safe
without paying for unused columns.
The page-index-skip experiment helped cold (−12-17%) but regressed
warm because load_page_index doesn't update the cache. The fix
is to also persist page-index-loaded metadata back into the cache.
Schema::index_of / Fields::find are O(N) linear scans in arrow.
Profile shows them at <1% on Q04 — small now, but they're in
enough hot loops that adding a lazy name-index hashmap to Fields
would future-proof anything that does name lookups in inner loops.

Full investigation log lives at report.md on the DataFusion branch
(direct link).

Add wide-schema benchmark suite for measuring per-file metadata overh…

Wide-schema perf — PR breakdown

The investigation work is now split into focused PRs:

arrow-rs (primitives):

parquet: drop the leaf scan in parquet_column; add reusable ArrowReaderMetadata primitives arrow-rs#9882 — O(1) parquet_column, parquet_to_arrow_schema_and_field_levels + ArrowReaderMetadata::from_field_levels, StatisticsConverter::from_arrow_field, AsyncFileReader::get_arrow_reader_metadata. Ready for review.

DataFusion:

Reduce per-file metadata overhead for wide-schema parquet scans #22829 — the arrow-rs-independent wins: cache entry memory_size in DefaultFilesMetadataCache, and apply_file_schema_type_coercions early-return. Mergeable now.
parquet: cache per-file ArrowReaderMetadata for wide-schema scans #22830 (draft) — cache per-file ArrowReaderMetadata across opens. Depends on arrow-rs#9882; blocked on a build until those primitives ship in a release DataFusion bumps to (DataFusion main is on arrow 58.3.0, the primitives are on arrow-rs main/59.0.0).

The original statistics_from_parquet_metadata O(N²)→O(N) idea is not included — upstream #22462 already refactored that path, and combined with arrow-rs#9882's O(1) parquet_column it is already O(N).

The earlier all-in-one draft #21987 is superseded by the split above.

bench: make wide_schema honor DATA_DIR like the other sql_benchmarks (a…

perf: avoid building a DFSchema for every pruning comparison (apache#…

perf: resolve parquet leaf columns in O(1) during statistics collecti…

perf: avoid building a DFSchema for every pruning comparison (apache#…

perf: resolve parquet leaf columns in O(1) during statistics collecti…

perf: decode Parquet footers for statistics in parallel during planni…

Navigation Menu

Repository navigation

Add benchmarks for queries against wide schema parquet files #21968

TL;DR

Setup

Wall time (3 hot runs each)

EXPLAIN ANALYZE — DataSourceExec metrics

Where the cost lands

Observations

Conclusion

Wide-schema parquet read perf — investigation

Setup

Worst offenders identified

Fixes shipped (on the draft branches)

Results (Q04, 1024 cols × 256 files, profiling build)

Microbench numbers (from the new `wide_schema_microbench`)

Open / next

Wide-schema perf — PR breakdown

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add benchmarks for queries against wide schema parquet files #21968

Description

TL;DR

Setup

Wall time (3 hot runs each)

EXPLAIN ANALYZE — DataSourceExec metrics

Where the cost lands

Observations

Conclusion

Activity

adriangb commented on May 1, 2026

adriangb commented on May 1, 2026

HippoBaro commented on May 1, 2026

adriangb commented on May 1, 2026

adriangb commented on May 2, 2026

Wide-schema parquet read perf — investigation

Setup

Worst offenders identified

Fixes shipped (on the draft branches)

Results (Q04, 1024 cols × 256 files, profiling build)

Microbench numbers (from the new wide_schema_microbench)

Open / next

adriangb commented on Jun 8, 2026

Wide-schema perf — PR breakdown

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Microbench numbers (from the new `wide_schema_microbench`)