Skip to content

Commit 432c4d9

Browse files
adriangbclaude
authored andcommitted
bench: cover 1024 columns in parquet_metadata_statistics
The existing column counts top out at 256, which understates the cost of a per-field O(N) leaf lookup. The quadratic term only becomes obvious past a few hundred columns. Co-Authored-By: Claude Opus 5 <[email protected]>
1 parent e87c5f3 commit 432c4d9

1 file changed

Lines changed: 4 additions & 1 deletion

File tree

‎datafusion/datasource-parquet/benches/parquet_metadata_statistics.rs‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,10 @@ fn parquet_metadata_statistics(c: &mut Criterion) {
7272
MetadataState::Mixed,
7373
MetadataState::None,
7474
];
75-
let column_counts = [8, 64, 256];
75+
// 1024 is included because the per-field parquet leaf lookup used to be
76+
// O(N), making the whole per-file statistics pass O(N²) — a term that only
77+
// becomes obvious past a few hundred columns.
78+
let column_counts = [8, 64, 256, 1024];
7679
let row_group_counts = [1, 32, 128];
7780

7881
let mut group = c.benchmark_group("parquet_metadata_statistics");

0 commit comments

Comments
 (0)