Record the uncompressed size in file stats when editions allow it - #10923

Summary

Last of a stack. The first two PRs have merged: #10925 fixed the uncompressed size, and #10922 added vortex.uncompressed_size_in_bytes to the preview edition.

The default file statistics now include each field's uncompressed size in bytes, in both the nested statistics (entry_sets/partials) and the legacy ArrayStats statistics. Both are gated by the existing edition check, so the size is written only when the write's enabled editions include vortex.uncompressed_size_in_bytes (currently the preview edition) or when editions are disabled. Default writes, which enable only the core editions, don't store it, and their files are byte-for-byte the same size as before.

DataFusion doesn't use the stat yet: given it as column and table byte sizes, its join selection picked worse plans (see Benchmarks below).

Changes

• vortex-layout file_stats:
  • default_aggregate_fns (renamed from default_pruning_aggregate_fns, since it no longer returns only pruning aggregates) adds UncompressedSizeInBytes. Explicit with_file_statistics lists are unchanged.
  • The legacy statistics now go through the same allows_aggregate filter as the nested ones, and an explicitly requested aggregate the context forbids fails the write in either layout. Every other default aggregate is already in the core editions, so this only changes whether the uncompressed size is recorded.
• vortex-datafusion: column byte_size and table total_byte_size stay Absent rather than coming from Stat::UncompressedSizeInBytes. A doc comment on stats_set_to_df gives the reasons (below), and a test checks that a stats set holding the size still converts to an absent byte_size.
• vortex-duckdb: DuckDB's max_string_length comes from this stat, and it was cast with as u32. A column larger than 4 GiB wrapped into a bound smaller than its longest string. Such a size now gives no bound.
• Tests:
  • vortex-layout: an rstest over allowed/forbidden contexts checks that both layouts record the size only when allowed; forbidden_aggregates_are_gated now expects forbidden defaults to be left out of the legacy stats too.
  • vortex-file: uncompressed_size_follows_editions writes a file with core only, core + preview, and editions disabled, and checks the nested and legacy footer entries in each case.
  • vortex-datafusion: uncompressed_size_is_not_reported_as_byte_size.

API Changes

Files written with the preview edition (or with editions disabled) carry the uncompressed size in both footer layouts. Default writes are unchanged.

Benchmarks

An earlier revision passed the stat to DataFusion. action/bench-all showed DataFusion on Vortex 1.25× slower on TPC-DS SF=1 (46 queries slower) and 1.16× slower on TPC-H SF=1, with Parquet and DuckDB unchanged, and 30 TPC-DS SQL logic test plans changed join order. This revision keeps the stat out of DataFusion and restores the plans; benchmarks are re-running on it.

Why DataFusion gets no byte size

The sizes are accurate: TPC-H q4's filtered orders build side was estimated at 1.5 MB and produced 1.34 MB, and they are close to what Parquet reports (2.07 MB against 1.70 MB for the 7 supplier columns q2 reads; the difference is the 16-byte Utf8View header). The regression comes from two things in DataFusion 55:

1. Join selection threshold. JoinSelection picks CollectLeft below hash_join_single_partition_threshold (1 MiB) when a byte size is known, and below hash_join_single_partition_threshold_rows (128Ki rows) otherwise. Without sizes, 57K–150K row build sides passed the row limit; with sizes, the same 1–7 MB build sides fail the byte limit and become partitioned joins that repartition the probe side (all 3.79M lineitem rows in q4). With a 4 MiB threshold, TPC-H q2, q4 and q22 plan exactly as on develop.
2. ProjectionExprs::project_statistics. It keeps the input's total_byte_size (marked inexact) for any projection with a variable-width column, even when every projected column's byte_size is known. A two-column orders scan reported 243.8 MB instead of 40.5 MB. Summing the projected columns' sizes, as Statistics::with_fetch already does, fixes the inflation but not item 1.

I also tried sizing the scan's partitions from the uncompressed size (a fixed 8 or 32 MiB per partition): join modes don't change, since JoinSelection doesn't look at partition counts, and TPC-DS got slower (+5.6% and +6.9% sum of medians against not reporting sizes).

Correctness check on real data

I converted Parquet to Vortex with the preview edition and compared, per top-level column: the nested footer value as written, the nested value re-read from the file, the legacy ArrayStats value parsed from the raw footer, and the size recomputed from the input arrays.

As a hand check, income_band (20 rows × 3 int32) reports 240 bytes.

Checks

• cargo clippy -p vortex-layout -p vortex-file -p vortex-datafusion --all-targets --all-features -- -D warnings
• cargo +nightly-2026-09-10 fmt -p vortex-layout -p vortex-file -p vortex-datafusion
• cargo test -p vortex-layout --lib (729), cargo test -p vortex-file --all-features (190 + doctests), cargo test -p vortex-datafusion (784 + 14)
• slt::datafusion::tpch/datafusion/{tpch,parquet}.slt and slt::datafusion::tpcds/datafusion/{tpcds,parquet}.slt on SF0.1 data pass with the plans from develop

🤖 Generated with Claude Code

https://claude.ai/code/session_01UeniBBEktquqUp8DYPNJyR