Populate and read distinct_count from parquet column statistics

Is your feature request related to a problem or challenge?

There are several open DataFusion issues ((apache/datafusion#24111), (apache/datafusion#24114), (apache/datafusion#22891)) around using cardinality information to drive better query optimization -- things like choosing dictionary encoding for low-cardinality string columns, improving join ordering, and producing more accurate NDV estimates for aggregations. All of these depend on the optimizer having access to distinct value counts per column at planning time.

In DataFusion, this information flows in through TableProvider::statistics(). When a parquet-backed table is scanned, DataFusion calls through ParquetFormat::infer_stats_and_ordering() into statistics_from_parquet_metadata() in datasource-parquet/src/metadata.rs, which uses arrow-rs's StatisticsConverter to read per-column statistics out of the parquet file footer and build ColumnStatistics. This all happens before any data pages are read, it's purely metadata.

The problem is that StatisticsConverter exposes row_group_mins, row_group_maxes, and row_group_null_counts, but has no equivalent for distinct_count. So ColumnStatistics::distinct_count is always Precision::Absent for parquet files, regardless of whether the file actually contains that information. The optimizer never sees cardinality data for any parquet column, and the DataFusion issues above are blocked on it.

arrow-rs (currently) doesn't write this meta data but other writers may and we aren't taking advantage of it.

Describe the solution you'd like

Read side : targeting arrow-rs 60

The most impactful and lowest-risk fix is on the read path. StatisticsConverter in parquet/src/arrow/arrow_reader/statistics.rs needs a row_group_distinct_counts() method that follows the same pattern as row_group_null_counts(), reading distinct_count_opt() from each row group's column chunk metadata and returning a UInt64Array with nulls where the value is absent.

This is the right place to start because parquet files written by other implementations, I.E parquet-java, parquet-cpp, may already have distinct_count populated. Wiring up the read path gives DataFusion immediate value from those files with no write-side changes required and no behavioral restrictions.

Once the method exists in arrow-rs, the DataFusion change is straightforward: summarize_column_statistics() in datasource-parquet/src/metadata.rs calls the new method and uses it to populate ColumnStatistics::distinct_count as Precision::Exact when all row groups have the value, or Precision::Inexact when it's only partially available.

Write side

The ArrowWriter currently never populates distinct_count in the statistics it writes. The field exists (column_distinct_count: Option in ColumnMetrics in parquet/src/column/writer/mod.rs) but only the low-level write_batch_with_statistics() API can reach it — the standard ArrowWriter path never fills it in.

Doing this accurately requires some form of counting per column per row group. A HashSet gives exact values but memory usage grows with cardinality, which gets expensive for high-cardinality string columns across large row groups. A HyperLogLog gives approximate counts with bounded memory but introduces estimation error that would need to surface as Precision::Inexact. The right tradeoff isn't obvious and we'd want to benchmark write overhead before settling on an approach.

This could also be opt-in through WriterProperties rather than on by default, since not every parquet writer needs it and the overhead isn't free.

The write side is worth doing but shouldn't hold up the read path, which is the cleaner immediate win with no restrictions.