Repository navigation
Improve NDV statistics for Parquet columns (parquet RLE & Dictionary array focus) #24114
Description
Is your feature request related to a problem or challenge?
Deciding whether to dictionary-encode a column requires knowing its cardinality (NDV), Datafusion does support https://docs.rs/datafusion/latest/datafusion/common/struct.ColumnStatistics.html but we need to validate this.
Describe the solution you'd like
Improve NDV statistics collection focused on Parquet, and get these stats into DataFusion in a form the optimizer can actually use (e.g. exposed on ColumnStatistics).
Describe alternatives you've considered
No response
Activity
[-]Improve NDV statistics for Parquet columns[/-][+]Improve NDV statistics for Parquet columns (parquet RLE & Dictionary array focus)[/+][Parquet] Add row group distinct counts to `StatisticsConverter` (#10652
[Parquet] Add row group distinct counts to `StatisticsConverter` (apa…
[Parquet] Implement num distinct values for parquet writer (#10654)
Rich-T-kid commentedon Aug 20, 2026
with apache/arrow-rs#10650 being closed in arrow-rs we just need to wait for datafusion to bump the version so we can use the new methods
[Parquet] Add row group distinct counts to `StatisticsConverter` (apa…
[Parquet] Implement num distinct values for parquet writer (apache#10654
[Parquet] Add row group distinct counts to `StatisticsConverter` (apa…
[Parquet] Implement num distinct values for parquet writer (apache#10654
[Parquet] Add row group distinct counts to `StatisticsConverter` (apa…
[Parquet] Implement num distinct values for parquet writer (apache#10654
Rich-T-kid commentedon Sep 15, 2026
waiting for arrow-rs v60.0.0 fyi
[Parquet] Add row group distinct counts to `StatisticsConverter` (apa…
[Parquet] Implement num distinct values for parquet writer (apache#10654
Rich-T-kid commentedon Sep 20, 2026
take
[Parquet] Add row group distinct counts to `StatisticsConverter` (apa…
[Parquet] Implement num distinct values for parquet writer (apache#10654
[Parquet] Add row group distinct counts to `StatisticsConverter` (apa…
[Parquet] Implement num distinct values for parquet writer (apache#10654
Feat(parquet) : Introduce support for optionally writing Distinct Val…
Feat(parquet) : Introduce support for optionally writing Distinct Val…