)

# Which issue does this PR close?

- Closes apache#8608.
- Closes apache#10650

# Rationale for this change

see apache#8608, apache#10650 & apache/datafusion#24114

# What changes are included in this PR?
Adds `set_write_row_group_number_distinct_values` to WriterProperties,
which when enabled causes the ArrowWriter to track the exact number of
distinct non-null values per column across the full row group and write
it into the column chunk statistics footer as `distinct_count`. Tracking
is implemented by hashing each non-null value using XxHash64 into a
per-column HashSet<u64> that persists across batch writes and is
finalized in `close()`. **The flag defaults to false so there is no
impact on existing writers.**

# Are these changes tested?
yes, see test.

# Are there any user-facing changes?
yes, users will not be able to write `distinct_count` meta data to
parquet files.

)

# Which issue does this PR close?

- Closes apache#8608.
- Closes apache#10650

# Rationale for this change

see apache#8608, apache#10650 & apache/datafusion#24114

# What changes are included in this PR?
Adds `set_write_row_group_number_distinct_values` to WriterProperties,
which when enabled causes the ArrowWriter to track the exact number of
distinct non-null values per column across the full row group and write
it into the column chunk statistics footer as `distinct_count`. Tracking
is implemented by hashing each non-null value using XxHash64 into a
per-column HashSet<u64> that persists across batch writes and is
finalized in `close()`. **The flag defaults to false so there is no
impact on existing writers.**

# Are these changes tested?
yes, see test.

# Are there any user-facing changes?
yes, users will not be able to write `distinct_count` meta data to
parquet files.

)

# Which issue does this PR close?

- Closes apache#8608.
- Closes apache#10650

# Rationale for this change

see apache#8608, apache#10650 & apache/datafusion#24114

# What changes are included in this PR?
Adds `set_write_row_group_number_distinct_values` to WriterProperties,
which when enabled causes the ArrowWriter to track the exact number of
distinct non-null values per column across the full row group and write
it into the column chunk statistics footer as `distinct_count`. Tracking
is implemented by hashing each non-null value using XxHash64 into a
per-column HashSet<u64> that persists across batch writes and is
finalized in `close()`. **The flag defaults to false so there is no
impact on existing writers.**

# Are these changes tested?
yes, see test.

# Are there any user-facing changes?
yes, users will not be able to write `distinct_count` meta data to
parquet files.

)

# Which issue does this PR close?

- Closes apache#8608.
- Closes apache#10650

# Rationale for this change

see apache#8608, apache#10650 & apache/datafusion#24114

# What changes are included in this PR?
Adds `set_write_row_group_number_distinct_values` to WriterProperties,
which when enabled causes the ArrowWriter to track the exact number of
distinct non-null values per column across the full row group and write
it into the column chunk statistics footer as `distinct_count`. Tracking
is implemented by hashing each non-null value using XxHash64 into a
per-column HashSet<u64> that persists across batch writes and is
finalized in `close()`. **The flag defaults to false so there is no
impact on existing writers.**

# Are these changes tested?
yes, see test.

# Are there any user-facing changes?
yes, users will not be able to write `distinct_count` meta data to
parquet files.

)

# Which issue does this PR close?

- Closes apache#8608.
- Closes apache#10650

# Rationale for this change

see apache#8608, apache#10650 & apache/datafusion#24114

# What changes are included in this PR?
Adds `set_write_row_group_number_distinct_values` to WriterProperties,
which when enabled causes the ArrowWriter to track the exact number of
distinct non-null values per column across the full row group and write
it into the column chunk statistics footer as `distinct_count`. Tracking
is implemented by hashing each non-null value using XxHash64 into a
per-column HashSet<u64> that persists across batch writes and is
finalized in `close()`. **The flag defaults to false so there is no
impact on existing writers.**

# Are these changes tested?
yes, see test.

# Are there any user-facing changes?
yes, users will not be able to write `distinct_count` meta data to
parquet files.

# Which issue does this PR close?

- Closes #10804..

# Rationale for this change

set_write_row_group_number_distinct_values was added in #10654 with no
benchmarks. This PR adds them so the cost of NDV tracking is visible and
regressions are catchable going forward.

# What changes are included in this PR?

Adds a `bench_ndv` benchmark group to `parquet/benches/arrow_writer.rs`
covering the data types that NDV tracking supports: Int32 (primitive),
Boolean, Utf8, LargeBinary, and FixedSizeBinary(16).

Each group writes 10 batches of 65,536 rows through a single ArrowWriter
with `set_write_row_group_number_distinct_values(true)`. Two cardinality
shapes are exercised per type: low (100 distinct values, cycling) and
high (all unique, random), since hash-set behaviour differs meaningfully
between the two.

# Are these changes tested?
n/a

# Are there any user-facing changes?
n/a
cargo bench --bench arrow_writer --features arrow/test_utils -- "ndv/"

)

# Which issue does this PR close?

- Closes apache#8608.
- Closes apache#10650

# Rationale for this change

see apache#8608, apache#10650 & apache/datafusion#24114

# What changes are included in this PR?
Adds `set_write_row_group_number_distinct_values` to WriterProperties,
which when enabled causes the ArrowWriter to track the exact number of
distinct non-null values per column across the full row group and write
it into the column chunk statistics footer as `distinct_count`. Tracking
is implemented by hashing each non-null value using XxHash64 into a
per-column HashSet<u64> that persists across batch writes and is
finalized in `close()`. **The flag defaults to false so there is no
impact on existing writers.**

# Are these changes tested?
yes, see test.

# Are there any user-facing changes?
yes, users will not be able to write `distinct_count` meta data to
parquet files.

…ues to statistics (apache#25576)

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes apache#123` indicates that this PR will close issue apache#123.
-->

- Closes apache#24114.

## Rationale for this change

DataFusion can use NDV (number of distinct values) from Parquet row
group statistics to power optimizations like COUNT(DISTINCT) aggregation
pushdown and cardinality estimation (apache#24111) . However, currently the
Parquet writers doesn't emit NDV by default, so these statistics are
absent for roundtrip queries.

Two recent arrow-rs additions make it possible to close this gap
end-to-end:
- [arrow-rs apache#10652 —
`StatisticsConverter::row_group_distinct_counts()`](apache/arrow-rs#10652)
: a typed API for reading NDV from row group metadata
- [arrow-rs apache#10654 —
`WriterPropertiesBuilder::set_write_row_group_number_distinct_values()`](apache/arrow-rs#10654):
a writer-side option to track and emit NDV during write

This PR wires both into DataFusion so that users can write parquet files
with NDV and have DataFusion read them back correctly.

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.

Please explain the problem you are trying to solve in terms of the
user-visible
behavior, rather than the implementation.

For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of
the
implementation. "COUNT(DISTINCT) returns wrong results when the column
contains
nulls" is the user-visible problem.
-->

## What changes are included in this PR?

- summarize_distinct_counts now calls
`StatisticsConverter::row_group_distinct_counts()` instead of manually
walking column chunk statistics.
- Adds
`datafusion.execution.parquet.write_row_group_number_distinct_values`
(bool, default false) to ParquetOptions. When enabled, the ArrowWriter
hashes every non-null value per column per row group and writes the
distinct count into the Parquet statistics footer.

<!--
There is no need to duplicate the description in the issue here, but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## What is the testing strategy for this PR?

- The NDV read change is covered by the 8 existing unit tests in
metadata.rs (test_distinct_count_*), which all continue to pass.
- The write config is tested by
sqllogictest/test_files/parquet_ndv_write.slt, which writes 4 rows with
3 distinct values and verifies:
- Without the option: Distinct is absent from DataSourceExec statistics
- With the option: Distinct=Exact(3) appears in DataSourceExec
statistics


<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

Briefly describe how this PR is tested, and point to the specific tests
you added. For example: 'This new feature is covered by the
`sqllogictest` cases added in `foo.slt`'.

If this PR does not add tests, explain why. For example, if the change
is already covered by existing tests, please mention it.

You should also check the `codecov` bot reply on this PR to confirm the
changed code is exercised.
-->

## Are there any user-facing changes?
yes, new `write_row_group_number_distinct_values` to parquet config
options.
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.

If there are any breaking changes to public APIs, please add the `api
change` label.
-->

…ues to statistics (apache#25576)

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes apache#123` indicates that this PR will close issue apache#123.
-->

- Closes apache#24114.

## Rationale for this change

DataFusion can use NDV (number of distinct values) from Parquet row
group statistics to power optimizations like COUNT(DISTINCT) aggregation
pushdown and cardinality estimation (apache#24111) . However, currently the
Parquet writers doesn't emit NDV by default, so these statistics are
absent for roundtrip queries.

Two recent arrow-rs additions make it possible to close this gap
end-to-end:
- [arrow-rs apache#10652 —
`StatisticsConverter::row_group_distinct_counts()`](apache/arrow-rs#10652)
: a typed API for reading NDV from row group metadata
- [arrow-rs apache#10654 —
`WriterPropertiesBuilder::set_write_row_group_number_distinct_values()`](apache/arrow-rs#10654):
a writer-side option to track and emit NDV during write

This PR wires both into DataFusion so that users can write parquet files
with NDV and have DataFusion read them back correctly.

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.

Please explain the problem you are trying to solve in terms of the
user-visible
behavior, rather than the implementation.

For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of
the
implementation. "COUNT(DISTINCT) returns wrong results when the column
contains
nulls" is the user-visible problem.
-->

## What changes are included in this PR?

- summarize_distinct_counts now calls
`StatisticsConverter::row_group_distinct_counts()` instead of manually
walking column chunk statistics.
- Adds
`datafusion.execution.parquet.write_row_group_number_distinct_values`
(bool, default false) to ParquetOptions. When enabled, the ArrowWriter
hashes every non-null value per column per row group and writes the
distinct count into the Parquet statistics footer.

<!--
There is no need to duplicate the description in the issue here, but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## What is the testing strategy for this PR?

- The NDV read change is covered by the 8 existing unit tests in
metadata.rs (test_distinct_count_*), which all continue to pass.
- The write config is tested by
sqllogictest/test_files/parquet_ndv_write.slt, which writes 4 rows with
3 distinct values and verifies:
- Without the option: Distinct is absent from DataSourceExec statistics
- With the option: Distinct=Exact(3) appears in DataSourceExec
statistics


<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

Briefly describe how this PR is tested, and point to the specific tests
you added. For example: 'This new feature is covered by the
`sqllogictest` cases added in `foo.slt`'.

If this PR does not add tests, explain why. For example, if the change
is already covered by existing tests, please mention it.

You should also check the `codecov` bot reply on this PR to confirm the
changed code is exercised.
-->

## Are there any user-facing changes?
yes, new `write_row_group_number_distinct_values` to parquet config
options.
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.

If there are any breaking changes to public APIs, please add the `api
change` label.
-->