Expose PruningPredicate's IN-list rewrite size limit as a config option #24059

Describe the bug or feature

Is your feature request related to a problem or challenge?

PredicateRewriter::rewrite_predicate_to_statistics_predicate (and PruningPredicate::try_new) treats col IN (v1..vn) specially: when n <= MAX_LIST_VALUE_SIZE_REWRITE (currently a hardcoded 20), the IN list is expanded into a chain of per-value min/max statistics checks; otherwise it falls back to unhandled_hook, which by default returns the constant TRUE — so row-group / file-range statistics pruning does not fire at all for IN lists longer than 20.

The 20 limit was introduced by #8815 as a "minor extract-const" refactor and, before that, was a bare literal inside build_predicate_expression. Neither change discusses why 20 specifically was chosen, and there is no way at runtime for a query engine to opt into a higher (or lower) cap.

Real-world impact

Query patterns that pass a batch of identifiers as col IN (...) are common — REST endpoints that filter by a page of ~25-100 tickers/customer IDs/document IDs, ORM-generated WHERE id IN (25 items) queries, and batched crawlers. On a table whose leading sort column matches the filter column (a very common physical layout for a view intended to serve exactly these queries), row-group and file-range statistics pruning would eliminate the vast majority of containers cheaply — but the current 20 cap forces the reader to fall back to row-level filter pushdown on every surviving row group, which materializes the filter column across the full logical row count.

Example: on a table sorted by `col`, an `IN (25 items)` query on `col`:

• Today: `row_groups_pruned_statistics = 0 / N` (all row groups survive), every row group's `col` chunk is fetched and decoded, matches identified only during row-level pushdown.
• With a higher cap: the IN would rewrite into a 25-way OR of range checks, most row groups would be pruned before their column chunks are fetched, only a small handful decoded.

Also affects file-level pruning where the same PruningPredicate is evaluated against file-range stats before opening the parquet footer.

Describe the solution you'd like

Expose the cap as a runtime config option:

• Add datafusion.execution.parquet.pruning_max_in_list_size: usize (default 20, preserving existing behaviour) to TableParquetOptions.global.
• Add PredicateRewriter::with_max_in_list_size(usize) -> Self builder method, mirroring the existing with_unhandled_hook.
• Add PruningPredicate::try_new_with_max_in_list_size(expr, schema, max_in_list_size) variant so callers can opt in without going through PredicateRewriter.
• Add build_pruning_predicate_with_max_in_list_size(...) variant for the public helper.
• Wire the parquet source / opener to read the config value and pass it through when building row-group + file-range pruning predicates.

The internal signature of build_predicate_expression gets an extra max_in_list_size: usize argument (private crate-local, safe to break).

Existing PruningPredicate::try_new and build_pruning_predicate are preserved as thin wrappers that pass MAX_LIST_VALUE_SIZE_REWRITE (which becomes pub const so callers can reference the default), so no downstream API breakage.

Describe alternatives you've considered

1. Just bump the default. Simpler, but takes away the ability to opt out for engines that measured 20 as their sweet spot.
2. Make the constant pub without a config knob. Callers would have to fork or wrap the predicate rewriter; doesn't compose with SessionConfig.
3. Rewrite IN → OR at the caller side before passing to PruningPredicate::try_new. Works, but every downstream engine reinvents the wheel and pays the cost of not knowing DF's own internal 20.

Additional context

Willing to submit a PR. Change is mechanical:

• Add config option to datafusion/common/src/config.rs.
• Thread max_in_list_size: usize through build_predicate_expression, PredicateRewriter, PruningPredicate::try_new, and build_pruning_predicate (backward-compatible wrappers).
• Wire it through datasource-parquet alongside the existing max_predicate_cache_size field on ParquetMorselizer / PreparedParquetOpen.
• Tests: unit tests on PredicateRewriter at cap=0/20/32 verifying rewrite behaviour + an integration test showing row-group pruning for a col IN (>20 items) predicate against a sorted table.