…ize (#24384) (#24529)

## Which issue does this PR close?

- Part of #24462
- Backport of #24384 to `branch-55` (for 55.1.0, tracked in #24462).
- Fixes #24383

## Rationale for this change

`UnnestExec` emitted exactly one output batch per input batch, however
many rows the unnesting produced, never consulting
`datafusion.execution.batch_size`. This let downstream operators receive
arbitrarily large batches and made peak memory scale with input batch
size times list length instead of `batch_size`. No API change, so it
fits the backport criteria.

## What changes are included in this PR?

Clean cherry-pick of #24384 (commit
6187d47). No adaptation required.

## Are these changes tested?

Yes. Carries the original regression coverage, all tests pass.

## Are there any user-facing changes?

`UnnestExec` output batches now respect
`datafusion.execution.batch_size` as an upper bound. No API changes.

Co-authored-by: Andy Grove <[email protected]>

…s adaptation (#24125) (#24530)

## Which issue does this PR close?

- Part of #24462
- Backport of #24125 to `branch-55` (for 55.1.0, tracked in #24462).
- Fixes #24109

## Rationale for this change

With `datafusion.execution.parquet.pushdown_filters = true`, a predicate
on a struct field was reported as fully handled by the scan whenever the
file needed schema adaptation, so `FilterExec` was removed from the plan
and the predicate was silently dropped — returning every row instead of
the filtered set. This is a correctness bug (wrong results), not
specific to 55.0.0, so it fits the backport criteria.

## What changes are included in this PR?

Cherry-pick of #24125 (commit 40c208e).
Git's recursive merge auto-resolved surrounding context differences in
`datafusion/physical-expr-adapter/src/schema_rewriter.rs` and
`datafusion/sqllogictest/test_files/parquet_nested_schema_pruning.slt`;
no manual conflict resolution or adaptation of the fix itself was
required.

One follow-up commit adapts a test expectation: `branch-55` doesn't have
#24130/#24315, which taught nested schema pruning to union the leaves
needed by mixed whole-column + field-access reads (e.g. `select s,
s['y'] from narrow`). Without that optimization, the mixed-access case
falls back to reading every physical leaf, so `bytes_scanned` is `219`
here instead of the `146` the original PR's test expects on `main`. This
is a pre-existing difference in pruning capability, not a correctness
regression from this fix.

## Are these changes tested?

Yes. Carries the original regression coverage, all tests pass.

## Are there any user-facing changes?

`WHERE s['field'] = ...` predicates on struct columns now filter
correctly when Parquet filter pushdown requires schema adaptation. No
API changes.

---------

Co-authored-by: Adrian Garcia Badaracco <[email protected]>
Co-authored-by: Claude <[email protected]>

…te_physical_plan (#24690) (#24694)

This PR is a cherry-pick of #24690 onto `branch-55`. See the original PR
for a description of the issue.

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>

…uction (#24723) (#24752)

This is a back port of #24723 onto `branch-55` to support
`datafusion-python` upgrade to 55.1.0. The details can be found in the
linked PR. This is needed for
apache/datafusion-python#1677

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>

… to planned schema in aggregation (#24394) (#24699)

This is a back port of #24394 for `branch-55`.

The original issue is #24069

Note that the original PR was marked as an auto-detected api change, but
I believe that is a false positive.

---------

Co-authored-by: Patrick Ribbsaeter <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>

…ng (#24718) (#24759)

… and new_default (#24582) (#24876)

This is a back port of #24582 into `branch-55` for inclusion in 55.1.0.
Please see the original PR for details.

Co-authored-by: Unik Dahal <[email protected]>

This is a small update to the Cargo.lock file to update

- chacha20 v0.10.0 -> v0.10.2
- h2 v0.4.13 -> v0.4.19

The first was due to a yanked crate. The second was due to a security
vulnerability, RUSTSEC-2026-0258.

The yanked crate is a warning in cargo audit and will auto-resolve on
building to the newer 0.10.2, and the security vulnerability is already
updated in `main`.

…asts (#23169) (#24875)

This PR is a backport of #23169 onto `branch-55`. 

I made one update the `use` statement in `planner.rs`.

Co-authored-by: Dewey Dunnington <[email protected]>
Co-authored-by: Andrew Lamb <[email protected]>

…4670) (#24992)

Backports #24670 to branch-55.

Co-authored-by: Tim Saucer <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>

This PR updates the version number and changelog for the 55.1.0 release.