[arrow] Fix pushdown errors - #27319

…onaries (duckdb#27626)

Fixes Arrow scans of lists whose first chunk holds no elements and of
fixed-size lists with a run-end encoded child, and adds tests for
run-end encoded filters and NULL indices in dictionaries of unions.

Combines the PRs labeled `Ready To Merge` in
https://github.com/duckdb-bot/duckdb/pulls below, on top of the current
head of `main`. The changes of each PR are applied as one squashed
commit.

duckdb-bot#36 removes the early return for lists without elements
in `ArrowToDuckDBList`, which duckdb-bot#144 changed to skip
run-end encoded children. Removing it covers both, and the tests of both
PRs are kept.

## Changes applied (4 PRs)

- duckdb-bot#36: Scan an Arrow list with an encoded child when
its first chunk holds no elements
- duckdb-bot#144: Decode a run-end encoded child of a fixed-size
list in an Arrow scan
- duckdb-bot#133: Test Arrow scan filters on tables with run-end
encoded strings and integers
- duckdb-bot#103: Test VARCHAR casts, union_tag and union_extract
with NULL indices in Arrow dictionaries of unions

## Original descriptions

<details><summary>duckdb-bot#36: Scan an Arrow list with an
encoded child when its first chunk holds no elements</summary>

Moved from duckdb#27356.

Scanning an Arrow `list<dictionary<int32, string>>` column failed with
`INTERNAL Error: Attempted to dereference unique_ptr that is NULL!` when
the first 2048 rows had no list elements and later rows did, for example
2048 empty lists followed by `['a']`. This was reported from Python
(pyarrow) in duckdb#26272, including on v1.5.6. In DuckDB-WASM,
it invalidated the database.

`ArrowToDuckDBList` had a shortcut for chunks containing only empty
lists starting at child offset 0. It converted the child as a plain
column, guarded by `D_ASSERT(!child_array.dictionary)` only in debug
builds. A dictionary child’s Arrow type describes its indices and
carries no string type information, so VARCHAR conversion dereferenced a
null `ArrowStringInfo`. The shortcut also skipped the run-end encoded
path.

This removes the shortcut; the general path already handles empty slices
of plain, dictionary and run-end encoded children. New tests scan lists
with an all-empty first chunk: string dictionary values in
`test/arrow/arrow_dict_nested_offset.cpp` and a run-end encoded child in
`test/arrow/arrow_run_end_encoded.cpp`. The dictionary test fails on
`main`; both pass with the change. Arrow tests pass on reldebug and
relassert with sanitizers. All reldebug unit tests pass except failures
also seen on `main`. Another contributor’s open duckdb#26394
removes the same shortcut; this PR adds tests for the reported string
dictionary case and run-end encoding.

Fixes duckdblabs/bot-issue-tracker#76
Related: duckdb#26272

</details>

<details><summary>duckdb-bot#144: Decode a run-end encoded child
of a fixed-size list in an Arrow scan</summary>

We found that scanning Arrow data with a fixed-size list column (`+w:N`)
whose child is run-end encoded (`+r`), for example through
`duckdb_arrow_scan`, returned wrong values or crashed with a
segmentation fault. The column binds as the values' type, such as
`INTEGER[2]`, so nothing fails before the scan.

In `src/function/table/arrow_conversion.cpp`, `ArrowToDuckDBArray` only
checked `child_array.dictionary` and otherwise called
`ColumnArrowToDuckDB`. This read the run-end encoded child, a struct of
run ends and values with a single buffer, as a plain array.
`ArrowToDuckDBList` already dispatches on the child's physical type and
decodes this child with `ColumnArrowToDuckDBRunEndEncoded`.

`ArrowToDuckDBArray` now uses `ColumnArrowToDuckDBRunEndEncoded` and
reapplies parent-row NULLs to the child afterward, because decoding sets
every child row's validity. Empty-child shortcuts in
`ArrowToDuckDBArray` and `ArrowToDuckDBList` also send run-end encoded
children to the decoder with zero rows, avoiding plain conversion's read
of a values buffer those arrays do not have. Tests in
`test/arrow/arrow_run_end_encoded.cpp` cover a small `FIXED_SIZE_LIST`
with a run-end encoded child (wrong values on `main`), one crossing
vector chunks with an array offset and a NULL run (crashes on `main`),
and NULL rows over values outside the `TINYINT` range cast in a query
over the scan (fails on `main`). Tests also cover `LIST` and
`LIST(FIXED_SIZE_LIST)` columns containing only empty lists; these pass
on `main` too because the extra read happens to stay inside the test's
arrays. The `[arrow]` tests pass on reldebug and on relassert with
sanitizers. All unit tests pass on reldebug except failures also present
on `main` on this machine. No benchmark was run: the change adds one
branch per vector of a fixed-size list. A run-end encoded values child
of a run-end encoded array, and ancestor offsets lost through nested
structs, remain unaddressed because they use separate paths. We found no
earlier pull requests in duckdb/duckdb.

Fixes duckdblabs/bot-issue-tracker#580

</details>

<details><summary>duckdb-bot#133: Test Arrow scan filters on
tables with run-end encoded strings and integers</summary>

Filtering an Arrow table with run-end encoded strings failed during
planning with `INTERNAL Error: Failed to cast ArrowTypeInfo, type
mismatch (expected: STRING, got: STRUCT)`, even on a plain column:
`SELECT i FROM t WHERE i > 1`. Unfiltered selects worked.
duckdb#27319 fixed this on main, so this pull request now only
adds tests. duckdb/duckdb-python#666 reports the same bug.

A run-end encoded column uses its values' DuckDB type, but its type info
holds both run ends and values. The pushdown check in
`src/function/table/arrow.cpp` switched on that type and tried to cast
this info to string info, which threw. Main now checks the encoding
before reading type info and keeps every filter above scans containing
run-end encoded columns, as for string and binary views, because PyArrow
cannot filter these arrays. Main's tests in
`test/arrow/arrow_filter_pushdown.cpp` verify that producers receive no
filter. They filter encoded int32 directly and plain columns alongside
encoded int32, encoded utf8, or lists of encoded utf8, returning only
plain columns. They leave direct string filters, combined plain and
encoded filters, and decoded values after filtering unchecked.

This pull request adds “Arrow scan of a table with a run-end encoded
column and a filter” to `test/arrow/arrow_run_end_encoded.cpp`. It tests
filters on plain int32, encoded utf8 strings (`r = 'a'`), and both
combined, returning only the plain column. Two further queries filter
plain columns alongside encoded int32 or lists of encoded int32, also
returning the encoded column and checking its decoded values. Every
query checks rows and row count. The new `RunEndStringColumn` helper
builds the encoded parent with no buffers (`n_buffers` 0), as the Arrow
C data interface requires; PyArrow 26 rejects one buffer. The test
passes on main and in reldebug and relassert (sanitizer) builds. All
reldebug unit tests pass except
`test/sql/settings/reset/reset_threads.test`, which also fails on main
under our runner's thread limit.

Related: duckdblabs/bot-issue-tracker#222

</details>

<details><summary>duckdb-bot#103: Test VARCHAR casts, union_tag
and union_extract with NULL indices in Arrow dictionaries of
unions</summary>

An Arrow dictionary column with sparse union values gave inconsistent
results for a NULL index: `x IS NULL` was true, but `x::VARCHAR` and
`union_extract(x, 'd')` returned `0`, and `union_tag(x)` returned the
first member. All three should return NULL. They now do on main,
following duckdb/duckdb@61e8a48a0f5, merged with duckdb#27575.

`ArrowToDuckDBConversion::ColumnArrowToDuckDBDictionary` converts
dictionary values into a vector with an extra NULL sentinel slot for
NULL indices. Previously, it marked only the slot's top-level validity
NULL, leaving the union tag and members unchanged. Union functions read
the tag, so the row appeared as the first member with value `0`. Main
now flattens the values with `FlattenForSetNull` and calls
`FlatVector::SetNull` on the sentinel for every value type. A union is
internally a struct, so its tag and members are now NULL too.

After merging main, this PR only adds a C++ regression test in
`test/arrow/arrow_union_offset.cpp`. Its original union-only
`FlatVector::SetNull` call duplicated main's and was dropped during the
merge, leaving no changes to `src/function/table/arrow_conversion.cpp`.
The test builds a dictionary of sparse unions through the Arrow C data
interface, including a dictionary-encoded member and a NULL index. A new
optional projection in `ScanMatches` checks the cast, tag and extraction
of both members. Main's tests in
`test/arrow/arrow_dictionary_sentinel.cpp` cover struct, list, array and
null type values, but no union. The new test passes on main and fails
with the old behavior of marking only the sentinel's top-level validity
NULL. The `[arrow]` tests, including the C API Arrow tests, pass in
reldebug and relassert. The full reldebug unit suite fails only in
`test/sql/settings/reset/reset_threads.test`, which also fails on main.

Related: duckdblabs/bot-issue-tracker#544

</details>

## Summary

Backport every PR carrying the `backport v2.0-cyanoptera` label in the
captured PR-list snapshot to `v2.0-cyanoptera`.

The immutable original target used for patch-equivalence checks was
`d25d1ce513ac501e79fcda1fbcb7c47d0d20773c`.

Included PRs:

duckdb#26311, duckdb#26356, duckdb#26379, duckdb#26465, duckdb#26568, duckdb#26578, duckdb#26834, duckdb#26856, duckdb#26893,
duckdb#26897,
duckdb#26920, duckdb#26927, duckdb#26976, duckdb#27074, duckdb#27101, duckdb#27110, duckdb#27120, duckdb#27126, duckdb#27160,
duckdb#27196,
duckdb#27199, duckdb#27210, duckdb#27213, duckdb#27233, duckdb#27237, duckdb#27240, duckdb#27241, duckdb#27246, duckdb#27248,
duckdb#27272,
duckdb#27278, duckdb#27282, duckdb#27284, duckdb#27287, duckdb#27291, duckdb#27292, duckdb#27293, duckdb#27297, duckdb#27298,
duckdb#27300,
duckdb#27301, duckdb#27306, duckdb#27319, duckdb#27370, duckdb#27377, duckdb#27389, duckdb#27390, duckdb#27391, duckdb#27393,
duckdb#27395,
duckdb#27399, duckdb#27406, duckdb#27425, duckdb#27471, duckdb#27478, duckdb#27483, duckdb#27484, duckdb#27490, duckdb#27491,
duckdb#27493,
duckdb#27498, duckdb#27499, duckdb#27506, duckdb#27512, duckdb#27516, duckdb#27522, duckdb#27527, duckdb#27535, duckdb#27543,
duckdb#27549,
duckdb#27558, duckdb#27560, duckdb#27561, duckdb#27562, duckdb#27571, duckdb#27588, duckdb#27592, duckdb#27595, duckdb#27596,
duckdb#27597,
duckdb#27604, duckdb#27607, duckdb#27609, duckdb#27614, duckdb#27619, duckdb#27621, duckdb#27622, duckdb#27626, duckdb#27670.

PRs duckdb#27074, duckdb#27110, duckdb#27493, and duckdb#27626 had no non-merge topic commits in
their fetched head refs. PR duckdb#27604 was already present exactly. Other
already-present or patch-equivalent commits were omitted.

## Replay method

- Queried the labeled PR numbers once and fetched `main`,
`v2.0-cyanoptera`, and all PR head refs together.
- Derived each PR's non-merge topic commits locally and processed PRs
oldest-first.
- Replayed topic commits with `git cherry-pick -x`, preserving authors,
messages, and ordering.
- Checked exact/patch-equivalent presence against the immutable original
target and excluded synchronization merge commits.
- On resume, identified replayed commits from their `-x` trailers.
- A final local audit confirmed that every fetched topic commit is
replayed, exactly present, or patch-equivalent.

## Conflict resolutions and release adaptations

- Preserved release-specific profiling, parser, checkpoint, shell,
progress, storage, and connection APIs while applying the intended
behavior.
- Adapted the V2 C API statistics changes to the release branch's
factory-based registration and connection model, then regenerated the V2
API artifacts.
- Preserved release corruption-test exclusions and the release AWS
extension revision while rebasing the source patch; patch applicability
is verified.
- Rebuilt generated parser files from the release grammar with the
updated generator, avoiding unrelated newer-main grammar rules.
- Adapted index-scan thread limiting to the release branch's
arena/pointer row-ID representation, including transaction-local row
counts.
- Combined the overlapping IEJoin/ASOF follow-ups so task production and
lock ownership retain the later deadlock and thread-safety fixes.
- Restored the V2 C++ scalar execution callback initializer exposed by
the full test suite.
- Removed a superseded release-only path restriction so mixed
STRUCT/VARIANT projection pushdown uses the backported reconstruction
logic.