Apply the outstanding changes in master merge order:
apache#69545 7d16729
apache#69551 7224a2d
apache#69612 61d1aef
apache#69615 8d2bd1e
apache#69650 3dc7341
apache#69651 c00ecb1
apache#69646 e493c61
apache#69668 7865646

apache#69614 already exists on branch-4.2 through apache#69644.
Adapt datasource APIs and tests to the release branch, retaining its
existing scanner dispatch and statement-bound metadata contracts.

### What problem does this PR solve?

Backport the requested external-data and FileScannerV2 changes to
`branch-4.2` in their actual master merge order:

| Order | Master PR | Change | Backport |
| --- | --- | --- | --- |
| 1 | #69545 | Preserve Paimon schema options after field renames |
Included |
| 2 | #69551 | Expand Iceberg capability, transaction and snapshot tests
| Included with 4.2 API adaptations |
| 3 | #69614 | Elasticsearch HTTP scans in FileScannerV2 | Already
present through #69644 (`2104b7509cda`); not applied twice |
| 4 | #69612 | Check catalog visibility before SHOW DATABASES accesses
metadata | Included |
| 5 | #69615 | Publish Iceberg Parquet split offsets | Included |
| 6 | #69650 | Preserve ES compound residuals and expand scanner
coverage | Included |
| 7 | #69651 | Accept Elasticsearch 9.x during node discovery | Included
|
| 8 | #69646 | Prioritize Parquet V2 row groups using TopN bounds |
Included |
| 9 | #69668 | Arrow loads and schema inference in FileScannerV2 |
Included |

The eight outstanding changes are combined in one commit. This branch
retains the 4.2 FE datasource APIs instead of importing master's
connector-module refactor.

### Backport adaptations

- Port Paimon schema restoration, catalog-option provenance, fallback
handling and tests to the existing table-copy/serialization lifecycle.
Keep statement-bound fields when reader options are applied after a
schema-only rename.
- Exercise Iceberg's existing metadata and transaction entry points with
real in-memory tables and retained-generation fixtures. Preserve the
nine disabled known-bug reproductions; omit two connector-only lifecycle
tests. Expiration checks verify live data-file sets and exact
delete-file counters; 4.2's separate manifest-name/data-counter issue
remains unchanged. The delete/history SQL suite checks visible rows and
eager/late-materialization parity because 4.2 does not expose master's
`_file`/`_pos` columns. The inventory labels upstream-only evidence
explicitly.
- Port ES residual tracking to the legacy planner and add compound
OR/NOT, docvalue eligibility and Thrift-context tests there. Use Java
8-compatible HTTP APIs for opt-in ES 9 tests and the branch's supported
node-address fixtures.
- Keep the existing full-schema Iceberg metadata-table scan, which
already supplies the file-size dependency for split offsets. Adapt the
external write suite to verify offsets without requiring master-only
distribution plans or golden output.
- Use the existing Parquet statistics interface and physical
row-location representation. Keep the TopN scheduling and pruning
invariants.
- Include the shared Arrow array normalizer required by the new reader,
taken from the same master revision. Preserve 4.2 Lance/Avro dispatch;
omit the schema-inference test for the unavailable Hive OpenCSV parser.

### Validation

- FE production and test sources compile with the branch's Java 8
target. Across the targeted runs and focused reruns, 20 test classes
report 181 discovered, 171 passed, zero failures/errors, and 10 skipped:
nine intentionally disabled known-bug reproductions plus the opt-in live
ES 9 class.
- FE Checkstyle: passed with zero violations.
- Clang compiler syntax checks: all 23 affected C++ translation units
passed against the current sources and freshly generated Thrift, using
cached third-party/generated build dependencies. Focused BE runtime
validation is listed below.
- clang-format 16: all 30 affected C++ source/header files passed.
- Groovy 4.0.19 syntax and AST conversion: all seven changed suites
passed. All 105 local inventory links resolve.
- Complete staged-diff self-review, whitespace/conflict checks, and
sensitive-information checks passed.

A full BE rebuild and SQL/external-service regressions were not run
locally. Live ES 9 tests require their opt-in endpoint. CI is requested
for the combined backport.

### Release note

Fix Paimon reads after field renames, external-catalog SHOW DATABASES
authorization, and ES compound residual filtering. Support ES 9.x node
discovery, publish Iceberg Parquet split offsets, improve Parquet V2
TopN row-group scheduling, and use FileScannerV2 for Arrow loads and
text/JSON schema inference.

### Check List (For Author)

- Test
    - [x] Unit Test
- [x] Regression test additions and syntax checks; SQL execution pending
CI
- Behavior changed:
    - [x] Yes. Backport the changes listed above.
- Does this need documentation?
- [x] No. Existing interfaces and settings are retained; the test
coverage inventory is included.

### Follow-up fixes

- Set the Thrift `enable_file_scanner_v2` default to `true`, matching
the session default. Preserve explicit `false` for legacy query scans
and extend the existing load-selection test.
- Keep the selected Paimon snapshot and schema together during relation
projection. Historical sequence/aggregation options must not be combined
with the latest field names. Read-only principals can resolve historical
selectors without INSERT permission.
- Read legacy append schemas with missing or empty bucket keys without
bucket pruning, preserving every historical file. The regression covers
both unaware and multiple fixed buckets.

Follow-up validation: 169 targeted Paimon FE tests passed with zero
failures/errors/skips. FE production compilation and Checkstyle passed.
Both changed C++ test translation units passed compiler syntax checks
and clang-format 16. Regenerated Java/C++ defaults are enabled; a Java
8-targeted protocol check verifies constructor/clear defaults, an
omitted wire field, and an explicit-false round trip. Complete BE CI and
live external SQL suites remain pending.

The two new reproductions first failed with the same missing-bucket-key
and renamed-sequence-field errors as CI. A separate read-only-principal
reproduction also failed before the permission correction. Complete
external regression execution remains pending CI.

### BE unit-test compatibility

- Complete the native Arrow Variant input dependency for `branch-4.2`:
accept binary storage and the Parquet Variant extension, validate
metadata and payloads before appending, and retain owned data after
Arrow buffers are released. Allow that ingestion path through the load
reader while preserving query-format restrictions and source-default
validation.
- Preserve Variant V2 markers in the Arrow test fixtures, including
nested types. Verify timestamp input against this branch's SerDe for
positive, negative, and second-boundary values. Match the existing DATE
range diagnostic in the TopN pruning test.
- Add tests for sliced and empty reads, invalid ranges and carriers,
malformed metadata/payloads, NULL children, atomic failure, buffer
ownership, and the query capability boundary.

Validation: all 13 previously failing BE tests passed. The expanded run
passed all 309 tests across nine suites, including Arrow/JSON loads,
load validation, Iceberg, table mapping, and Variant input/output. The
focused executable rebuilds the affected readers and SerDe against
cached 4.2 build dependencies; this is not a complete BE rebuild. All
seven changed C++ translation units passed compiler checks with current
headers/generated Thrift, and all 33 affected C/C++ files passed
clang-format 16. The complete staged diff, whitespace, and
sensitive-information checks passed. Full CI is requested with `run
buildall` and `/review`.