…pache#66529)

## Proposed changes

- make Iceberg compatibility gates conservative and bounded while
pinning column handles to historical schemas
- reset connector metadata across INSERT retries and align V1/V2
defaults, required-field checks, and position-delete row projection
- support primitive partition sources nested in structs, including
nullable-parent propagation and regression coverage
- cache the compact equality-delete field-ID projection by immutable
table snapshot even when the optional full manifest cache is disabled
- resolve nested partition sources through the top-level Nereids slot ID
and fail closed when stable Iceberg IDs are unavailable

## Compatibility-gate trade-offs

- Equality-delete fencing remains conservative across all delete
manifests in the selected snapshot, including partition-pruned scans.
The snapshot-scoped field-ID cache removes repeated manifest walks
without weakening correctness; initial-load failures still fail closed
and remain retryable.
- Requiredness fencing intentionally uses bounded schema-history
inspection rather than an O(snapshot-count) ancestry walk because
snapshot schema IDs are optional. Once a projected requiredness hazard
exists, every non-empty selected snapshot is fenced. This can reduce
rolling-upgrade availability but cannot create a correctness false
negative.

## Testing

- `mvn -pl fe-core,fe-connector/fe-connector-iceberg -am
-Dtest=PhysicalExternalRowLevelMergeSinkTest,IcebergManifestCacheTest,IcebergScanPlanProviderTest
-Dsurefire.failIfNoSpecifiedTests=false test`
- `mvn -pl fe-connector/fe-connector-iceberg -am
-Dtest=IcebergScanPlanProviderTest,IcebergConnectorMetadataTest,IcebergWritePlanProviderTest
-Dsurefire.failIfNoSpecifiedTests=false test`
- `mvn -pl fe-core -am
-Dtest=ConnectorStatementScopeTest,InsertIntoTableCommandTest
-Dsurefire.failIfNoSpecifiedTests=false test`
- `./run-be-ut.sh --run
--filter=SchemaTest.*:VIcebergTableWriterTest.*:IcebergReaderTest.v1_materializes_non_finite_initial_defaults:IcebergV2ReaderTest.PreparesIcebergNonFiniteInitialDefaults:IcebergPositionDeleteSysTableV2ProfileTest.*`
- FE Checkstyle for all affected modules
- clang-format 16 check for all changed C/C++ files

…ter (#66413)

## Summary

- Forward-port the Parquet nullable selection optimization from
branch-4.1 to the latest master.
- Forward-port Iceberg Variant reads to master's plugin-driven connector
implementation.
- Preserve File Scanner V2 safety checks, mixed-version backend gating,
typed-leaf projection, and delete-only MERGE behavior.
- Resolve the latest-master benchmark and deferred-column layout
conflicts while retaining coverage from both sides.
- Adapt the catalog Hadoop-property concurrency fix to master's storage
adapter architecture by atomically publishing an immutable shared
snapshot.
- Expand Iceberg Variant reader coverage for primitive types, nested
containers, multi-file scans, row groups, delete vectors,
equality/position deletes, and lazy materialization.
- Preserve append atomicity for nullable, STRUCT, ARRAY, and MAP
destinations when lazy Variant fallback discovers corrupt input.
- Cache Variant schema presence so ordinary Parquet scans avoid
Variant-specific planning and statistics work.
- Forward-port compound Parquet Page Index pruning, combining AND ranges
by intersection and OR ranges by union while preserving residual
evaluation.
- Keep compound predicate pruning behind master's metadata-pruning
safety fence.
- Forward-port the hardened projected shredded Variant lifecycle fixes,
preserving projected state across exchange, TopN, truncation, mixed file
layouts, and scanner-profile destruction.
- Add native Paimon reads for schema-matched unannotated Variant
carriers without misclassifying ordinary structs.
- Apply the external-table guardrail stabilization by removing
nondeterministic scanner-distribution assertions while retaining
deterministic parallel correctness coverage.
- Fix late runtime-filter refreshes so predicate/non-predicate
reclassification is accepted only when physical slots and projections
remain unchanged.
- Preserve Paimon nested timestamp semantics with Parquet's independent
predicate and deferred-output projections after rebasing.
- Map Paimon Variant schemas to the execution-only VariantV2 carrier so
both JNI and native scan plans remain queryable in Nereids.
- Align the Iceberg nested-path EXPLAIN assertion and Paimon six-column
DESC golden with master's plugin-driven metadata format.

- Forward-port Iceberg schema-evolution and nested partition-write
hardening, including nested defaults and rollback-safe append behavior.
- Preserve Doris floating-point equality semantics for Parquet pruning
across NaN values and signed zero.
- Safely localize nested Parquet Bloom probes while preserving predicate
order, schema-validation barriers, and conservative fallback.

## Original pull requests

- #66397
- #66302
- #66392
- #66421
- #66441
- #66412
- #66503
- #66509

- #66529
- #66470
- #66471

## Verification

- FE reactor build, Checkstyle, and targeted tests: 277 tests passed for
the Parquet and Iceberg forward ports.
- `CatalogPropertyTest`: 2 tests passed for atomic publication and
snapshot immutability.
- Final FE Checkstyle passed for all 74 reactor modules after rebasing
onto master at `a82564ced5`.
- Final post-rebase FE reactor compilation and targeted tests passed:
104 tests across `ConnectorPluginSurfaceTest`,
`IcebergWritePlanProviderTest`, `RequestPropertyDeriverTest`, and
`PluginDrivenTableSinkTest`.
- Repository clang-format 16 verification passed for all 85 affected
C/C++ source and header files.
- BE ASAN targeted tests: 408 tests from 29 suites passed for the
earlier forward ports.
- Focused BE ASAN compound Page Index tests: 2 tests from 2 suites
passed.
- Final focused BE ASAN verification after rebasing: 225 tests from 18
suites passed, including late runtime-filter refresh, nested timestamp
semantics, Variant, Parquet, and Paimon coverage.
- Clang-format 16 dry run passed for all 85 C/C++ source and header
files changed by this PR.
- `PaimonTypeMappingReadTest`: 3 tests passed with Maven build cache
disabled.
- `ConnectorColumnConverterTest#testComputeVariantCarrierConversion`: 1
test passed with Maven build cache disabled.
- Full FE Checkstyle passed for all 74 reactor modules after the Paimon
Variant mapping fix.
- Regression framework build and tests passed: 3 tests, 0 failures; the
Paimon DESC golden was also validated as six columns with an empty Extra
field.
- `git diff --check` passed for the Groovy source changes; the generated
Paimon `.out` rows retain the required trailing delimiter for an empty
Extra field.

- Full FE Checkstyle passed for all 74 reactor modules after applying
#66529.
- Targeted FE Iceberg planner, metadata, manifest-cache, and
statement-scope tests passed after applying #66529.
- BE ASAN floating-point pruning verification passed: 58 tests from 6
suites after applying #66470.
- Final BE ASAN nested-pruning verification passed: 356 tests from 6
suites after applying #66471 and the master API adaptations.
- Clang-format 16 verification passed for all 47 C/C++ source and header
files changed by #66529, #66470, and #66471.

…pache#66529)

## Proposed changes

- make Iceberg compatibility gates conservative and bounded while
pinning column handles to historical schemas
- reset connector metadata across INSERT retries and align V1/V2
defaults, required-field checks, and position-delete row projection
- support primitive partition sources nested in structs, including
nullable-parent propagation and regression coverage
- cache the compact equality-delete field-ID projection by immutable
table snapshot even when the optional full manifest cache is disabled
- resolve nested partition sources through the top-level Nereids slot ID
and fail closed when stable Iceberg IDs are unavailable

## Compatibility-gate trade-offs

- Equality-delete fencing remains conservative across all delete
manifests in the selected snapshot, including partition-pruned scans.
The snapshot-scoped field-ID cache removes repeated manifest walks
without weakening correctness; initial-load failures still fail closed
and remain retryable.
- Requiredness fencing intentionally uses bounded schema-history
inspection rather than an O(snapshot-count) ancestry walk because
snapshot schema IDs are optional. Once a projected requiredness hazard
exists, every non-empty selected snapshot is fenced. This can reduce
rolling-upgrade availability but cannot create a correctness false
negative.

## Testing

- `mvn -pl fe-core,fe-connector/fe-connector-iceberg -am
-Dtest=PhysicalExternalRowLevelMergeSinkTest,IcebergManifestCacheTest,IcebergScanPlanProviderTest
-Dsurefire.failIfNoSpecifiedTests=false test`
- `mvn -pl fe-connector/fe-connector-iceberg -am
-Dtest=IcebergScanPlanProviderTest,IcebergConnectorMetadataTest,IcebergWritePlanProviderTest
-Dsurefire.failIfNoSpecifiedTests=false test`
- `mvn -pl fe-core -am
-Dtest=ConnectorStatementScopeTest,InsertIntoTableCommandTest
-Dsurefire.failIfNoSpecifiedTests=false test`
- `./run-be-ut.sh --run
--filter=SchemaTest.*:VIcebergTableWriterTest.*:IcebergReaderTest.v1_materializes_non_finite_initial_defaults:IcebergV2ReaderTest.PreparesIcebergNonFiniteInitialDefaults:IcebergPositionDeleteSysTableV2ProfileTest.*`
- FE Checkstyle for all affected modules
- clang-format 16 check for all changed C/C++ files

Adapt the remaining applicable fixes from apache#66529 to the legacy Iceberg implementation. Scope review for apache#66750 found that branch-4.1 already preserves the Paimon system-table diagnostic capitalization.

…4.1 (#66825)

### What problem does this PR solve?

Backport the remaining applicable fixes from #66529 to the legacy
Iceberg implementation on `branch-4.1`. The requested scope is limited
to #66529 and #66750.

| Source PR | Applicability on branch-4.1 |
| --- | --- |
| #66529 | Port nested partition-source handling, INSERT retry metadata
reset, format V2 non-finite initial defaults and position-delete row
defaults/requiredness. Adapt the changes to the existing synchronous
writer and statement-pinned schema. |
| #66750 | No code change needed: its lowercase system-table diagnostic
regression belongs to the connector-provider implementation. The legacy
Paimon scan paths already use `Paimon` in these diagnostics. Keep the
branch's existing engine metadata and routing behavior. |

Format V1 reader bug fixes and compatibility changes are outside this
PR.

### Changes

- Accept nested primitive partition sources in the pinned writer schema;
resolve them by stable field IDs and pass the root slot plus child path
through FE/BE routing.
- Preserve parent NULLs in writer and UPDATE/MERGE exchange partition
transforms. Bind nested identity predicates by their full schema path
and reject static nested partition labels that cannot be validated
against row contents.
- Refresh planner-loaded MVCC metadata on INSERT retries while retaining
caller-injected source snapshot fences across every attempt. A new
execution still clears all snapshots.
- Exclude Iceberg V3 lineage columns from positional field-ID binding so
transformed and nested partitions retain partition-aware UPDATE/MERGE
routing.
- In the format V2 reader, materialize Iceberg `NaN`, `Infinity`, and
`-Infinity` initial defaults without sending them through Doris's
finite-only numeric parser. Prepare typed nested defaults and enforce
required-field checks for position-delete row projections.
- Include the source regression coverage and focused tests for nested
writer metadata, field-ID routing, parent NULLs, static partition
rejection, retry snapshot ownership, V3 routing, and default
materialization.

### Validation

- The new retry-snapshot and V3 routing tests each reproduced the
reported failure before their corresponding fixes.
- FE focused tests: 231 passed, with no failures, errors, or skipped
tests.
- ASAN BE focused tests: 257 passed from 27 suites, including nested
partition routing, format V2 non-finite defaults, and position-delete
row defaults.
- FE Checkstyle: passed.
- clang-format 16: all 16 affected C/C++ files passed, including the
restored V1 files.
- Full update-diff self-review and whitespace checks: passed.

Full external-cluster regressions were not run locally. Existing
regression output is retained from the earlier backport; it was not
regenerated during this update.

…4.1 (#66825)

### What problem does this PR solve?

Backport the remaining applicable fixes from #66529 to the legacy
Iceberg implementation on `branch-4.1`. The requested scope is limited
to #66529 and #66750.

| Source PR | Applicability on branch-4.1 |
| --- | --- |
| #66529 | Port nested partition-source handling, INSERT retry metadata
reset, format V2 non-finite initial defaults and position-delete row
defaults/requiredness. Adapt the changes to the existing synchronous
writer and statement-pinned schema. |
| #66750 | No code change needed: its lowercase system-table diagnostic
regression belongs to the connector-provider implementation. The legacy
Paimon scan paths already use `Paimon` in these diagnostics. Keep the
branch's existing engine metadata and routing behavior. |

Format V1 reader bug fixes and compatibility changes are outside this
PR.

### Changes

- Accept nested primitive partition sources in the pinned writer schema;
resolve them by stable field IDs and pass the root slot plus child path
through FE/BE routing.
- Preserve parent NULLs in writer and UPDATE/MERGE exchange partition
transforms. Bind nested identity predicates by their full schema path
and reject static nested partition labels that cannot be validated
against row contents.
- Refresh planner-loaded MVCC metadata on INSERT retries while retaining
caller-injected source snapshot fences across every attempt. A new
execution still clears all snapshots.
- Exclude Iceberg V3 lineage columns from positional field-ID binding so
transformed and nested partitions retain partition-aware UPDATE/MERGE
routing.
- In the format V2 reader, materialize Iceberg `NaN`, `Infinity`, and
`-Infinity` initial defaults without sending them through Doris's
finite-only numeric parser. Prepare typed nested defaults and enforce
required-field checks for position-delete row projections.
- Include the source regression coverage and focused tests for nested
writer metadata, field-ID routing, parent NULLs, static partition
rejection, retry snapshot ownership, V3 routing, and default
materialization.

### Validation

- The new retry-snapshot and V3 routing tests each reproduced the
reported failure before their corresponding fixes.
- FE focused tests: 231 passed, with no failures, errors, or skipped
tests.
- ASAN BE focused tests: 257 passed from 27 suites, including nested
partition routing, format V2 non-finite defaults, and position-delete
row defaults.
- FE Checkstyle: passed.
- clang-format 16: all 16 affected C/C++ files passed, including the
restored V1 files.
- Full update-diff self-review and whitespace checks: passed.

Full external-cluster regressions were not run locally. Existing
regression output is retained from the earlier backport; it was not
regenerated during this update.

…n its planning passes

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66538, apache#65851

Problem Summary: Review follow-up. INSERT OVERWRITE into a connector table
plans twice. InsertOverwriteTableCommand.run pins the writer schema and
normalizes the plan against it: bare DEFAULT and empty rows are expanded by
position and values are cast to the pinned column types. The inner
InsertIntoTableCommand that runs the plan is built with
needNormalizePlan=false, and every attempt of its initPlan resets the
statement's connector scope and pins the writer schema again. Without a
column list BindSink then maps the normalized values by position onto the
second pin, and every later fence (the bound write metadata identity,
validateBoundWriteColumns, the begin and commit schema checks) compares
against that second pin. So when another client reorders the columns
between the passes, e.g. updateSchema().moveFirst("b") on [a, b] during

    INSERT OVERWRITE t VALUES (DEFAULT, DEFAULT)

the defaults are silently written as b <- 1, a <- 2. Main-branch overwrites
have had this exposure since apache#65851 (pin, then expand defaults) and apache#66529
(repin on every attempt); the previous commit gives branch overwrites the
same one, because a branch write now pins the table's current schema.
branch-4.2 carries the pinned writer context from the outer pass into the
inner insert, so both passes share one generation.

The overwrite now records on the inner insert's context the writer columns
and write metadata identity it normalized against, and the inner insert,
after pinning again, fails the statement with "retry the statement" when
either differs. Nothing changes when the table did not change between the
passes or for any other insert.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - InsertUtilsTest: a reordered writer schema and a changed write metadata identity between the passes fail; the same generation, and an insert that normalizes its own plan, pass
    - InsertOverwriteTableCommandTest, InsertOverwriteDroppedTargetTest, InsertOverwriteSkipAuthResetTest, IcebergStaticPartitionLiteralTest, InsertIntoTableCommandTest, InsertIntoTableCommandTableStreamTest, CloudInsertProfileRetryTest, DictionaryInsertTargetDropRaceTest, PluginDrivenTableSinkBindingTest, PluginDrivenTableSinkTest, ReadLockTest, RowLevelDmlCommandTest
- Behavior changed: Yes. An INSERT OVERWRITE into a connector table whose writer schema or write metadata changes while the statement plans now fails with "retry the statement" instead of binding its values by position to the new schema.
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…69499)

### What problem does this PR solve?

Issue Number: None

Related PR: #66538, #65851

Problem Summary:

**In short.** A write to an Iceberg branch binds to the schema of the
branch's head snapshot. Iceberg, Spark and the Iceberg docs all bind a
branch write to the table's current schema, and so do branch-4.1 and
branch-4.2 (#66538). This PR does the same on master.

**Background.** #65851 pins a statement-scoped writer schema for every
Iceberg write (`IcebergWriteSchemaContext`): the columns that the target
list binds to, their write-defaults, and the partition spec and sort
order bound to that schema. For `INSERT [OVERWRITE] INTO t@branch(b)` it
pins the schema of the branch head snapshot, although a read of the
branch already uses the table-current schema. Iceberg commits a branch
write with the table-current schema: `SnapshotProducer` records
`base.currentSchemaId()` as the schema of the new snapshot, whatever its
target ref. So master also needs `validateBranchWriterSchema`, which
checks that the current schema can describe files written with the
branch head schema. The Iceberg docs ("Schema selection with branches
and tags") say that a write to a branch uses the table schema and only a
read of a tag uses the snapshot's schema. Spark 4.0 with Iceberg 1.10.1
binds branch writes the same way: after a rename on main, a write to an
older branch accepts the new name and rejects the old one. branch-4.1
and branch-4.2 (#66538) pin `table.schema()` for branch writes.

**The problem, and what it cost.** Reproduced on a local cluster with a
`file://` format-version 3 table `t (id INT, c INT)` holding one row,
with a branch `b` created before the change on main:

| Change on main after the branch | Statement | Master today | After
this PR (same as Spark and branch-4.2) |
|---|---|---|---|
| `RENAME COLUMN c c2` | `INSERT INTO t@branch(b) (id, c2) VALUES (1,
1)` | `Unknown column 'c2' in target table` | writes the row |
| `RENAME COLUMN c c2` | `INSERT INTO t@branch(b) (id, c) VALUES (2, 2)`
| writes the row | `Unknown column 'c' in target table` |
| `ADD COLUMN p INT`, then `ADD PARTITION KEY p` | `INSERT INTO
t@branch(b) (id, c) VALUES (1, 1)` | `Iceberg partition field 1000
references source field 3 outside pinned schema 0` | writes the row, `p`
is NULL |
| write-default of `default_int` changed from 35 to 36
(`test_iceberg_initial_defaults`, through the Iceberg API) | `INSERT
OVERWRITE TABLE t@branch(b) (id, default_int) VALUES (14, DEFAULT)` |
writes 35 | writes 36 |

The partition case has no way out: only a successful write to the branch
moves its head to a newer schema, and every write to it fails.

**How this PR fixes it.**

- `IcebergWriteSchemaContext.create` still checks that the target ref
exists and is a branch, not a tag. It then pins `table.schema()` for a
branch, as for main, and binds the partition spec and sort order to that
schema.
- `validateCurrentSchema`, the check before the commit, compares the
pinned schema id with the table-current schema id for branch writes too.
A schema change after the statement pinned its schema now fails the
commit. Before, only a change of the branch head did.
- `validateBranchWriterSchema` is removed. The written files now carry
the schema that Iceberg records for the new snapshot.
- An INSERT OVERWRITE now records the writer schema and write metadata
identity it normalized its plan against, and its inner insert fails with
"retry the statement" if the table changed between the two planning
passes, instead of binding the expanded values by position to the new
schema. This also closes the same window for main-branch overwrites,
which master has had since #65851 / #66529.

**Results.**

- Local cluster: the table above shows each statement before and after.
- Unit tests in `IcebergWritePlanProviderTest`: the five tests that
pinned the branch head schema now expect the current schema. Together
they cover the column names (also through `getWriteColumns` and the
sink's schema JSON), the write-defaults, a current required column with
and without a default (without one, an omitted value fails at analysis,
as for main), and a schema change after pinning, which must fail the
commit. Two new tests cover a current partition spec whose source column
the branch head lacks, and a tag or a missing ref, which is still
rejected. With master's `IcebergWriteSchemaContext`, 6 of these 7 tests
fail and the tag test passes. All 1691 tests of `fe-connector-iceberg`
pass.
- Regression: `test_iceberg_initial_defaults`,
`write/test_iceberg_write_evolution_refs`,
`branch_tag/iceberg_branch_tag_schema_change_extended` and
`test_iceberg_schema_ref_actions_matrix` now expect what branch-4.2
expects. In `test_iceberg_initial_defaults` the branch write gets the
current write-default 36. The other three write an older branch with the
current column names and expect the old names to fail. All four pass
locally. Without this change each one fails at the new expectation: the
branch rows of `test_iceberg_initial_defaults` read 35,
`test_iceberg_write_evolution_refs` and
`iceberg_branch_tag_schema_change_extended` get `Unknown column 'zone'`
and `Unknown column 'new_col'`, and in
`test_iceberg_schema_ref_actions_matrix` the INSERT with the old column
name succeeds.