…hema (#69488)

### What problem does this PR solve?

Issue Number: None

Related PR: #66538, #57531, #65851

Problem Summary:

**In short.** On an Iceberg table, `DEFAULT(column)` anywhere except
`UPDATE ... SET` and `MERGE ... VALUES` reads the default from the read
schema, which carries no Iceberg write-default. This PR resolves it
against the write schema that the statement pinned for its target, as
branch-4.1 and branch-4.2 do.

**Background.** An Iceberg `write-default` applies to rows written after
it is set. So it lives in the writer schema that a write statement pins
(`IcebergWriteSchemaContext`), not in the columns of the read schema.
#65851 resolves `DEFAULT(column)` against that pinned schema in `UPDATE
... SET`, `MERGE ... UPDATE SET` and `MERGE ... INSERT VALUES`
(`ConnectorWriteSchemaUtils`). Every other `DEFAULT(column)` either
fails at binding or reaches `RewriteDefaultExpression` (#57531), which
reads the default from the bound column. branch-4.1 #66538, also in
branch-4.2, resolves these against the pinned write target too. It
rejects `DEFAULT(column)` on an Iceberg column in a statement that does
not write the table.

**The problem, and what it cost.** Reproduced on a local cluster with a
`file://` format-version 3 table: `CREATE TABLE t (id INT, c INT)`,
`ALTER TABLE t ADD COLUMN d INT DEFAULT 42`, one row `(0, 0)`, so `d =
42`.

| Statement | Master today | After this PR (same as branch-4.2) |
|---|---|---|
| `INSERT INTO t (id, c, d) VALUES (1, 1, DEFAULT(d))` | `Unknown column
'd' in 'table list'` | writes 42 |
| `INSERT INTO t (id, c, d) SELECT 2, 2, DEFAULT(d) FROM t WHERE id = 0`
| writes NULL | writes 42 |
| `INSERT INTO t (id, c, d) SELECT 3, 3, DEFAULT(d)` | `Unknown column
'd'` | writes 42 |
| `UPDATE t SET c = 100 WHERE d = DEFAULT(d)` | updates nothing: `d =
NULL` makes the plan an empty set | updates every row, `d = 42` is
pushed down to the scan |
| `MERGE INTO t USING (SELECT 0 AS id) s ON t.id = s.id AND t.d =
DEFAULT(t.d) WHEN MATCHED THEN UPDATE SET c = 300` | matches nothing |
updates row 0 |
| `SELECT DEFAULT(d) FROM t` | NULL | `DEFAULT(d) must reference a
column of the table that this statement writes` |

The `UPDATE` and `MERGE` cases fail silently: the statement succeeds and
changes nothing.

**How this PR fixes it.**

- fe-core cannot tell from a column whether its connector keeps defaults
in the read schema. Hive does: an HMS default shows up in both schemas.
Iceberg does not. The new connector capability
`SUPPORTS_WRITE_ONLY_DEFAULTS` says which kind a table is. The Iceberg
connector declares it. The Hive connector passes it on to iceberg-on-HMS
tables, like the other per-table capabilities, but only through the
two-argument `getTableSchema`. Since #69299, a statement that pins the
table at its latest snapshot reads the schema from the snapshot
overload, which reflects none of them, so on an iceberg-on-HMS table
such a statement keeps master's results (for example, `UPDATE t SET c =
100 WHERE d = DEFAULT(d)` still changes no row). Reflecting the
sibling's capabilities there changes planning for all of them and is
left to a follow-up PR.
- For a target that declares it, `InsertUtils.normalizePlan` resolves
`DEFAULT(column)` in `INSERT ... VALUES` and in the select lists of
`INSERT ... SELECT`, including the replacements of `SELECT *
REPLACE(...)`, against the pinned write schema. It does this by name,
before the source relations are bound, so the name always refers to a
target column, as in branch-4.2. In MySQL, `DEFAULT(column)` in a
`SELECT` refers to a source column instead. A derived table in `FROM` is
a query of its own and is left alone: `DEFAULT(column)` in it still
names a column of its own `FROM` clause.
- `RewriteDefaultExpression` takes the default for a `DEFAULT(column)`
bound to such a table from the pinned write column with the same field
id. That covers the `WHERE` of `UPDATE` and the `ON` of `MERGE`. A MERGE
resolves it whatever its WHEN branches, also when they all delete, as
branch-4.2 does. If the statement pinned no write schema for that table,
it fails. A `DELETE` pins the writer schema too, for its delete sink,
but it writes no row values, so `RowLevelDmlCommand` marks its target
and `DEFAULT(column)` on it fails with `DEFAULT(d) cannot be used in a
DELETE: the defaults of this table apply only to the rows that a
statement writes`, as in branch-4.2, whose DELETE installs no write
schema context. Master reads NULL there, so `DELETE FROM t WHERE d =
DEFAULT(d)` deletes nothing; resolved to the write-default it would
delete every row with `d = 42`.
- A resolved default is cast to the type of its writer column before the
expression around it is bound, as `RewriteDefaultExpression` already
does for a bound `DEFAULT(column)`. The parsed literal `-1` is a
TINYINT, so without the cast `bit_count(DEFAULT(c))` on an INT column
would count 8 bits instead of 32. The external `UPDATE`, `MERGE` and
changelog paths use the same helper. `RewriteDefaultExpression` rejects
a bound `DEFAULT(column)` whose writer column has another type than the
read column the statement was bound with, which happens only when the
table changed outside Doris after Doris cached it, and asks to refresh
the table: the expressions around it were bound against the read type.
- Hive, Paimon and internal tables do not declare the capability, so
they behave as before.
- The new enum constant changes the connector plugin API surface, so
`connector.plugin.api.version` goes from 26.0 to 27.0, as for the other
recent changes to that surface (#69299, #69588, #69298, #69683 and
#69835 took 22.0 to 26.0).

**Where `DEFAULT(column)` resolves on an Iceberg target (direct Iceberg
catalogs).**

| Statement | Position of `DEFAULT(column)` | Result | branch-4.2 |
|---|---|---|---|
| `INSERT` / `INSERT OVERWRITE` | `VALUES` rows | the target column's
write-default, typed as the column | same |
| | the INSERT's own select lists (with or without `FROM`, set operation
branches, `SELECT * REPLACE`) | the target column's write-default, by
name, before binding | same, except `* REPLACE`, which branch-4.2 binds
to the source |
| | `GROUP BY` / `ROLLUP` outputs, a derived table in `FROM`, `WITH`
definitions, subqueries, `WHERE` / `ON` | bound like any
`DEFAULT(column)` (last row) | same, except derived tables, which
branch-4.2 resolves against the target |
| | a `VALUES` branch of a set operation | fails with an unknown column,
as on master | same |
| | a qualifier other than the target (`DEFAULT(src.c)`,
`DEFAULT(ctl.db.t.c)`) in the positions above | fails: `column in
assignment list is invalid` | same |
| `UPDATE` | `SET` / `WHERE` | write-default | same |
| `MERGE` | `ON` and `WHEN ... AND` conditions, whatever the WHEN
branches, also when they all delete | write-default | same |
| | `UPDATE SET` / `INSERT VALUES` | write-default | same |
| `DELETE` (also `DELETE ... USING`) | anywhere | fails: `cannot be used
in a DELETE` | fails: its DELETE has no write schema context |
| any | a bound `DEFAULT(x.c)` | `x` is the written table: its
write-default by field id; an OLAP or Hive table: that column's default;
another Iceberg table: fails with `must reference a column of the table
that this statement writes` | same |

A bound `DEFAULT(column)` whose writer column has another type than the
read column fails and asks to refresh the table. Where `DEFAULT(column)`
is bound rather than resolved by name (the third row), it keeps its
meaning from before this PR and names a column of its own `FROM` clause,
as on master and branch-4.2: for example in `INSERT INTO t SELECT id,
DEFAULT(d) FROM src GROUP BY id, d` with an OLAP or Hive `src`, it takes
the default of `src.d`.

**Results.**

- Local cluster, with the table above: the table shows what each
statement did before and after. `EXPLAIN UPDATE t SET c = 5 WHERE d =
DEFAULT(d)` goes from an empty set to a scan with `d = 42` pushed down
to Iceberg.
- Unit tests: 6 new tests in `InsertUtilsTest`, 6 in the new
`RewriteDefaultExpressionTest` and 1 in `RowLevelDmlCommandTest`. They
cover VALUES, and INSERT ... SELECT both from a staging relation whose
own column is also `name` and without FROM: each resolves to the pinned
default, and an unknown column fails. `DEFAULT(column)` inside a derived
table, qualified or not and also under a UNION branch, is left to
binding. A `SELECT * REPLACE(DEFAULT(name) AS name)` replacement
resolves like the rest of the select list and keeps the star's qualifier
and `EXCEPT` list. The pinned write column wins over the read column. A
statement that writes nothing fails. A table without the capability is
unchanged. With master's `InsertUtils` and `RewriteDefaultExpression`,
the 5 tests of the fix fail and the 2 tests of the unchanged path pass.
The `SELECT * REPLACE` test fails with this PR's previous `InsertUtils`,
where the replacement stays `DEFAULT(name)`. `IcebergConnectorTest`,
`HiveConnectorMetadataSiblingDelegationTest` and
`ConnectorPluginSurfaceTest` pass. So do the fe-core tests around the
write schema: `ConnectorWriteSchemaUtilsTest`,
`ExternalRowLevelUpdatePlanBuilderTest`,
`ExternalRowLevelMergePlanBuilderTest`,
`ChangelogRowLevelDmlTransformTest` and `PluginDrivenExternalTableTest`.
Two of the tests pin the typing: `bit_count(DEFAULT(bits))` gets the
default as an INT in VALUES and in SELECT, and fails without the cast; a
BIGINT writer column over an INT read column is rejected, directly and
under an IF bound to INT. `DEFAULT(column)` on the target of a DELETE
fails even with a pinned write-default, and only a DELETE marks its
target (an UPDATE does not); skipping the check, or marking UPDATE
instead of DELETE, makes exactly these two tests fail.
`PositionDeleteRowLevelDmlTransformTest`,
`PhysicalPlanTranslatorIcebergRowLevelDmlTest`, `DeleteFromCommandTest`
and `UpdateCommandTest` pass.
- Regression: `test_iceberg_initial_defaults` now writes rows 20 to 22
with `DEFAULT(default_int)` in VALUES and in INSERT ... SELECT with and
without FROM. It then updates them with `WHERE default_int =
DEFAULT(default_int)` and `MERGE ... ON ... = DEFAULT(...)`, for Parquet
and ORC, and checks the rows in the `.out`. A plain `SELECT
DEFAULT(default_int)` must fail, and so must a `DELETE ... WHERE
default_int = DEFAULT(default_int)` on rows 20 to 22, which hold the
write-default. The suite passed locally before the DELETE block was
added, and the whole suite, DELETE block included, has since passed in
two External regression runs on CI. With master's FE it fails at the
first of these statements: `Unknown column 'default_int' in 'table
list'`.

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69299

Problem Summary: The snapshot-id test passed a null partition-item map because it only exercised partition aliases. Iceberg partition projections now copy their maps before cache publication, so this invalid fixture fails before reaching its snapshot assertions. Supply an empty map and retain all snapshot-selection assertions. No production behavior changes.

### Release note

None

### Check List (For Author)

- Test: Unit Test: reproduced the original IcebergPartitionInfoTest failure, then passed its 2 tests and all 25 FrontendServiceImplTest tests; full build.sh --fe package build and Checkstyle passed. The remote authorization-test failure was not reproduced locally.
- Behavior changed: No
- Does this need documentation: No

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69299

Problem Summary: Moving Iceberg partition views to a separate cache caused an MV refresh to enumerate the complete view once per requested partition whenever caching was disabled or weight admission rejected the view. Table-generation replacement also left obsolete partition views behind, and a previously pinned projection could republish an old view after retirement. Load one view for the native/HMS Iceberg bulk MV operation, share the existing version fallback, and preserve per-partition failures. Retire obsolete physical-generation entries during replacement and prevent stale lazy loads from republishing them. Same-generation reuse and cache accounting remain intact.

### Release note

Reduce repeated Iceberg metadata loading during materialized-view refresh and retire obsolete partition-cache views after table metadata refresh.

### Check List (For Author)

- Test: Unit Test
    - Reproduced all four lifecycle/load-count regressions before the fix.
    - All 197 related FE tests pass, including native/HMS bulk reads, disabled/rejected cache admission, replacement and late loads, version fallback and per-partition errors.
    - Full FE packaging and Checkstyle pass through build.sh --fe -j4 (UI build disabled).
- Behavior changed: Yes (reuse operation-local partition views and retire obsolete cached generations)
- Does this need documentation: No

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69299

Problem Summary: Iceberg partition views are loaded lazily from a retained table generation. Catalog reset can remove its cache group before projection creation or the first partition read. Looking up current partition and schema entries then fails pinned work despite its live table lease. Capture only an existing shared partition entry when applicable and derive partition columns from the same retained table/schema under captured authentication. Session-catalog and retired-group paths load directly without resolving another catalog generation, while existing cache admission and retirement fences remain in place.

### Release note

Keep retained Iceberg partition metadata usable after catalog cache reset.

### Check List (For Author)

- Test: Unit Test: 204 related FE tests; full FE packaging and Checkstyle via build.sh --fe
- Behavior changed: Yes; retained partition projections can finish lazy metadata loading after catalog reset
- Does this need documentation: No

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69299

Problem Summary: Repeated Iceberg dictionary freshness checks detach the latest snapshot by eagerly loading its partition view. When the partition entry is disabled, rejected by its weight limit or retired, three checks of the same immutable generation invoke the complete view loader three times. Use the existing scoped snapshot callback and retain only the exact maximum partition-update timestamp in that generation. A synchronized lazy getter computes it once, including an empty view's zero result; a new metadata generation computes its own value. Reserve the fixed scalar/flag fields in snapshot accounting and keep complete partition maps subject to the existing admission policy. Focused tests reduce three loader invocations to one without retaining an unadmitted full view; this is a loader-count result, not a latency benchmark.

### Release note

Avoid repeated partition metadata enumeration during Iceberg dictionary freshness checks.

### Check List (For Author)

- Test: Unit Test: 207 related FE tests; full FE packaging and Checkstyle via build.sh --fe
- Behavior changed: Yes; reuse the same freshness value within one immutable metadata generation without retaining full partition views
- Does this need documentation: No

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69299, apache#69983

Problem Summary: Reset-safe lazy partition metadata loading parses a retained Iceberg table directly instead of using the live schema cache. It therefore bypassed SchemaCacheValue.validateSchema(), allowing case-insensitive duplicate columns into partition-range and MV/dictionary snapshot derivation even though normal schema loading rejects them. Reuse the same pure validator before deriving partition columns, inside the captured authenticator and existing table/snapshot AnalysisException wrapper. Preserve the retained table/schema generation and avoid a live catalog/cache lookup after reset.

### Release note

Reject case-insensitive duplicate Iceberg columns consistently when deriving lazy partition metadata for materialized views and dictionaries.

### Check List (For Author)

- Test: Unit tests and packaging
    - Two real Iceberg metadata tests reproduced the missing rejection for active and retired catalog projections before repair.
    - All 173 related schema, snapshot, partition and cache FE tests passed, including existing reset/session success paths.
    - Full FE packaging and Checkstyle passed.
    - git diff --check passed; no local full P0 or external-service regression in this cycle.
- Behavior changed: Yes. Lazy retained partition loads reject ambiguous schemas consistently and keep table/snapshot error context. Snapshot identity, cache admission and generation-retirement semantics remain unchanged.
- Does this need documentation: No

… and SQL cache metadata (#69299) (#69983)

cherry pick from #69299

…t a snapshot (#69999)

### What problem does this PR solve?

Issue Number: None

Related PR: #69299, #69488

Problem Summary:

#### In short

An Iceberg table stored in HMS and read through an HMS catalog
(`'type'='hms'`) loses every capability it inherits from the embedded
Iceberg connector in any statement that pins the table at a snapshot.
Since #69299 that is every statement that reads it: it gets no Top-N
lazy materialization and no nested column pruning, and `DEFAULT(column)`
in the conditions of `UPDATE` and `MERGE` silently matches nothing. This
PR makes the snapshot overload of `HiveConnectorMetadata#getTableSchema`
pass the inherited capabilities on, as the latest-schema overload
already does. It is a master-only fix (branch-4.x has no connector SPI),
left over from #69488 (review comment 4355508100).

#### Background

fe-core resolves a table-scoped capability
(`PluginDrivenExternalTable.hasCapability`) from the catalog's connector
or from the per-table set in the table's schema. It never asks the
embedded connector that built the schema of an iceberg-on-HMS table. So
`HiveConnectorMetadata` copies the capabilities that fe-core resolves
per table (`SIBLING_INHERITABLE_CAPABILITIES`: auto and sampled analyze,
Top-N lazy materialization, nested column pruning, field-id access
paths, storage predicate pruning, nested column DDL and write-only
defaults) from the owning connector into the per-table set of each
schema it delegates. Iceberg declares seven of the eight; it does not
declare sampled analyze.

It did this only in `getTableSchema(session, handle)`.
`getTableSchema(session, handle, snapshot)` returned the sibling's
schema unchanged. fe-core uses that overload for explicit time travel.
Since #69299 it also uses it for the latest-snapshot pin of every
statement that reads an Iceberg table: the pin retains its schema
(`retainSchema(true)`), `PluginDrivenMvccExternalTable` builds the
schema at that pin, and `getSchemaCacheValue()`, where `hasCapability`
reads the per-table set, returns it for the rest of the statement.

#### The problem, and what it cost

Reproduced on a local cluster with a standalone Hive 3.1.2 metastore.
Two identical format-version 3 tables `(id INT, c INT, s STRUCT<a:INT,
b:STRING>)` get two rows, then `ADD COLUMN d INT DEFAULT 42`, `RENAME
COLUMN s.a TO a2` and a third row with `d = 7`. The same statements then
run against one table through the Iceberg catalog and against the other
through the HMS catalog.

| Statement | Iceberg catalog | HMS catalog, master | HMS catalog, this
PR |
|---|---|---|---|
| `EXPLAIN SELECT * FROM t ORDER BY id LIMIT 1`, also with `FOR VERSION
AS OF` | `VMaterializeNode` | no lazy materialization |
`VMaterializeNode` |
| `EXPLAIN SELECT struct_element(s, 'a2') FROM t` | `all access paths:
[s.a2]` | no pruning | `all access paths: [s.a2]` |
| `UPDATE t SET c = 100 WHERE d = DEFAULT(d)` | updates rows 0 and 1 |
updates nothing | updates rows 0 and 1 |
| `MERGE INTO t USING (SELECT 0 AS id) s ON t.id = s.id AND t.d =
DEFAULT(t.d) WHEN MATCHED THEN UPDATE SET c = 300` | updates row 0 |
updates nothing | updates row 0 |
| `DELETE FROM t WHERE d = DEFAULT(d)` | fails: `cannot be used in a
DELETE` | deletes nothing | fails the same way |
| `SELECT DEFAULT(d) FROM t` | fails: `must reference a column of the
table that this statement writes` | returns NULL | fails the same way |

The `UPDATE`, `MERGE` and `DELETE` rows fail silently: the statement
succeeds and changes nothing. Reads of the renamed field return the
right values in all three columns: on master the HMS catalog does not
prune the struct at all, and with this PR it prunes it with field-id
access paths, as the Iceberg catalog does. `INSERT ... VALUES
(DEFAULT(d))` and `INSERT ... SELECT DEFAULT(d)` already wrote 42
through both catalogs.

#### How this PR fixes it

Both overloads now go through one private helper,
`delegatedTableSchema`. It resolves the owning sibling once, asks it for
the schema (at the pin, for the snapshot overload), and passes the
result through the existing `reflectSiblingCapabilities`. The snapshot
reaches the sibling unchanged. A hudi-on-HMS table is unaffected: the
Hudi connector declares no capabilities, so its schema comes back as it
was. A plain Hive table never reaches this branch.

#### Results

Through the HMS catalog, every statement in the table above now plans
and answers as it does through the Iceberg catalog.