…y the harness (#66195)

### What problem does this PR solve?

FE unit tests dominate CI wall time, and most of that time is not spent
running test logic.

Full attribution of one `fe-core` surefire window (4092s x 12 forks =
**49,104 fork-seconds**, 86% of an 79.7-min build):

| component | fork-seconds | share | nature |
|---|---|---|---|
| JVM lifecycle *between* classes (1219x) | 17,317 | 35.3% | fixed
overhead, proportional to class count |
| non-FE classes: real test execution | 13,259 | 27.0% | **real work** |
| FE startup + teardown (304x) | 7,066 | 14.4% | fixed overhead,
proportional to FE-starting classes |
| FE classes: class loading before FE start | 4,463 | 9.1% | fixed
overhead |
| FE classes: real test execution | 3,797 | 7.7% | **real work** |
| non-FE classes: class-level setup | 3,172 | 6.5% | fixed overhead |

**Real work 17,055s (34.7%) vs fixed overhead 32,019s (65.2%).**
Unattributed: 30s.

The overhead is per *class*, so the only lever with real magnitude is
**reducing class count by merging test classes**. Eliminating one class
is worth **55.2 fork-seconds if it starts an FE** (304 such classes)
versus **16.0 if it does not** (915 classes) — FE-starting classes are
3.5x more valuable to merge.

### What is changed?

Six commits, in dependency order:

1. **Avoid a 10s heartbeat wait in every FE unit test class.**
`Daemon.run()` runs one cycle before sleeping, so `HeartbeatMgr`'s first
cycle completes before `createDorisCluster()` registers a backend;
`checkBEHeartbeat()` then waits a full `heartbeat_interval_second`
(default 10). Sets the interval to 1 before the `Env` singleton is
created, and makes `checkBEHeartbeatStatus()` check-then-sleep at 20ms
granularity instead of sleeping 1s first. Timeout budget unchanged.

2. **Merge 12 tiny nereids test classes into 3 suites.**

3. **Give `DistributeHintTest` assertions and cut its runtime by 95%.**
It enumerated a large parameter space while asserting almost nothing;
now it asserts the distribution actually chosen and covers the space at
a sane size.

4. **Restore spied `Env` fields after every test method**, to stop
unbounded spy nesting leaking across methods.

5. **Merge six command privilege tests into one suite.**

6. **Migrate 25 legacy `UtFrameUtils` test classes to
`TestWithFeService`.** This one saves no time by itself — it is the
precondition for merging, since only classes on the same harness can
share an FE.

### Measurements (CI, build 1009445 unless noted)

| change | measured | note |
|---|---|---|
| heartbeat fix | -3.7s/class, **within noise** | **not the lever** —
see caveat below |
| merge 12 -> 3 classes | 614.4s -> 168.5s | -72.6% |
| `DistributeHintTest` | 458.7s -> 64.2s | -86% |
| merge 6 -> 1 class | 302.8s -> 63.9s | -78.9%, 13 test cases unchanged
|
| migrate 25 classes | `DropTableTest` 4.483s -> 4.566s | no gain by
design; enables merging |

The three verified items save roughly **840 test-seconds** in total.

**Caveat, stated plainly:** at ~7x effective parallelism, and with
per-build noise of 2 sigma = 151s, that lands as only about **2 minutes
of wall clock and is partly masked by noise**. Do not expect the
total-seconds number in the build summary to move cleanly. The
measurement method that does work is per-class comparison against an
unaffected control group, not the build total. The heartbeat fix in
commit 1 in particular looked large in local single-fork measurement and
turned out to be noise-level on CI — it is kept because it is correct
and harmless, not because it is a win.

Full-suite result on build 1009445: **SUCCESS, 8773 passed, 0 failed, 1
muted.** The muted one is `ForwardToMasterTest.testAddBeDropBe`
(ClassCastException), which is known upstream issue #66004 and unrelated
to this branch.

### Rebase note

Rebased onto `af6dcff9051` today. Two files conflicted with upstream,
both against commit 6:

- `FrontendServiceImplTest` — #64304 moved `MCTransaction` /
`MaxComputeExternalCatalog` out of `fe-core`; upstream's
`WriteBlockAllocatingTransaction` mock kept verbatim.
- `DynamicPartitionTableTest` — #65219 added 1478 lines of TIMESTAMPTZ
tests. Resolved by taking upstream's file whole and replaying the
harness migration onto it, then diffing against upstream to confirm only
harness lines moved. All 241 `Assert.*` calls left untouched.

Post-rebase verification: `test-compile` clean, `checkstyle:check`
clean, and the two conflicted classes run **69/69 passing, 0 skipped** —
counts matching upstream's annotation counts exactly.

### A note for reviewers on how this was verified

The JUnit4 -> JUnit5 migration in commit 6 has two failure modes that
**pass silently** rather than erroring:

- `@Rule ExpectedException` is ignored outright by JUnit5, turning six
exception tests into tests that assert nothing. Converted to
`ExceptionChecker.expectThrowsWithMsg`, which matches JUnit4 semantics
(`isInstance` for subclasses, `contains` for messages).
- JUnit4 `Assert.assertEquals(msg, expected, actual)` versus JUnit5
`Assertions.assertEquals(expected, actual, msg)` have **reversed
argument order**. When all three arguments are Strings this compiles
fine and silently compares the wrong things. 36 such call sites exist
across the migrated classes; message-first overloads were identified by
argument *count*, not by whether the first argument looks like a string.

Because of this, `BUILD SUCCESS` is not sufficient evidence for these
commits. Every migrated class was accepted only after its `Tests run:`
count matched the pre-migration count exactly. Reviewers checking this
PR should apply the same standard.

### Release note

None

### Check List

- [x] Test
- [x] Regression test — not applicable, this changes only the FE unit
test harness and test classes
- [x] Unit Test — full FE UT suite green on CI (build 1009445: 8773
passed, 0 failed); post-rebase spot check 69/69 on the two conflicted
classes
- [x] Behavior changed: No (test-only, no production code paths altered)
- [x] Does this need documentation? No

…66303)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #64304

Problem Summary:

Follow-up cleanup for #64304. After external catalogs moved behind the
connector plugin SPI, fe-core still declared a name for every data
source it no longer knows anything about. This removes those names:

- **`TableIf.TableType`**: the per-source `*_EXTERNAL_TABLE` constants
(`HMS`, `ES`, `JDBC`, `ICEBERG`, `PAIMON`, `MAX_COMPUTE`, `HUDI`,
`TRINO_CONNECTOR`, `LAKESOUl`) and the deprecated internal-catalog
`ICEBERG` / `HUDI`. An external table served by a connector plugin is a
`PLUGIN_EXTERNAL_TABLE`; the source's own name is answered by
`PluginDrivenExternalCatalog#getDisplayEngineName`, never by a mapping
held in fe-core.
- **`TableFormatType`**: `hive`, `iceberg`, `hudi`, `paimon`,
`max_compute`, `transactional_hive`, `lakesoul`, `trino_connector`. A
connector names its own format string through
`ConnectorScanRange#getTableFormatType()`, which `PluginDrivenScanNode`
forwards to BE verbatim. Only `tvf` and `remote_doris` are still
produced by fe-core itself. The enum is not persisted anywhere — it only
builds the thrift string.
- **`InitDatabaseLog.Type`**, and with it `ExternalDatabase#dbLogType`,
a field that was assigned and never read. `OP_INIT_EXTERNAL_DB` has been
ignored on replay since 4.0 and `EditLog#logInitExternalDb` has no
callers, so nothing writes one any more; the payload class stays only so
`JournalEntity` can still consume such an entry out of an old journal.
- **`InitCatalogLog.Type.HUDI`**, which was never produced: hudi tables
have always lived in an hms catalog and no `HudiExternalCatalog` ever
existed.
- **`Database#discardHudiTable`**, dead once `TableType.HUDI` is gone.

The remaining `InitCatalogLog.Type` values are kept on purpose. That
enum is also the type of the persisted `ExternalCatalog#logType` field,
which `PluginDrivenExternalCatalog#gsonPostProcess` reads to backfill
the catalog type for the resource-backed catalogs (`es`, `jdbc`) that
never persisted one. Deleting a name there would make it deserialize to
`null` and lose that catalog's type on upgrade.

**Upgrade compatibility.** An image written before the cutover still
carries the old `TableType` names. It stays readable because the
persisted table class is remapped by `GsonUtils`' compatible-subtype
registry, and the stale `type` string deserializes to `null` — GSON
returns `null` for an enum name it does not know rather than throwing —
which `PluginDrivenExternalTable#gsonPostProcess` then normalizes to
`PLUGIN_EXTERNAL_TABLE`, the same normalization it already applied to a
recognized legacy name.

Nothing in the compiler enforces that two-step property, and a
regression would surface not as a build error but as an FE replaying a
persisted external table with a `null` type. So the new
`LegacyExternalTableTypeReplayTest` pins it for every deleted name,
deliberately spelled as string literals so the constants cannot come
back just to keep a test compiling.

### What problem does this PR solve?

Related PR: #64304 (catalog SPI)

Related PR: #65126

Problem Summary: External metadata caching currently has parallel
implementations in FE core and connector
modules. Cache policy, statistics, invalidation, lifecycle, and
concurrency behavior are therefore duplicated,
while connector implementations still depend on FE-owned concepts.

This PR introduces an implementation-free MetaCache SPI module and a
shared Caffeine-backed runtime module.
Common lifecycle, entry, registry, statistics, invalidation, and ID/name
coordination are moved into these
modules. FE core retains only FE-specific catalog routing, configuration
injection, refresh orchestration,
edit-log integration, and schema validation. Hive, HMS, Iceberg, Paimon,
and MaxCompute consumers are migrated
to the shared implementation.

The refactor preserves the existing connector constructor policy and the
guarded bulk-load contract. A guarded
bulk publication participates in the same short generation protocol as
manual miss loads, exact-key actions, and
asynchronous refreshes, while REFRESH/flush invalidation continues to
win without holding a publication monitor
during external I/O.

…ng (#66369)

### What problem does this PR solve?

Related PR: #64304 (catalog SPI)

Problem Summary:

A connector never decides which columns to read — it renders whatever
list `ConnectorScanRequest.getColumns()` carries. The jdbc connector
turns that list verbatim into the remote `SELECT` list and falls back to
`SELECT *` when it is empty (`JdbcQueryBuilder#buildQuery`). So column
pruning for every plugin-driven external scan rests entirely on
`PluginDrivenScanNode#buildColumnHandles()`, which intersects the
connector's column handles with this scan's tuple slots.

That method had **no direct coverage**, and its failure mode (projecting
more columns than the query needs) is a pure performance regression that
no result-comparing test can observe. The existing jdbc explain
assertions all pass a column list and assert those same columns are
present; none of them can fail on an over-wide projection they did not
anticipate.

This PR started as coverage for that gap. The new coverage immediately
found a real bug, so it now carries the fix as well.

**1. The bug: the connector's scan properties are computed before column
pruning**

A plugin-driven scan asks its connector for one property bundle — the
jdbc remote `SELECT`, per-column dictionaries, file format, path
partition keys — and caches it (`cachedPropertiesResult` /
`scanNodeProperties`).

That cache is first filled from `init()`:

```
PhysicalPlanTranslator#getPlanFragmentForPhysicalFileScan
  -> scanNode.init()
     -> FileQueryScanNode#doInitialize -> initSchemaParams -> getPathPartitionKeys()
        -> PluginDrivenScanNode#getPathPartitionKeys -> getOrLoadScanNodeProperties()
```

`init()` runs while the translator is still translating this scan —
strictly **before** the project above it prunes the tuple down to the
columns the query reads (`updateScanSlotsMaterialization`). Everything
the connector derives from the projection at that point therefore
describes the **full table schema**.

The only thing that dropped the cache was `convertPredicate()`, and only
when there was a conjunct to push down. Queries with a `WHERE` clause
were rebuilt from the pruned tuple by accident; filter-less ones kept
the pre-pruning bundle. Result:

```sql
-- doris_test.test1 has 12 columns
explain select count(*) from test1;
-- before: QUERY: SELECT `k1`, `k2`, ..., `k12` FROM `doris_test`.`test1`
-- after:  QUERY: SELECT `k1` FROM `doris_test`.`test1`
```

This holds for **every** `WHERE`-less query on any plugin-driven
external table, not just `count(*)`.

The scan itself was not affected — `getSplits()` rebuilds the column
handles from the final tuple, so the query actually sent to the source
was already pruned. What was wrong is the reported remote query, and
anything else a connector derives from the projection through this
bundle (`populateScanLevelParams`, `getFileAttributes`). Reviewers of
the iceberg connector may want to check the field-id dictionary applied
in `IcebergScanPlanProvider#populateScanLevelParams`, which is built
from the requested columns and, on a filter-less query, was built over
the full schema.

Fix: drop the cache in `doFinalize()`, the first point at which the
tuple is final. Every filtered query already exercises this rebuild path
today — including the second MVCC-snapshot / rewrite-scope pin it
implies — so the filter-less path is only being moved onto an
already-exercised path.

Why it survived since the SPI migration: all 13 remote-query assertions
in the tree filter, and a filter is exactly what used to hide this. The
one filter-less assertion that exists (`test_gbase_jdbc_catalog`,
commented out) expects the pruned single column, i.e. the behavior this
restores.

**2. The projection decision itself (`fe-core`)**

`PluginDrivenScanNodeColumnPruningTest` drives the real
`buildColumnHandles()` and pins:

- only tuple-slot columns are projected (3-column table, 1 requested →
exactly 1 handle);
- the order follows the slot order, not the connector's handle-map order
— the connector renders this list positionally;
- slots with no backing column, and slots with no matching handle, are
skipped rather than leaking into the list;
- an empty tuple projects nothing — the sole input that reaches the
connector's `SELECT *` fallback.

Every assertion was mutation-checked against the production method:
returning `allHandles.values()` kills 4 of the 5, and making the
unmatched-slot path fail loud unconditionally kills the 5th.

**3. Explain assertions (`external_table_p0`)**

Two additions to `test_mysql_jdbc_catalog`, both of which fail without
the fix above:

- a filter-less projection (`select k8 from test1`) — the shape no
existing assertion covered, and the most direct pin for the caching bug;
- `count(*)`, the one shape whose projection would otherwise go empty.
The engine keeps a single smallest slot
(`PhysicalPlanTranslator#updateScanSlotsMaterialization`) instead of
letting the tuple go empty, and an empty tuple is exactly what makes the
jdbc connector emit `SELECT *`. The assertion pins that the remote
select list stays one column wide and is not `*`; it deliberately does
**not** pin which column wins, since that is `getSmallestSlot`'s
business and tracks type widths.

### What problem does this PR solve?

Issue Number: None

Related PR: apache#64304

Problem Summary: Snapshot-pinned Iceberg scans built the full field-id schema dictionary with lower-cased top-level names while scan slots and identity partition keys retained the source names. Mixed-case partition columns therefore missed the BE schema mapping and could abort the backend. Preserve Iceberg top-level names in full-schema dictionaries, return query errors for malformed mappings instead of aborting, and align Hudi partition carriers with the Hudi lower-case schema convention.

### Release note

Fix mixed-case partition reads for Iceberg and Hudi external tables.

### Check List (For Author)

- Test: Unit Test
    - FE Iceberg, Hudi, Paimon, and Hive connector unit tests: 278 passed after rebase
    - BE Iceberg reader unit tests: passed before rebase; latest-master rerun did not reach test execution because the updated BE dependency graph required a full rebuild
- Behavior changed: Yes. Mixed-case external partition columns are matched consistently, and malformed schema mappings fail the query instead of terminating BE.
- Does this need documentation: No