…mory management

The development docs describe the connector-SPI implementation merged in
apache/doris#67726, which differs from the branch-4.1 implementation
documented under version-4.x:

- Every managed cache entry has a size estimator and is governed by memory as
  soon as an FE-wide, Catalog, or entry limit applies; there is no count-only
  group. Governed engines: default/schema, hive, iceberg, paimon, hudi (HMS
  caches), max_compute, adbc, doris.
- Entry tokens and ENTRY_NAME values follow master (hive table/
  partition_names/partition/column_stats/file/partition_view; iceberg table/
  partition/manifest/partition_view; paimon partition_view; max_compute
  partition; adbc metadata).
- Only entry <= Catalog is validated at DDL time; Catalog vs FE-wide is clamped
  at runtime. `0` is rejected at Catalog and entry level.
- Values are strong references; admission uses type-specific estimators plus a
  fail-closed reflective estimator with a visit budget; rejection triggers
  asynchronous peer reclamation and records budget_exceeded /
  entry_too_large / incomplete_estimate / invalid_zero_estimate.
- catalog_meta_cache_statistics lists master's 28 columns (EVICTION_RATE,
  MAX_WEIGHT, ESTIMATED_WEIGHT, WEIGHT_REJECT_COUNT, LAST_WEIGHT_REJECT_REASON).
- Rewrite the metadata cache sections of the Hive, Iceberg, Paimon, Hudi and
  MaxCompute pages to master's entries and defaults, link the guide from the
  metadata cache page and add it to the sidebar.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

…1.4 (#4061)

## What

- Add Chinese and English usage guides for external metadata-cache
memory management (FE-wide, Catalog-level, and cache-entry-level limits,
configuration precedence, examples, monitoring and troubleshooting).
- **version-4.x** documents the branch-4.1 implementation
(apache/doris#66717, Doris 4.1.4): weighted entries Hive
`partition_values`; Iceberg `table`, `snapshot`, `manifest`; Paimon
`table`, `snapshot`; soft values; 33-column
`catalog_meta_cache_statistics`.
- **current (dev)** documents the master implementation
(apache/doris#67726), which differs from 4.1: every managed entry is
memory-governed once any limit applies; engines `default/schema`,
`hive`, `iceberg`, `paimon`, `hudi`, `max_compute`, `adbc`, `doris`;
master entry tokens and `ENTRY_NAME` values; strong values;
type-specific plus fail-closed reflective estimation; asynchronous peer
reclamation; 28-column `catalog_meta_cache_statistics`. The Hive,
Iceberg, Paimon, Hudi, and MaxCompute pages' metadata-cache sections are
rewritten to master's entries and defaults.
- Add a "Monitoring and troubleshooting memory pressure" runbook to both
versions (per-FE, per-Catalog, per-entry queries, control table, symptom
reference). All example queries use only the weight columns shared by
every version: `MAX_WEIGHT`, `ESTIMATED_WEIGHT`, `WEIGHT_REJECT_COUNT`,
`LAST_WEIGHT_REJECT_REASON`.
- Add the guide to the 4.x and current sidebars and link it from the
metadata-cache pages.

…erg comment cache limit

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66717, apache#67726

Problem Summary: Three of the ported metadata cache budget tests could pass while the behavior
they guard is broken:

- `concurrentReservationsNeverExceedTheGlobalLimit` dropped the futures of its 200 submitted
  reservations. If half of them threw instead of returning an empty Optional, the reservation
  count and the global weight would still match. The test now calls `get()` on every future
  before checking the totals.
- The bucket lifecycle test gives every physical cache its own budget group, so it cannot see a
  group bucket removed while another cache of the same group is still live. Iceberg's MVCC and
  list partition views share the `partition_view` group; if closing one of them dropped the
  bucket, a re-created cache would start from an empty bucket and admit weight on top of what
  its sibling still holds. `sharedEntryGroupKeepsItsLimitUntilItsLastPhysicalCacheCloses` keeps
  60 bytes reserved in one sibling, closes and re-creates the other, and checks that it can take
  only the remaining 40 and that the bucket goes away with the last sibling.
- The Iceberg catalog-limit check built only plain catalogs, which never create the
  `iceberg-comment` cache. A REST vended-credentials catalog does, through the same governed
  owner; the test now builds one and checks that the comment cache gets the catalog limit.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - `MetaCacheBudgetManagerTest` (13) and `IcebergConnectorCacheTest` (31) pass locally.
      Removing the live-sibling check from `MetaCacheBudgetManager#close` turns the new shared
      group test red, and building the comment cache outside the catalog's governed owner turns
      the catalog-limit test red.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66717, apache#67726

Problem Summary: With `paimon.data-token.enabled=true` each cached Paimon table owns a
`RESTTokenFileIO`, and the vended token it holds is fetched on first data access, after the table
was weighed and admitted. The estimator reserved 4 KB for it, about twice the STS token seen in
practice (key id, secret, a 1,200-character security token and the endpoint). The token map is
whatever the REST server returns, so a server that vends a larger credential, or a refresh that
brings one, leaves the table cached below its retained size.

Paimon keeps one token per `RESTTokenFileIO` and replaces it on refresh, so the retained size is
bounded by the largest single token, but Paimon gives no hook to reweigh the table when the token
changes. Raise the fixed reserve to 16 KB, the allowance branch-4.x charges for a table's FileIO,
and test it with both a first fetch and a refresh to a security token of 8,192 characters.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - `PaimonCacheSizeEstimatorTest.restDataTokenFileIoIsChargedToItsTable` now checks a 1,200- and
      an 8,192-character security token; with the old 4 KB reserve the second case fails (4,128
      reserved, 9,232 retained). All 678 fe-connector-paimon tests pass locally.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…l-update growth

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66717, apache#67726

Problem Summary: The Paimon table cache estimator charged a FileIO only when it was a
RESTTokenFileIO, but other catalogs give each table its own FileIO as well. A REST catalog
without data tokens creates a ResolvingFileIO per table (`RESTCatalog.fileIOFromOptions`), and a
Hive catalog resolves one per table location (`HiveCatalog.fileIO(Path)`). Both build their
backend maps after the table was weighed, so a weight-limited cache of such tables holds more
heap than its limit allows. Paimon offers no IO-free way to tell an owned FileIO from one the
catalog shares, so every table now carries the fixed 16 KB FileIO allowance, as on branch-4.x.
Charging a shared FileIO to each table only makes the estimate conservative.

A partial-update table also builds a merge function factory with its store, which keeps
sequence-group and aggregator maps for every field. On Paimon 1.4.2 a 300-field partial-update
table grows about 460 bytes per field once a scan has planned, about 150 more than the other
merge engines, while the store reserve allows 320. Reserve another 256 bytes per field for
partial-update tables.

### Release note

Under an external metadata cache weight limit, every cached Paimon table also counts a fixed
16 KB for its FileIO, and partial-update tables count more per field, so fewer Paimon tables fit
in the same limit.

### Check List (For Author)

- Test: Unit Test
    - PaimonCacheSizeEstimatorTest: tables on LocalFileIO, ResolvingFileIO, HadoopFileIO and
      RESTTokenFileIO carry the same FileIO allowance, which covers a ResolvingFileIO with its
      backend and an 8,192-character REST token; the grown-graph check now leaves the FileIO
      allowance out and includes 300-field partial-update and aggregation tables. Charging only
      RESTTokenFileIO, dropping the partial-update reserve, or a 4 KB allowance each make it fail.
      All 679 fe-connector-paimon tests pass locally.
- Behavior changed: Yes (see Release note)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…requests

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66717, apache#67726

Problem Summary: `peerReclaimCoalescesQueuedRequestsToTheLargestAdmission` checked the reclaim
targets as soon as the second reclaim callback counted down its latch. At that point the second
drain is still running: its callback has not returned, and it has not yet decided whether a
request is left for another drain. A regression that leaves a request behind after the coalesced
drain could therefore pass if the next drain ran after the assertion. The test now waits on two
barriers before it checks the targets. Each barrier is a drain of a separate budget manager on the
same process-wide reclaim thread. The first barrier runs after the second drain has finished and
queued anything it left behind; the second runs after that queued drain.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - `MetaCacheBudgetManagerTest` (13) passes locally. With a mutation that keeps the request
      pending after a drain and pauses 50 ms at the start of each drain, the previous version of
      the test passes, and this version fails with targets `[10, 20, 20, 20]`.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…their FileIO

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66717, apache#67726

Problem Summary: The Paimon table cache estimator charged its fixed 16 KB FileIO allowance only on
the FileStoreTable path, but the other table kinds the cache admits own a FileIO as well. A REST
catalog builds every external table, whatever its kind, on a ResolvingFileIO of its own
(`RESTCatalog.fileIOFromOptions`), even when data tokens are enabled, and a Hive catalog resolves a
FileIO per table location. That FileIO builds its backend map after the table was weighed, so a
weight-limited cache admitted format, object, Lance and Iceberg tables below their retained size.
They now carry the same allowance as a FileStoreTable.

### Release note

Under an external metadata cache weight limit, cached Paimon format, object, Lance and Iceberg tables
also count the fixed 16 KB FileIO allowance.

### Check List (For Author)

- Test: Unit Test
    - PaimonCacheSizeEstimatorTest: format, object, Lance and Iceberg tables built on a
      ResolvingFileIO each weigh more than the FileIO allowance and, once the FileIO has built its
      backend, still cover their retained graph. Dropping the allowance from that path makes it fail.
- Behavior changed: Yes (see Release note)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…che weight limit

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66717, apache#67726

Problem Summary: Under a Doris metadata cache weight limit the Iceberg catalog left
`io.manifest.cache-enabled` out unless the user set it, so the SDK manifest content cache, which
keeps up to `io.manifest.cache.max-total-bytes` of manifests per FileIO outside the Doris limit,
stayed at its default of off. An Iceberg REST catalog, however, merges the server's `/config`
response into the client properties, and its defaults fill in any key the client leaves out: a
server default of `true` turned the cache on. The catalog now writes `false` explicitly, which the
merge keeps over the server's defaults. The server's overrides and a table's own config still take
precedence over client properties, as the REST protocol intends, and an explicit user setting
still wins as before.

### Release note

Under an external metadata cache weight limit, an Iceberg catalog keeps the SDK manifest cache off
even when an Iceberg REST server's /config defaults enable it, unless the catalog sets
io.manifest.cache-enabled itself.

### Check List (For Author)

- Test: Unit Test
    - IcebergCatalogFactoryTest: under a weight limit the catalog properties carry
      io.manifest.cache-enabled=false, and merging them with a REST /config response whose defaults
      enable the cache keeps it off. Leaving the key out makes it fail.
- Behavior changed: Yes (see Release note)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…68839)

### What problem does this PR solve?

Issue Number: None

Related PR: #66717, #67726

Problem Summary:

**In short.** This ports the test coverage of branch-4.1 #66717 (memory
governance for external metadata caches) to master. master already has
the feature (#67726), but none of the 4.x tests came along, and several
admission rules are now unpinned. Only tests are added; no production
code changes.

**Background.** #66717 bounds the estimated retained memory of external
metadata caches with three limits: the FE-wide
`external_meta_cache_max_weight`, the catalog property
`meta.cache.max-weight`, and per-entry
`meta.cache.<engine>.<entry>.max-weight`. master re-implemented it in
#67726 on the connector cache framework instead of cherry-picking it, so
the 4.x tests, written against fe-core classes master does not have
(`ExternalMetaCacheBudgetManager`, `MetaCacheEntry`,
`IcebergExternalMetaCache`, ...), could not be carried over.

| | branch-4.1 #66717 | master #67726 |
|---|---|---|
| Where it lives | fe-core `datasource/metacache` | `fe-connector-cache`
plus each connector |
| Budget levels | global / catalog / entry | global / catalog / entry
group (physical caches of one logical entry share it) |
| Cached values | soft values | strong values with explicit reservations
|
| Budget short on admission | synchronously evict the entry's coldest
value, then admit | reject this load, reclaim from peers in the
background, admit on the next miss |

**The problem, and what it cost.** The admission logic itself is
equivalent, but on master:

- `MetaCacheBudgetManager` has no test of its own; it is only reached
through caches built on top of it. The three-level limits, clamping a
catalog replayed from an FE with a larger global quota, the max(old,
new) hand-off between two generations of one key, force release on
close, and the rules for whom a peer reclamation may ask (only the
requester's catalog when the catalog limit is short, only the same
budget group when the group limit is short, queued requests collapsed to
the largest one) could all regress without a red test.
- At the cache level, zero and incomplete estimates, disabled entries,
oversized values, exact accounting when a key is replaced by a smaller
or larger value, reclamation across catalogs when only the global quota
is short, and failed refreshes are not pinned.
- At the connector level nothing checks which Hive and Iceberg caches a
catalog or entry limit actually reaches, that invalidating a Paimon
table gives its reservation back, or that statement copies of historical
Iceberg snapshots stay apart from the cached generation.

**How this PR fixes it.** It adds those checks in master's terms:

| Test | What it pins |
|---|---|
| `MetaCacheBudgetManagerTest` (new, 13) | global / catalog / entry
limits; 200 concurrent reservations never exceed the global quota;
clamping to the smallest enclosing limit; weight parsing (binary units,
heap percentages, rejected forms); idempotent release; replacement
commit and rollback; closed budgets; close-time force release; peer
reclamation coalescing and its catalog / group scoping; catalog and
group bucket lifecycle; a group shared by two physical caches keeps its
limit until the last of them closes |
| `MetaCacheWeightGovernanceTest` (+6) | zero or incomplete estimates
are returned but not cached; a disabled entry never estimates; an
oversized value does not evict cached ones; same-key replacement charges
exactly the current generation; a catalog short only on the global quota
gets it back from another catalog; a failed refresh keeps the cached
value and its reservation |
| `HiveConnectorWeightGovernanceTest` (new, 2) | a catalog limit bounds
the connector's file listing and partition view caches and closing the
connector releases them; an entry limit bounds only its own entry |
| `PaimonMetaCacheCatalogTest` (+1) | invalidating a table releases its
reservation and the reload is admitted again |
| `IcebergConnectorCacheTest` (+1) | which connector caches the catalog,
manifest and partition view limits reach, including the comment cache of
a REST vended-credentials catalog |
| `IcebergTableCacheTest` (+1) | statement copies of historical
snapshots are not the cached generation's objects |

Not ported, because master deliberately differs and has its own tests
for its choice: soft values, synchronous local eviction before
admission, keeping the old value when a replacement is rejected, and
Iceberg generation fencing. The `test_hive_meta_cache` message change
("is wrong" to "must be >= -1") does not apply because master keeps the
legacy message.

**Results.** All new tests pass on current master. Gaps in product
behavior found while comparing the two implementations (for example the
system table columns 4.x exposes and master does not) are left to
separate PRs.

Original author: @CalvinKirs

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
- `run-fe-ut.sh --run` on `MetaCacheBudgetManagerTest`,
`MetaCacheWeightGovernanceTest`, `HiveConnectorWeightGovernanceTest`,
`PaimonMetaCacheCatalogTest`, `IcebergConnectorCacheTest`,
`IcebergTableCacheTest`: 111 tests, 0 failures

… limits (#68842)

### What problem does this PR solve?

Issue Number: None

Related PR: #66717, #67726

Problem Summary:

**In short.** This ports four fixes from branch-4.1 #66717 (memory
governance for external metadata caches) that master's re-implementation
(#67726) does not have. Under a metadata cache weight limit, master
still lets the Iceberg SDK keep its own unbounded manifest cache,
fetches the whole snapshot history of REST `snapshot-loading-mode=refs`
tables on every table cache miss, and under-weighs cached Paimon tables
by 2-5x because they keep growing after admission. Catalogs without a
weight limit are not affected.

**Background.** #66717 bounds the estimated retained memory of external
metadata caches with the FE-wide `external_meta_cache_max_weight`, the
catalog property `meta.cache.max-weight`, and per-entry
`meta.cache.<engine>.<entry>.max-weight`. A value is weighed once, when
it is admitted; if it does not fit, it is returned to the query but not
cached. master re-implemented this in #67726 on the connector cache
framework, and #68839 ports the 4.x tests. Comparing the two
implementations found the gaps below; branch-4.1 handles all of them.

**The problem, and what it cost.**

- **Iceberg SDK manifest cache outside the limit.**
`IcebergCatalogFactory` derives `io.manifest.cache-enabled=true`
whenever `meta.cache.iceberg.manifest.*` is enabled. That SDK cache
keeps up to `io.manifest.cache.max-total-bytes` (100 MB by default) of
manifest content per FileIO, outside every Doris limit, and Glue / S3
Tables catalogs have one FileIO per table. A user who sets a limit gets
a second, unbounded manifest cache next to the bounded one.
- **Remote IO while weighing REST refs-mode tables.** A weight-limited
`IcebergTableCache` serializes the loaded metadata
(`TableMetadataParser.toJson`) and weighs it. Both call
`TableMetadata.snapshots()`, which in refs mode runs the catalog's
supplier and fetches every snapshot. So each table cache miss pays one
extra full-history REST load, exactly the cost refs mode exists to
avoid, and the cached value then holds the whole history.
- **Paimon tables grow after admission.** `paimon-table` is weighed from
the schema, location and catalog environment. A cached `FileStoreTable`
then grows: the first `latestSnapshot()` or scan builds its transient
store (`RowType` copies of the schema, a copy of the options, and for a
partial-update table a merge function factory with sequence-group and
aggregator maps for every field), and field lookups build the four lazy
index maps of every `RowType`. Measured on Paimon 1.4.2, the grown graph
is 2-5x the admitted weight, so the heap the cache really holds can
exceed `MAX_WEIGHT` by that factor.
- **Paimon tables can own their FileIO, which is not charged.** A REST
catalog gives each table its own `RESTTokenFileIO` with data tokens (the
vended token, about 2 KB, arrives on first data access) or
`ResolvingFileIO` without them (`RESTCatalog.fileIOFromOptions`), and a
Hive catalog resolves a FileIO per table location
(`HiveCatalog.fileIO(Path)`). Their tokens and backend maps appear after
admission.

**How this PR fixes it.**

- `IcebergConnector#hasMetaCacheWeightLimit` decides whether any limit
applies: the FE-global limit, the catalog limit, or the `max-weight` of
one of the entries the connector reads (`table`, `partition`,
`manifest`, `partition_view`, now one constant shared with the DDL
validation). The three catalog option builders take that flag, and
`appendManifestCacheProperties` does not derive
`io.manifest.cache-enabled` under a limit. An explicit user value still
wins, as in branch-4.1.
- `IcebergCacheSizeEstimator.areSnapshotsLoaded` reads
`TableMetadata.snapshotsLoaded` (Iceberg exposes no accessor for it). A
weight-limited `TableOwner` checks it first and, if the list is still
behind its supplier, declines retention with
`incomplete_estimate:iceberg_snapshots_not_loaded` instead of
serializing. The statement keeps the live table, whose supplier still
serves historical snapshots on demand.
- `PaimonCacheSizeEstimator` reserves the post-admission growth of every
concrete `FileStoreTable` from its schema shape: a base, each field,
each field nested in a row, each child type of an array / map / multiset
/ vector, each option and each key. The constants are calibrated against
a retained-size oracle that walks the grown graph (new test helper
`EstimatorCalibrationAssertions`). A partial-update table reserves
another 256 bytes per field for its merge function. Every concrete table
also carries a fixed 16 KB allowance for its FileIO, as on branch-4.x:
Paimon offers no IO-free way to tell an owned FileIO from one the
catalog shares, and charging a shared FileIO to each table only makes
the estimate conservative. The allowance covers an 8,192-character REST
token and a `ResolvingFileIO` with its backend; a token refresh replaces
the token rather than adding to it.

**Results.**

| Case | Before | After |
|---|---|---|
| `meta.cache.iceberg.manifest.enable=true` plus any weight limit | SDK
manifest cache derived on, up to 100 MB per FileIO outside the limit |
not derived; `io.manifest.cache-enabled=true` still turns it on |
| Weight-limited REST catalog, `snapshot-loading-mode=refs`, table cache
miss | one extra remote load of the full snapshot list, full history
retained | no extra load; table not retained, statement reads the live
table |
| Paimon table with 301 INT columns, after one scan | 35 KB admitted,
114 KB retained | 134 KB admitted, 114 KB retained |
| Paimon table with a 200-field ROW column, after field lookups | 25 KB
admitted, 117 KB retained | 143 KB admitted, 117 KB retained |
| Paimon partial-update table with 303 INT columns, after one scan |
merge function not reserved | 233 KB admitted, 143 KB retained |
| Paimon table on its own FileIO (REST with or without data tokens,
Hive) | FileIO and token not charged | 16 KB allowance per table |

Across 13 Paimon fixtures (append and primary-key tables, wide schemas,
many options, nested rows, arrays and maps of rows, multisets, wide
partial-update and aggregation tables) the admitted weight without the
FileIO allowance is now 1.2-1.7x the fully grown graph.

Not in this PR: wide schemas that exceed the 10,000-visit budget of
`ReflectiveObjectSizeEstimator.estimateComplete` (for example HMS tables
with about 1,000 primitive columns, or Paimon tables with about 3,400)
are still never admitted under a limit. The same budget is shared by
several connectors and the FE schema cache, so it needs its own change.

**Classes, and how they call each other.**

- `IcebergConnector`: `WEIGHTED_CACHE_ENTRIES`,
`hasMetaCacheWeightLimit()`, and passes the flag at the three catalog
builds.
- `IcebergCatalogFactory`: `buildBaseCatalogProperties`,
`buildCatalogProperties` and `buildS3TablesCatalogProperties` take
`metaCacheWeightLimited` and stay pure.
- `IcebergCatalogProperties`: validates `max-weight` keys against the
shared entry list.
- `IcebergCacheSizeEstimator` / `IcebergTableCache.TableOwner`: snapshot
probe and the guard before serialization.
- `PaimonCacheSizeEstimator`: `estimateStoreGrowth` and
`estimateTableFileIO` / `estimateFileIO`.

```
IcebergConnector.createCatalog / createS3TablesCatalog / createDlfCatalog
  └─ hasMetaCacheWeightLimit(): metaCache.hasEnclosingWeightLimit() || cacheSpec(entry).isWeightBounded()
  └─ IcebergCatalogFactory.build*CatalogProperties(..., metaCacheWeightLimited)
       └─ appendManifestCacheProperties: derive io.manifest.cache-enabled only when not limited

IcebergTableCache.borrow ─ loader ─ new TableOwner(table, ..., weighted)
  └─ areSnapshotsLoaded(table) ? toJson + estimateTable : incomplete("iceberg_snapshots_not_loaded")

PaimonMetaCacheCatalog "paimon-table" size estimator
  └─ PaimonCacheSizeEstimator.estimateTable ─ estimateFileStoreTable (each concrete table)
       ├─ estimateStoreGrowth(schema)
       └─ estimateTableFileIO(fileIO): fixed allowance, once per FileIO
```

Nothing changes for catalogs without a weight limit: the flag is false,
and the probe and the reservations only run when an entry is weighed.
Independent of #68839 (tests only); both add tests to
`IcebergConnectorCacheTest` and `IcebergTableCacheTest`, in different
places.

Co-authored-by: Calvin Kirs <[email protected]>

… own engines (#69556)

### What problem does this PR solve?

Issue Number: None

Related PR: #66717, #67726

Problem Summary:

**In short.** This brings the DDL-time checks of `meta.cache.*` catalog
properties from branch-4.x #66717 to master. CREATE / ALTER CATALOG now
rejects a `meta.cache.<engine>.<entry>.<option>` key that the statement
submits for an engine the catalog owns when the option is not `enable`,
`ttl-second`, `capacity` or `max-weight`, or when its value has the
wrong type, and it rejects a catalog quota (or an entry quota of a
catalog without one) above the FE global quota
`external_meta_cache_max_weight`. Unknown entries and keys stored by
earlier statements stay accepted, as master intends.

**Background.** External catalogs size their metadata caches with keys
such as `meta.cache.hive.file.ttl-second` or
`meta.cache.iceberg.table.max-weight`, plus a catalog quota
`meta.cache.max-weight` that is capped by the FE-wide
`external_meta_cache_max_weight`. Master (#67726) rewrote the cache
governance of branch-4.x (#66717) on the connector SPI. Branch-4.x
validates every `meta.cache.*` key at CREATE / ALTER against a
whitelist. Master deliberately tolerates unknown entries, because a
newer connector version may write an entry an older FE does not know,
and it never re-validates stored keys, because ALTER can overwrite but
not remove a key.

**The problem, and what it cost.** Master only rejects an unknown
`.max-weight` entry that the statement submits and checks the values of
a few keys, so these DDLs succeed:

- a misspelled option such as `meta.cache.hive.file.max-weigth = 64MB`:
the limit the operator asked for is silently not applied;
- `meta.cache.hive.file.enable = 1`: `Boolean.parseBoolean` reads it as
false, so the cache is silently turned off;
- `meta.cache.hive.file.ttl-second = -2`;
- `meta.cache.max-weight = 1PB` on an FE whose global quota is 1GB:
admission clamps the caches to the global quota at run time, so SHOW
CREATE CATALOG promises far more memory than the caches can get.

**How this PR fixes it.**

- `CacheSpec.checkSubmittedEngineProperties` checks the keys a statement
submits under `meta.cache.<engine>.`: the option after the last dot must
be `enable` (true / false), `ttl-second` (>= -1), `capacity` (>= 0) or
`max-weight` (a positive size); anything else fails with `Unknown
external meta cache property: <key>`, the branch-4.x message. The value
messages are the existing ones (`The parameter <key> is wrong, value is
<value>`, `Invalid cache weight for '<key>': <value>`). Entry names are
not checked.
- `CacheSpec.checkSubmittedWeightsAgainstGlobal` rejects a submitted
`meta.cache.max-weight`, and a submitted entry `max-weight` of a catalog
without a catalog quota, above the global quota of the FE running the
statement (nothing is checked when the global quota is unset).
- Both run inside `CacheSpec.checkWeightProperties(properties,
submittedProperties, engine, ...)`, which the hive connector (for `hive`
and its `iceberg` sibling), iceberg, paimon, max_compute and adbc
already call with the statement's properties, so no connector changes.
For the namespace that fe-core owns (`default`, or `doris` for a remote
Doris catalog), a new
`ExternalCatalog.checkSubmittedMetaCacheProperties` hook runs them;
`CatalogMgr` calls it on CREATE and ALTER CATALOG only, never on replay.
- Only submitted keys are checked, so a key stored by an earlier
statement, or a stored quota above a global quota lowered after a
restart, never blocks an unrelated ALTER.

Differences from branch-4.x, kept on purpose:

- An unknown entry (for example `meta.cache.iceberg.schema.ttl-second`)
and a key of an engine the catalog does not route stay accepted and are
ignored.
- `meta.cache.hive.partition_values.*`, `meta.cache.iceberg.snapshot.*`
and `meta.cache.paimon.{table,snapshot}.max-weight` are not supported on
master: a submitted `max-weight` of them is rejected as an unknown
weight entry, their other options are accepted and ignored, and stored
values are ignored. A catalog that only had entry quotas on them has no
weight limit after upgrading.
- An entry `max-weight = 0` is rejected; a stored 0 means no entry limit
rather than a disabled entry.

**Results.**

| DDL | Master today | After this PR |
|---|---|---|
| `meta.cache.hive.file.max-weigth = 64MB` | accepted, ignored |
`Unknown external meta cache property: meta.cache.hive.file.max-weigth`
|
| `meta.cache.hive.file.enable = 1` | accepted, cache off | `The
parameter meta.cache.hive.file.enable is wrong, value is 1` |
| `meta.cache.iceberg.table.ttl-second = -2` on an hms catalog |
accepted | `The parameter meta.cache.iceberg.table.ttl-second is wrong,
value is -2` |
| `ALTER ... meta.cache.default.schema.enable = 1` | accepted | rejected
the same way |
| `meta.cache.max-weight = 1PB` with a 1GB global quota | accepted,
clamped at run time | `meta.cache.max-weight (1125899906842624 bytes)
can not exceed the FE global max weight external_meta_cache_max_weight
(1073741824 bytes)` |
| `meta.cache.default.future_entry.max-weight = 64MB`, then an unrelated
ALTER | accepted | accepted (unchanged) |

Unit tests: `CacheSpecTest` (3 new), one new test each in
`HiveCatalogPropertiesTest`, `IcebergConnectorValidatePropertiesTest`,
`PaimonConnectorValidatePropertiesTest`, `MaxComputePartitionCacheTest`
and `AdbcCatalogPropertiesTest`, and 3 new tests in
`ExternalCatalogMetaCacheWeightTest` (the fe-core hook, the global quota
passed in explicitly, and the remote Doris engine). Without the
submitted-key check the new hive, iceberg and paimon tests fail; without
the global-quota check, or without the remote Doris engine, the
corresponding tests fail. All tests of fe-connector-cache, -hive,
-iceberg, -paimon, -maxcompute and -adbc pass, and so do 280 fe-core
tests around catalog DDL and the metadata caches.
`external_table_p0/test_catalog_ddl` gains the misspelled option,
`enable=1`, `ttl-second=-2` and the ALTER case; its global-quota case
runs only when `external_meta_cache_max_weight` is set, since CI
frontends run without one. The suite passed on a local cluster both
without a global quota and with `external_meta_cache_max_weight = 1GB`.

…_meta_cache_statistics (#69557)

### What problem does this PR solve?

Issue Number: None

Related PR: #66717, #67726

Problem Summary:

**In short.** This brings the weight columns of
`information_schema.catalog_meta_cache_statistics` from branch-4.x
#66717 to master. Master shows four of the ten: this PR adds
`WEIGHT_BOUNDED`, `EVICTION_WEIGHT`, `CATALOG_MAX_WEIGHT`,
`CATALOG_ESTIMATED_WEIGHT`, `GLOBAL_MAX_WEIGHT` and
`GLOBAL_ESTIMATED_WEIGHT` with the 4.x names, types and meanings, and
makes a cache entry without a weight bound report `-1` in every weight
column, as 4.x does, instead of `0` in `ESTIMATED_WEIGHT` and
`WEIGHT_REJECT_COUNT`.

**Background.** External catalogs can bound the memory of their metadata
caches: an FE-global limit (`external_meta_cache_max_weight`), a catalog
limit (`meta.cache.max-weight`) and per-entry limits.
`catalog_meta_cache_statistics` is how users watch those caches. On
branch-4.x (#66717) the view tells, per entry, whether it is weight
bounded, what it holds and has evicted, and how full its catalog and the
FE-global budget are. Master (#67726) re-implemented the governance on
the connector cache framework and exposed only `MAX_WEIGHT`,
`ESTIMATED_WEIGHT`, `WEIGHT_REJECT_COUNT` and
`LAST_WEIGHT_REJECT_REASON`.

**The problem, and what it cost.** Monitoring queries written for 4.x
fail on master with an unknown column as soon as they name one of the
other six, and so do the weight scenarios of 4.x's
`test_paimon_table_meta_cache` and `test_iceberg_table_meta_cache`.
There is also no way on master to see how much of a catalog or of the
FE-global budget is used, or how many bytes were evicted to make room.
And an entry without a weight bound reports `ESTIMATED_WEIGHT = 0` and
`WEIGHT_REJECT_COUNT = 0`, which reads like an empty but bounded cache.

**How this PR fixes it.**
- `MetaCacheBudgetManager.EntryBudget` exposes the catalog limit the
entry is admitted under (the smaller of the catalog and FE-global
limits, as the admission check uses it), the bytes charged to its
catalog, the FE-global limit and the bytes charged to it. A limit that
does not exist is `-1`.
- `ScopedMetaCache` adds the bytes each value was admitted with to an
eviction-weight counter when it removes the value by eviction, by
Caffeine (capacity or expiry) or to make room in the budget. The bytes
are recorded when the value is admitted, because an exact invalidation
can release the reservation of an evicted value before the eviction is
accounted. Invalidation and replacement do not count.
- `ScopedMetaCache.CacheMetrics`, `MetaCacheEntryStats`, `SchemaTable`
and `MetadataGenerator` carry the six values. The columns are appended
after `LAST_WEIGHT_REJECT_REASON`. For an entry without a weight bound,
every weight column is `-1`.
- The BE scanner knows the six columns. An FE that does not know them
rejects the request, and the scanner then asks for the leading 28
columns, then 24, as it already did for the four earlier weight columns.

`SELECT *` positions still differ from 4.x, because master's base
columns also have `EVICTION_RATE`; queries that name their columns now
work on both.

**Results.**

| | master | this PR |
|---|---|---|
| `SELECT weight_bounded, catalog_max_weight ... FROM
catalog_meta_cache_statistics` | fails: unknown column | works, as on
4.x |
| Entry of a catalog with `meta.cache.max-weight = 128MB` | `MAX_WEIGHT
= 134217728` only | also `CATALOG_MAX_WEIGHT = 134217728`, the catalog's
and the FE-global charges, the evicted bytes |
| Entry without any weight bound | `ESTIMATED_WEIGHT = 0`,
`WEIGHT_REJECT_COUNT = 0` | `-1` in every weight column |
| New BE, FE without the six columns | n/a | the BE falls back to the
leading 28 columns |

Tests:
- `MetaCacheWeightGovernanceTest`: an entry without a weight bound
reports `-1` everywhere; the catalog and FE-global limits and charges of
bounded entries, with and without an FE-global limit; a budget reclaim
and a capacity eviction add the evicted reservation to the eviction
weight, an invalidation does not. An eviction keeps the bytes it was
admitted with when the key is invalidated before the deferred eviction
is accounted. Restoring master's `0` for unbounded entries, or dropping
the eviction accounting, turns these tests red.
- `FeMetaCacheEntryTest` checks the fe-core view of an unbounded and of
a bounded entry; `HudiConnectorHmsCacheTest` now expects `-1` for its
unbounded entry.
- Regression: `test_paimon_table_meta_cache` gets the weight scenario of
4.x on master's entry names (a catalog with `meta.cache.max-weight =
128MB` next to an unbounded one); `test_iceberg_table_meta_cache` puts
its first catalog under a catalog limit and checks the budget hierarchy
of its bounded entries, as 4.x does; the new
`test_remote_doris_meta_cache_weight` covers the `doris` engine without
any external service.
- Manually: a BE from this PR against an FE without the six columns
returns the rows (the FE rejects the 34 columns, the BE retries with
28).

Co-authored-by: Calvin Kirs <[email protected]>

…a time (#69558)

### What problem does this PR solve?

Issue Number: None

Related PR: #66717, #67726

Problem Summary:

**In short.** With a metadata-cache weight limit set, the cached
metadata of a wide table (thousands of columns), a long name list or a
long partition list was never cached: its size estimate gave up and the
value was reloaded on every statement. This PR weighs such values one
element at a time, so they are admitted and charged for their size,
while the safety bound of the estimate stays in place.

**Background.** Master (#67726) lets the metadata caches of external
catalogs be bounded by weight: the FE-global
`external_meta_cache_max_weight`, a catalog's `meta.cache.max-weight`,
or an entry's `meta.cache.<engine>.<entry>.max-weight`. Once any of them
applies, each value is weighed before it is admitted. Several entries
use the complete walk of `ReflectiveObjectSizeEstimator`, which visits
every object of the value and gives up after 10,000 of them, so that one
pathological value cannot cost unbounded CPU and temporary memory. An
estimate that gives up is incomplete, and an incomplete value is
returned to the query but not cached. branch-4.x (#66717) governs only a
few entries and weighs them with per-element formulas, so it has no such
limit.

**The problem, and what it cost.** The 10,000-visit budget is reached by
values that are large only because of one list, long before they are
large in bytes. Measured on master, the first value that never enters a
weighted cache:

| Entry | Value | Never cached from |
|---|---|---|
| `hms-table` | Hive table | 995 INT / 382 `struct<a:int,b:string>`
columns |
| `paimon-table`, `paimon-schema-at` | Paimon schema | 2,497 INT / 908
`ROW<a INT, b STRING>` columns |
| fe-core `schema` (`default`, `doris` engines) | schema of any other
external catalog | 3,332 INT / 832 struct columns |
| `adbc-table-schema` | Arrow schema | 2,499 INT / 1,110 struct fields |
| `adbc-table-names`, `adbc-namespaces` | name list | 9,999 names |
| `max-compute-partition` | partition list | 1,249 three-level
partitions |

For such a table every statement goes back to the source (an HMS call, a
Paimon schema file read, an ODPS partition listing, an ADBC metadata
call), and `weight_reject_count` in
`information_schema.catalog_meta_cache_statistics` keeps growing. In a
unit test with a 128 MiB global limit, three reads of a 2,000-column HMS
table loaded it three times and cached nothing.

**How this PR fixes it.**

- `ReflectiveObjectSizeEstimator.estimateCompletePerElement(root,
collections...)` walks `root` like the complete estimate, except that
each listed collection it reaches is counted as its own storage plus
every element estimated separately, each with its own 10,000-visit
budget. An element that appears in more than one of the listed
collections is counted once.
`MetaCacheSizeEstimators.reflectivePerElement(...)` and
`reflectiveEachElement()` wrap it as cache estimators.
- The affected entries use it: the HMS table (columns, partition keys),
the Paimon table schema and the schema-at snapshot (fields), the fe-core
schema cache (`SchemaCacheValue.getColumnLists()`: schema, and for
plugin tables also partition and display columns), the ADBC table schema
(fields) and name lists, and the MaxCompute partition list.
- The visit budget still bounds every element and the rest of the value,
so a value that reaches a huge shared structure still fails fast, and
the temporary identity set holds one element's graph at a time instead
of the whole value's. The whole-value walk and its budget are unchanged
for every other entry.
- Objects that elements share (for example a scalar type singleton) are
counted once per element, so the weight is somewhat above the retained
size (measured 1.1x to 1.6x) and never below it. A plugin table's
display columns reuse its column objects and are not counted twice.

**Results.**

| Value, under a weight limit | Before | After |
|---|---|---|
| 3,000-column HMS table (INT or struct columns) | estimate incomplete,
reloaded on every read | complete, loaded once, `weight_reject_count =
0` |
| 4,000-column Paimon table, INT or ROW columns; 4,000-field schema-at
snapshot | incomplete | complete, loaded once |
| fe-core schema with 4,000 INT + 1,000 struct columns | incomplete |
complete, loaded once |
| 3,000-field ADBC schema; 20,000 table names | incomplete | complete,
loaded once |
| 10,000 MaxCompute partitions | incomplete | complete, loaded once |
| single element larger than the visit budget | incomplete | incomplete
(unchanged) |

The new tests also check that the estimate is never below an unbounded
walk of the value and at most twice it, and that it doubles when the
column count doubles. With the per-element walk disabled, all nine new
wide-value tests fail; without the element de-duplication, a plugin
schema with reused display columns weighs twice as much (2,052,208 to
4,104,168 bytes) and its test fails.