Repository navigation
[improvement](fe) Estimate wide metadata-cache values one element at a time - #69558
Merged
morningman merged 1 commit intoOct 25, 2026
Merged
Conversation
…a time ### What problem does this PR solve? Issue Number: None Related PR: apache#66717, apache#67726 Problem Summary: Once any metadata-cache weight limit is set (the FE-global external_meta_cache_max_weight, a catalog meta.cache.max-weight or an entry max-weight), every cached value is weighed before it is admitted. Master (apache#67726) weighs several entries with the complete walk of ReflectiveObjectSizeEstimator over the whole value, which gives up after 10,000 visited objects. A value that is large only because of one long list reaches that long before it is large in bytes, and an incomplete estimate is rejected: the value is returned to the query but never cached, so every statement loads it again (an HMS call, a Paimon schema file read, an ODPS partition listing) and weight_reject_count keeps growing. Measured, the first value that never enters a weighted cache: hms-table 995 INT / 382 struct<a:int,b:string> columns paimon-table, paimon-schema-at 2,497 INT / 908 ROW columns fe-core schema cache 3,332 INT / 832 struct columns adbc-table-schema 2,499 INT / 1,110 struct fields adbc-table-names, namespaces 9,999 names max-compute-partition 1,249 three-level partitions branch-4.x weighs its few governed entries with per-element formulas and has no such limit. The complete estimate can now walk the long lists of a value element by element: ReflectiveObjectSizeEstimator.estimateCompletePerElement counts a listed collection's own storage and estimates each element with its own visit budget, counting an element that appears in several of the lists once. The HMS table (columns and partition keys), the Paimon table schema and schema-at snapshot (fields), the fe-core schema cache (schema, partition and display columns), the ADBC table schema (fields) and name lists, and the MaxCompute partition list use it. The visit budget still bounds every element and the rest of the value, so a value that reaches a huge shared structure still fails fast, and the temporary identity set holds one element's graph at a time. Objects that elements share, such as a scalar type singleton, are counted once per element: the weight can be about 1.1x to 1.6x of the retained size, never below it. The whole-value walk and its 10,000-visit budget are unchanged for every other entry. ### Release note Fix: with a metadata-cache weight limit set, the schema of a table with thousands of columns (HMS, Paimon, ADBC and other external catalogs), a long ADBC name list and a long MaxCompute partition list are cached instead of being loaded again by every statement. ### Check List (For Author) - Test: Unit Test - ReflectiveObjectSizeEstimatorTest, HmsCacheSizeEstimatorTest (new), PaimonCacheSizeEstimatorTest, PaimonSchemaAtMemoTest, AdbcMetadataCacheTest, MaxComputePartitionCacheTest, ExternalCatalogMetaCacheTest - fe-connector-cache / hms / paimon / adbc / maxcompute / hive / hudi / iceberg module tests, 14 fe-core metadata and schema cache test classes - Behavior changed: Yes. Wide values that a weighted metadata cache used to reject are admitted, weighed as the sum of their elements. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Reviewed #69558 at 65757d9. No verified new findings and no existing blocking inline findings. The review converged after two rounds, including a final check of this frozen review payload.
All 17 changed Java files and related load, admission, refresh, invalidation and schema-construction paths were reviewed statically. No builds or tests were run, and no repository source was modified. There was no additional user-provided focus.
Critical checkpoint conclusions:
- Goal and evidence: selected long metadata lists are weighed per element, allowing wide values to pass complete weighted admission. Added tests exercise real admission/repeated reads, growth, shared-column de-duplication and oversized-element rejection; they were assessed statically.
- Scope and reuse: the shared estimator and wrappers centralize the change; connector integrations and the engine schema accessor are focused. No unrelated source changes were found.
- Concurrency and locks: Walkers and identity sets are invocation-local. Existing ClassValue plans, SQL-thread loads, refresh-executor work, scope/key generation fences and reservation replacement remain in place. Miss estimation is outside publication locks; replacement paths retain their existing guarded sections. No new lock ordering or concrete deadlock was identified.
- Lifecycle and initialization: temporary graph state is released with each estimate; entry reservations still release through existing discard/invalidation/close paths. Paimon retains its existing allowance for later store growth. No new static initialization dependency was introduced.
- Configuration: no settings were added. Existing global/catalog/entry weight limits still control admission; unweighted caches avoid the estimation path.
- Compatibility: no shared connector SPI, persisted format or FE/BE protocol was changed; the new helpers and engine methods are additive.
- Thrift/protobuf: no definition, field ID or enum changes.
- Parallel paths: HMS columns/partition keys, Paimon file-store/wrapper tables and schema-at snapshots, ADBC namespaces/names/Arrow fields, MaxCompute partitions, and default/Doris/plugin engine schemas were checked. Copied display columns are charged and reused column identities are de-duplicated. Untouched Paimon non-FileStoreTable row-type estimates retain their existing whole-graph limit.
- Special conditions and errors: collection selection uses retained object identity. Every selected element keeps the depth and visit limits; element or residual-graph failures produce incomplete estimates and reject weighted admission while returning the loaded metadata.
- Tests and negative cases: new unit coverage includes wide primitive/nested schemas, long names/partitions and shared display roots; existing depth, inaccessible-field and budget failure cases remain. The publication machinery is unchanged. Runtime behavior and test execution were not verified in this static-only review.
- Test result artifacts: no regression output files or generated expected results changed.
- Observability: existing weight-reject counts/reasons and cache diagnostics cover this path; no new distributed operation requires additional tracing.
- Transactions and persistence: no journal, replay, failover or transaction-state changes; committed-data visibility, visible versions and delete-bitmap semantics are unaffected by these estimator changes.
- Data writes and atomicity: no data mutation path changed; existing cache publication and reservation fencing remains authoritative.
- FE/BE propagation: no new cross-process variables or sending paths.
- Performance and memory: work occurs at weighted admission rather than cache hits. The outer lists are traversed linearly with bounded object visits per element; shared descendant graphs may be revisited and conservatively overcounted. De-duplication retains O(number of selected elements) identities in addition to one bounded element graph, so total temporary memory is not constant in list length. No material failure was established from this tradeoff.
- Other issues and focus: the final recheck found no additional substantiated issue. No user-specific focus or prior inline finding required a separate disposition.
Coverage limit: static source and test review only; no compilation, runtime reproduction or performance measurements were performed. No unresolved candidate or uncovered changed-file scope remains.
Review converged after 2 round(s).
Contributor
TPC-H: Total hot run time: 29102 ms |
Contributor
TPC-DS: Total hot run time: 150755 ms |
Contributor
ClickBench: Total hot run time: 24.26 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
CalvinKirs
approved these changes
Oct 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #66717, #67726
Problem Summary:
In short. With a metadata-cache weight limit set, the cached metadata of a wide table (thousands of columns), a long name list or a long partition list was never cached: its size estimate gave up and the value was reloaded on every statement. This PR weighs such values one element at a time, so they are admitted and charged for their size, while the safety bound of the estimate stays in place.
Background. Master (#67726) lets the metadata caches of external catalogs be bounded by weight: the FE-global
external_meta_cache_max_weight, a catalog'smeta.cache.max-weight, or an entry'smeta.cache.<engine>.<entry>.max-weight. Once any of them applies, each value is weighed before it is admitted. Several entries use the complete walk ofReflectiveObjectSizeEstimator, which visits every object of the value and gives up after 10,000 of them, so that one pathological value cannot cost unbounded CPU and temporary memory. An estimate that gives up is incomplete, and an incomplete value is returned to the query but not cached. branch-4.x (#66717) governs only a few entries and weighs them with per-element formulas, so it has no such limit.The problem, and what it cost. The 10,000-visit budget is reached by values that are large only because of one list, long before they are large in bytes. Measured on master, the first value that never enters a weighted cache:
hms-tablestruct<a:int,b:string>columnspaimon-table,paimon-schema-atROW<a INT, b STRING>columnsschema(default,dorisengines)adbc-table-schemaadbc-table-names,adbc-namespacesmax-compute-partitionFor such a table every statement goes back to the source (an HMS call, a Paimon schema file read, an ODPS partition listing, an ADBC metadata call), and
weight_reject_countininformation_schema.catalog_meta_cache_statisticskeeps growing. In a unit test with a 128 MiB global limit, three reads of a 2,000-column HMS table loaded it three times and cached nothing.How this PR fixes it.
ReflectiveObjectSizeEstimator.estimateCompletePerElement(root, collections...)walksrootlike the complete estimate, except that each listed collection it reaches is counted as its own storage plus every element estimated separately, each with its own 10,000-visit budget. An element that appears in more than one of the listed collections is counted once.MetaCacheSizeEstimators.reflectivePerElement(...)andreflectiveEachElement()wrap it as cache estimators.SchemaCacheValue.getColumnLists(): schema, and for plugin tables also partition and display columns), the ADBC table schema (fields) and name lists, and the MaxCompute partition list.Results.
weight_reject_count = 0The new tests also check that the estimate is never below an unbounded walk of the value and at most twice it, and that it doubles when the column count doubles. With the per-element walk disabled, all nine new wide-value tests fail; without the element de-duplication, a plugin schema with reused display columns weighs twice as much (2,052,208 to 4,104,168 bytes) and its test fails.
Release note
Fix: with a metadata-cache weight limit set, the schema of a table with thousands of columns (HMS, Paimon, ADBC and other external catalogs), a long ADBC name list and a long MaxCompute partition list are cached instead of being loaded again by every statement.
Check List (For Author)
Test
New and changed tests:
ReflectiveObjectSizeEstimatorTest(4 new),HmsCacheSizeEstimatorTest(new, 2),PaimonCacheSizeEstimatorTest,PaimonSchemaAtMemoTest,AdbcMetadataCacheTest,MaxComputePartitionCacheTest(1 new each),ExternalCatalogMetaCacheTest(2 new). All tests of fe-connector-cache (164), hms (163), paimon (847), adbc (191), maxcompute (157), hive (510), hudi (270) and iceberg (1689) pass, as do 14 fe-core metadata and schema cache test classes (204 tests). No regression test: a weight limit and a table with thousands of columns are needed to see the difference, and the unit tests drive the real cache admission path.Behavior changed:
Wide values that a weighted metadata cache used to reject are admitted and weighed as the sum of their elements.
Does this need documentation?
Check List (For Reviewer who merge this PR)