You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 65757d9
Browse filesBrowse the repository at this point in the historyBrowse files
[improvement](fe) Estimate wide metadata-cache values one element at a time
### What problem does this PR solve?
Issue Number: None
Related PR: #66717, #67726
Problem Summary: Once any metadata-cache weight limit is set (the FE-global
external_meta_cache_max_weight, a catalog meta.cache.max-weight or an entry
max-weight), every cached value is weighed before it is admitted. Master
(#67726) weighs several entries with the complete walk of
ReflectiveObjectSizeEstimator over the whole value, which gives up after
10,000 visited objects. A value that is large only because of one long list
reaches that long before it is large in bytes, and an incomplete estimate is
rejected: the value is returned to the query but never cached, so every
statement loads it again (an HMS call, a Paimon schema file read, an ODPS
partition listing) and weight_reject_count keeps growing. Measured, the
first value that never enters a weighted cache:
hms-table 995 INT / 382 struct<a:int,b:string> columns
paimon-table, paimon-schema-at 2,497 INT / 908 ROW columns
fe-core schema cache 3,332 INT / 832 struct columns
adbc-table-schema 2,499 INT / 1,110 struct fields
adbc-table-names, namespaces 9,999 names
max-compute-partition 1,249 three-level partitions
branch-4.x weighs its few governed entries with per-element formulas and
has no such limit.
The complete estimate can now walk the long lists of a value element by
element: ReflectiveObjectSizeEstimator.estimateCompletePerElement counts a
listed collection's own storage and estimates each element with its own
visit budget, counting an element that appears in several of the lists
once. The HMS table (columns and partition keys), the Paimon table schema
and schema-at snapshot (fields), the fe-core schema cache (schema, partition
and display columns), the ADBC table schema (fields) and name lists, and the
MaxCompute partition list use it. The visit budget still bounds every
element and the rest of the value, so a value that reaches a huge shared
structure still fails fast, and the temporary identity set holds one
element's graph at a time. Objects that elements share, such as a scalar
type singleton, are counted once per element: the weight can be about 1.1x
to 1.6x of the retained size, never below it. The whole-value walk and its
10,000-visit budget are unchanged for every other entry.
### Release note
Fix: with a metadata-cache weight limit set, the schema of a table with
thousands of columns (HMS, Paimon, ADBC and other external catalogs), a long
ADBC name list and a long MaxCompute partition list are cached instead of
being loaded again by every statement.
### Check List (For Author)
- Test: Unit Test
- ReflectiveObjectSizeEstimatorTest, HmsCacheSizeEstimatorTest (new), PaimonCacheSizeEstimatorTest, PaimonSchemaAtMemoTest, AdbcMetadataCacheTest, MaxComputePartitionCacheTest, ExternalCatalogMetaCacheTest
- fe-connector-cache / hms / paimon / adbc / maxcompute / hive / hudi / iceberg module tests, 14 fe-core metadata and schema cache test classes
- Behavior changed: Yes. Wide values that a weighted metadata cache used to reject are admitted, weighed as the sum of their elements.
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-adbc/src/test/java/org/apache/doris/connector/adbc/AdbcMetadataCacheTest.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/MetaCacheSizeEstimators.java
+22Lines changed: 22 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -17,6 +17,10 @@
17
17
18
18
packageorg.apache.doris.connector.cache;
19
19
20
+
importjava.util.Collection;
21
+
importjava.util.Collections;
22
+
importjava.util.function.Function;
23
+
20
24
/** Shared estimators for small or structurally unknown metadata-cache entries. */
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/ReflectiveObjectSizeEstimator.java
+48-1Lines changed: 48 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -23,6 +23,7 @@
23
23
importjava.nio.ByteBuffer;
24
24
importjava.util.ArrayList;
25
25
importjava.util.Collection;
26
+
importjava.util.Collections;
26
27
importjava.util.HashMap;
27
28
importjava.util.HashSet;
28
29
importjava.util.IdentityHashMap;
@@ -48,7 +49,9 @@
48
49
* The diagnostic API samples collections and object arrays. The admission API instead traverses every element
49
50
* up to a fixed visit budget: this catches skewed tail values while bounding CPU and temporary identity-set
50
51
* memory. Strongly encapsulated reference fields, depth truncation, and visit-budget exhaustion fail the
51
-
* estimate so admission can reject the value without turning an incomplete graph into a false low weight.
52
+
* estimate so admission can reject the value without turning an incomplete graph into a false low weight. A value
53
+
* that is large only because of a few long lists, such as a wide table's columns, is estimated with those lists
54
+
* walked element by element instead, so the budget bounds one element rather than the whole value.
52
55
*
53
56
* <p>This is intentionally a construction-time safety net, not a Caffeine hit-path weigher. Cache values should
54
57
* combine it with a type-specific estimate once, store the larger result, and expose that stored number to
@@ -141,14 +144,36 @@ public static long estimateComplete(Object root, int visitBudget, int maxDepth)
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/ReflectiveObjectSizeEstimatorTest.java
0 commit comments