Skip to content

Commit 65757d9

Browse files
morningmanclaude
andcommitted
[improvement](fe) Estimate wide metadata-cache values one element at a time
### What problem does this PR solve? Issue Number: None Related PR: #66717, #67726 Problem Summary: Once any metadata-cache weight limit is set (the FE-global external_meta_cache_max_weight, a catalog meta.cache.max-weight or an entry max-weight), every cached value is weighed before it is admitted. Master (#67726) weighs several entries with the complete walk of ReflectiveObjectSizeEstimator over the whole value, which gives up after 10,000 visited objects. A value that is large only because of one long list reaches that long before it is large in bytes, and an incomplete estimate is rejected: the value is returned to the query but never cached, so every statement loads it again (an HMS call, a Paimon schema file read, an ODPS partition listing) and weight_reject_count keeps growing. Measured, the first value that never enters a weighted cache: hms-table 995 INT / 382 struct<a:int,b:string> columns paimon-table, paimon-schema-at 2,497 INT / 908 ROW columns fe-core schema cache 3,332 INT / 832 struct columns adbc-table-schema 2,499 INT / 1,110 struct fields adbc-table-names, namespaces 9,999 names max-compute-partition 1,249 three-level partitions branch-4.x weighs its few governed entries with per-element formulas and has no such limit. The complete estimate can now walk the long lists of a value element by element: ReflectiveObjectSizeEstimator.estimateCompletePerElement counts a listed collection's own storage and estimates each element with its own visit budget, counting an element that appears in several of the lists once. The HMS table (columns and partition keys), the Paimon table schema and schema-at snapshot (fields), the fe-core schema cache (schema, partition and display columns), the ADBC table schema (fields) and name lists, and the MaxCompute partition list use it. The visit budget still bounds every element and the rest of the value, so a value that reaches a huge shared structure still fails fast, and the temporary identity set holds one element's graph at a time. Objects that elements share, such as a scalar type singleton, are counted once per element: the weight can be about 1.1x to 1.6x of the retained size, never below it. The whole-value walk and its 10,000-visit budget are unchanged for every other entry. ### Release note Fix: with a metadata-cache weight limit set, the schema of a table with thousands of columns (HMS, Paimon, ADBC and other external catalogs), a long ADBC name list and a long MaxCompute partition list are cached instead of being loaded again by every statement. ### Check List (For Author) - Test: Unit Test - ReflectiveObjectSizeEstimatorTest, HmsCacheSizeEstimatorTest (new), PaimonCacheSizeEstimatorTest, PaimonSchemaAtMemoTest, AdbcMetadataCacheTest, MaxComputePartitionCacheTest, ExternalCatalogMetaCacheTest - fe-connector-cache / hms / paimon / adbc / maxcompute / hive / hudi / iceberg module tests, 14 fe-core metadata and schema cache test classes - Behavior changed: Yes. Wide values that a weighted metadata cache used to reject are admitted, weighed as the sum of their elements. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 1bb3537 commit 65757d9

17 files changed

Lines changed: 538 additions & 9 deletions

File tree

‎fe/fe-connector/fe-connector-adbc/src/main/java/org/apache/doris/connector/adbc/AdbcMetadataCache.java‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626

2727
import org.apache.arrow.vector.types.pojo.Schema;
2828

29+
import java.util.Collections;
2930
import java.util.List;
3031
import java.util.Map;
3132
import java.util.Objects;
@@ -104,22 +105,25 @@ public AdbcMetadataCache(Map<String, String> properties) {
104105
AdbcMetadataCache(CatalogMetaCache owner, Map<String, String> properties) {
105106
this.owner = Objects.requireNonNull(owner, "owner can not be null");
106107
CacheSpec spec = cacheSpec(properties);
108+
// Long name lists and wide schemas would exhaust the visit budget of a whole-value walk, so these values
109+
// are estimated one namespace, name or field at a time.
107110
this.namespaces = owner.create(MetaCacheDefinition
108111
.<String, List<AdbcNamespace>>builder("adbc-namespaces", spec, ignored -> ScopePath.catalog())
109112
.budgetGroup(ENTRY)
110-
.sizeEstimator(MetaCacheSizeEstimators.reflective())
113+
.sizeEstimator(MetaCacheSizeEstimators.reflectiveEachElement())
111114
.build());
112115
this.tableNames = owner.create(MetaCacheDefinition
113116
.<TableNameKey, List<String>>builder("adbc-table-names", spec,
114117
key -> ScopePath.database(key.dorisDbName))
115118
.budgetGroup(ENTRY)
116-
.sizeEstimator(MetaCacheSizeEstimators.reflective())
119+
.sizeEstimator(MetaCacheSizeEstimators.reflectiveEachElement())
117120
.build());
118121
this.tableSchemas = owner.create(MetaCacheDefinition
119122
.<TableKey, Schema>builder("adbc-table-schema", spec,
120123
key -> ScopePath.table(key.dorisDbName, key.table))
121124
.budgetGroup(ENTRY)
122-
.sizeEstimator(MetaCacheSizeEstimators.reflective())
125+
.sizeEstimator(MetaCacheSizeEstimators.reflectivePerElement(
126+
schema -> Collections.singletonList(schema.getFields())))
123127
.build());
124128
}
125129

‎fe/fe-connector/fe-connector-adbc/src/test/java/org/apache/doris/connector/adbc/AdbcMetadataCacheTest.java‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
import org.junit.jupiter.api.Assertions;
2828
import org.junit.jupiter.api.Test;
2929

30+
import java.util.ArrayList;
3031
import java.util.Collections;
3132
import java.util.List;
3233
import java.util.Map;
@@ -72,6 +73,36 @@ void managedCachesAccountAllPayloadsAndReleaseReservations() {
7273
}
7374
}
7475

76+
@Test
77+
void wideSchemasAndLongNameListsAreCachedUnderAWeightLimit() {
78+
// A walk of the whole value ran out of the 10,000-visit budget at about 2,500 Arrow fields or 10,000 names,
79+
// so such a value was asked from the source on every statement once a weight limit was set. MUTATION:
80+
// estimating the value as one graph -> two calls and a rejected admission -> red.
81+
Map<String, String> properties = Map.of("meta.cache.max-weight", "64MB");
82+
try (CatalogMetaCache owner = CatalogMetaCache.managed(91090L, "adbc", properties)) {
83+
AdbcMetadataCache cache = new AdbcMetadataCache(owner, properties);
84+
List<Field> fields = new ArrayList<>();
85+
for (int i = 0; i < 3_000; i++) {
86+
fields.add(Field.nullable("col_" + i, new ArrowType.Int(32, true)));
87+
}
88+
List<String> tableNames = new ArrayList<>();
89+
for (int i = 0; i < 20_000; i++) {
90+
tableNames.add("table_" + i);
91+
}
92+
Counting<Schema> schema = new Counting<>(new Schema(fields));
93+
Counting<List<String>> names = new Counting<>(tableNames);
94+
for (int i = 0; i < 2; i++) {
95+
cache.tableSchema(MAIN_T1, schema);
96+
cache.tableNames(MAIN, names);
97+
}
98+
99+
Assertions.assertEquals(1, schema.calls);
100+
Assertions.assertEquals(1, names.calls);
101+
Assertions.assertEquals(0L, owner.entries().get("adbc-table-schema").metrics().getWeightRejectCount());
102+
Assertions.assertEquals(0L, owner.entries().get("adbc-table-names").metrics().getWeightRejectCount());
103+
}
104+
}
105+
75106
@Test
76107
void rejectedSchemaIsReturnedWithoutBeingCached() {
77108
Map<String, String> properties = Map.of("meta.cache.adbc.metadata.max-weight", "1");

‎fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/MetaCacheSizeEstimators.java‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,10 @@
1717

1818
package org.apache.doris.connector.cache;
1919

20+
import java.util.Collection;
21+
import java.util.Collections;
22+
import java.util.function.Function;
23+
2024
/** Shared estimators for small or structurally unknown metadata-cache entries. */
2125
public final class MetaCacheSizeEstimators {
2226
private MetaCacheSizeEstimators() {
@@ -32,4 +36,22 @@ public static <K, V> MetaCacheSizeEstimator<K, V> reflective() {
3236
ReflectiveObjectSizeEstimator.estimateComplete(key),
3337
ReflectiveObjectSizeEstimator.estimateComplete(value)));
3438
}
39+
40+
/**
41+
* Returns a {@link #reflective()} estimator that walks the long lists {@code perElementCollections} picks from
42+
* a value element by element (see {@link ReflectiveObjectSizeEstimator#estimateCompletePerElement}), for values
43+
* that grow with one list, such as a wide table's schema.
44+
*/
45+
public static <K, V> MetaCacheSizeEstimator<K, V> reflectivePerElement(
46+
Function<? super V, ? extends Collection<? extends Collection<?>>> perElementCollections) {
47+
return (key, value) -> MetaCacheSizeEstimate.complete(JvmSizeUtils.saturatedAdd(
48+
ReflectiveObjectSizeEstimator.estimateComplete(key),
49+
ReflectiveObjectSizeEstimator.estimateCompletePerElement(value,
50+
perElementCollections.apply(value).toArray(new Collection<?>[0]))));
51+
}
52+
53+
/** {@link #reflectivePerElement} for a value that is itself the long list, such as a partition list. */
54+
public static <K, V extends Collection<?>> MetaCacheSizeEstimator<K, V> reflectiveEachElement() {
55+
return reflectivePerElement(value -> Collections.singletonList(value));
56+
}
3557
}

‎fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/ReflectiveObjectSizeEstimator.java‎

Lines changed: 48 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
import java.nio.ByteBuffer;
2424
import java.util.ArrayList;
2525
import java.util.Collection;
26+
import java.util.Collections;
2627
import java.util.HashMap;
2728
import java.util.HashSet;
2829
import java.util.IdentityHashMap;
@@ -48,7 +49,9 @@
4849
* The diagnostic API samples collections and object arrays. The admission API instead traverses every element
4950
* up to a fixed visit budget: this catches skewed tail values while bounding CPU and temporary identity-set
5051
* memory. Strongly encapsulated reference fields, depth truncation, and visit-budget exhaustion fail the
51-
* estimate so admission can reject the value without turning an incomplete graph into a false low weight.
52+
* estimate so admission can reject the value without turning an incomplete graph into a false low weight. A value
53+
* that is large only because of a few long lists, such as a wide table's columns, is estimated with those lists
54+
* walked element by element instead, so the budget bounds one element rather than the whole value.
5255
*
5356
* <p>This is intentionally a construction-time safety net, not a Caffeine hit-path weigher. Cache values should
5457
* combine it with a type-specific estimate once, store the larger result, and expose that stored number to
@@ -141,14 +144,36 @@ public static long estimateComplete(Object root, int visitBudget, int maxDepth)
141144
return new Walker(Integer.MAX_VALUE, visitBudget).estimate(root, maxDepth);
142145
}
143146

147+
/**
148+
* Completely estimates {@code root} like {@link #estimateComplete(Object)}, except that each of
149+
* {@code perElementCollections} that the walk reaches is estimated element by element, every element with its
150+
* own visit budget. A value that is large only because of such lists, such as a wide table's columns or a long
151+
* partition list, stays admissible, while the budget still bounds each element and the rest of the graph, and
152+
* the temporary identity set holds one element's graph at a time. Objects that elements share are counted
153+
* once per element, which can only over-estimate; an element in several of the collections is counted once.
154+
*/
155+
public static long estimateCompletePerElement(Object root, Collection<?>... perElementCollections) {
156+
Set<Object> collections = Collections.newSetFromMap(new IdentityHashMap<>());
157+
Collections.addAll(collections, perElementCollections);
158+
return new Walker(Integer.MAX_VALUE, DEFAULT_COMPLETE_VISIT_BUDGET, collections)
159+
.estimate(root, DEFAULT_MAX_DEPTH);
160+
}
161+
144162
private static final class Walker {
145163
private final int sampleSize;
146164
private final int visitBudget;
147165
private final Map<Object, Boolean> visited = new IdentityHashMap<>();
166+
private final Set<Object> perElementCollections;
167+
private final Set<Object> estimatedElements = Collections.newSetFromMap(new IdentityHashMap<>());
148168

149169
private Walker(int sampleSize, int visitBudget) {
170+
this(sampleSize, visitBudget, Collections.emptySet());
171+
}
172+
173+
private Walker(int sampleSize, int visitBudget, Set<Object> perElementCollections) {
150174
this.sampleSize = sampleSize;
151175
this.visitBudget = visitBudget;
176+
this.perElementCollections = perElementCollections;
152177
}
153178

154179
private long estimate(Object value, int depth) {
@@ -159,6 +184,9 @@ private long estimate(Object value, int depth) {
159184
throw new IllegalStateException(
160185
"Complete estimate exceeded visit budget " + visitBudget);
161186
}
187+
if (perElementCollections.contains(value)) {
188+
return estimateEachElement((Collection<?>) value, depth);
189+
}
162190

163191
Class<?> type = value.getClass();
164192
if (type.isEnum() || value instanceof Class<?>) {
@@ -230,6 +258,25 @@ private long estimateArray(Object value, int depth) {
230258
return add(bytes, scale(sampledBytes, samples, length));
231259
}
232260

261+
private long estimateEachElement(Collection<?> values, int depth) {
262+
int size = values.size();
263+
long bytes = add(
264+
JvmSizeUtils.instanceSize(values.getClass()),
265+
collectionStorageBytes(values, size));
266+
if (size == 0) {
267+
return bytes;
268+
}
269+
if (depth == 0) {
270+
throw incomplete(values.getClass());
271+
}
272+
for (Object element : values) {
273+
if (estimatedElements.add(element)) {
274+
bytes = add(bytes, new Walker(Integer.MAX_VALUE, visitBudget).estimate(element, depth - 1));
275+
}
276+
}
277+
return bytes;
278+
}
279+
233280
private long estimateByteBuffer(ByteBuffer value, int depth) {
234281
long bytes = JvmSizeUtils.instanceSize(value.getClass());
235282
byte[] backing = JvmSizeUtils.byteBufferBackingArray(value);

‎fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/ReflectiveObjectSizeEstimatorTest.java‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,9 @@
2323
import java.net.URI;
2424
import java.nio.ByteBuffer;
2525
import java.util.AbstractList;
26+
import java.util.ArrayList;
2627
import java.util.LinkedHashMap;
28+
import java.util.List;
2729
import java.util.Map;
2830
import java.util.RandomAccess;
2931
import java.util.concurrent.atomic.AtomicInteger;
@@ -129,6 +131,88 @@ void depthTruncationMakesTheEstimateIncomplete() {
129131
Assertions.assertTrue(estimate.getIncompleteReason().startsWith("depth_limit:"));
130132
}
131133

134+
@Test
135+
void perElementEstimateCompletesALongListThatTheWholeValueWalkRejects() {
136+
// A value that is large only because of one long list, such as a wide table's columns, has the list walked
137+
// element by element, each element under its own visit budget. MUTATION: walking the list as part of the
138+
// value's graph -> the visit budget runs out -> red.
139+
Holder small = holderOf(1_000);
140+
Holder large = holderOf(30_000);
141+
MetaCacheSizeEstimate whole = MetaCacheSizeEstimator.estimateSafely("whole_value",
142+
() -> MetaCacheSizeEstimate.complete(ReflectiveObjectSizeEstimator.estimateComplete(large)));
143+
Assertions.assertFalse(whole.isComplete());
144+
145+
long smallBytes = ReflectiveObjectSizeEstimator.estimateCompletePerElement(small, small.values);
146+
long largeBytes = ReflectiveObjectSizeEstimator.estimateCompletePerElement(large, large.values);
147+
148+
// The elements share no object, so the estimate is exactly an unbounded walk of the value.
149+
Assertions.assertEquals(ReflectiveObjectSizeEstimator.estimateComplete(large, Integer.MAX_VALUE, 20),
150+
largeBytes);
151+
Assertions.assertTrue(largeBytes > smallBytes * 25, "the estimate must grow with the list");
152+
}
153+
154+
@Test
155+
void perElementEstimateCountsAnElementInTwoOfTheListsOnce() {
156+
// A schema's display columns reuse most of its column objects. MUTATION: estimating every element of every
157+
// listed collection -> the shared elements count twice -> red.
158+
Holder holder = holderOf(20_000);
159+
holder.view = new ArrayList<>(holder.values);
160+
161+
Assertions.assertEquals(ReflectiveObjectSizeEstimator.estimateComplete(holder, Integer.MAX_VALUE, 20),
162+
ReflectiveObjectSizeEstimator.estimateCompletePerElement(holder, holder.values, holder.view));
163+
}
164+
165+
@Test
166+
void perElementEstimateStillBoundsEachElement() {
167+
AtomicInteger reads = new AtomicInteger();
168+
Holder holder = holderOf(10);
169+
holder.values.add(new CountingList(reads));
170+
171+
MetaCacheSizeEstimate estimate = MetaCacheSizeEstimator.estimateSafely("bounded_element",
172+
() -> MetaCacheSizeEstimate.complete(
173+
ReflectiveObjectSizeEstimator.estimateCompletePerElement(holder, holder.values)));
174+
175+
Assertions.assertFalse(estimate.isComplete());
176+
Assertions.assertTrue(reads.get() <= 10_001, "one element's walk must stop at the visit budget");
177+
}
178+
179+
@Test
180+
void eachElementEstimatorWeighsAValueThatIsTheLongList() {
181+
List<Object> partitions = holderOf(30_000).values;
182+
183+
MetaCacheSizeEstimate estimate = MetaCacheSizeEstimators
184+
.<String, List<Object>>reflectiveEachElement().estimate("key", partitions);
185+
186+
Assertions.assertTrue(estimate.isComplete(), estimate.getIncompleteReason());
187+
Assertions.assertEquals(ReflectiveObjectSizeEstimator.estimateComplete("key")
188+
+ ReflectiveObjectSizeEstimator.estimateComplete(partitions, Integer.MAX_VALUE, 20),
189+
estimate.getBytes());
190+
}
191+
192+
private static Holder holderOf(int elements) {
193+
Holder holder = new Holder();
194+
for (int i = 0; i < elements; i++) {
195+
holder.values.add(new Element("element-" + i, 1_000L + i));
196+
}
197+
return holder;
198+
}
199+
200+
private static final class Holder {
201+
private final String name = "holder";
202+
private final List<Object> values = new ArrayList<>();
203+
private List<Object> view;
204+
}
205+
206+
private static final class Element {
207+
private final String name;
208+
private final Long id;
209+
210+
private Element(String name, Long id) {
211+
this.name = name;
212+
this.id = id;
213+
}
214+
}
215+
132216
private static final class CountingList extends AbstractList<String> implements RandomAccess {
133217
private final AtomicInteger reads;
134218

‎fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsCacheSizeEstimator.java‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,12 @@ private HmsCacheSizeEstimator() {
3232
}
3333

3434
static MetaCacheSizeEstimate estimateTable(Object key, HmsTableInfo value) {
35-
return complete(key, value);
35+
// A whole-value walk of a wide table would exhaust the visit budget and keep the table out of a weighted
36+
// cache, so each column is estimated on its own.
37+
return MetaCacheSizeEstimate.complete(add(
38+
ReflectiveObjectSizeEstimator.estimateComplete(key),
39+
ReflectiveObjectSizeEstimator.estimateCompletePerElement(
40+
value, value.getColumns(), value.getPartitionKeys())));
3641
}
3742

3843
static MetaCacheSizeEstimate estimatePartition(Object key, HmsPartitionInfo value) {

0 commit comments

Comments
 (0)