Skip to content

Commit 742c58b

Browse files
morningmanclaude
andcommitted
[test](fe) Pin shared budget groups, concurrent failures and the Iceberg comment cache limit
### What problem does this PR solve? Issue Number: None Related PR: #66717, #67726 Problem Summary: Three of the ported metadata cache budget tests could pass while the behavior they guard is broken: - `concurrentReservationsNeverExceedTheGlobalLimit` dropped the futures of its 200 submitted reservations. If half of them threw instead of returning an empty Optional, the reservation count and the global weight would still match. The test now calls `get()` on every future before checking the totals. - The bucket lifecycle test gives every physical cache its own budget group, so it cannot see a group bucket removed while another cache of the same group is still live. Iceberg's MVCC and list partition views share the `partition_view` group; if closing one of them dropped the bucket, a re-created cache would start from an empty bucket and admit weight on top of what its sibling still holds. `sharedEntryGroupKeepsItsLimitUntilItsLastPhysicalCacheCloses` keeps 60 bytes reserved in one sibling, closes and re-creates the other, and checks that it can take only the remaining 40 and that the bucket goes away with the last sibling. - The Iceberg catalog-limit check built only plain catalogs, which never create the `iceberg-comment` cache. A REST vended-credentials catalog does, through the same governed owner; the test now builds one and checks that the comment cache gets the catalog limit. ### Release note None ### Check List (For Author) - Test: Unit Test - `MetaCacheBudgetManagerTest` (13) and `IcebergConnectorCacheTest` (31) pass locally. Removing the live-sibling check from `MetaCacheBudgetManager#close` turns the new shared group test red, and building the comment cache outside the catalog's governed owner turns the catalog-limit test red. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 5d76b69 commit 742c58b

2 files changed

Lines changed: 43 additions & 2 deletions

File tree

‎fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheBudgetManagerTest.java‎

Lines changed: 36 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
import java.util.concurrent.CountDownLatch;
3535
import java.util.concurrent.ExecutorService;
3636
import java.util.concurrent.Executors;
37+
import java.util.concurrent.Future;
3738
import java.util.concurrent.TimeUnit;
3839
import java.util.concurrent.atomic.AtomicInteger;
3940

@@ -81,16 +82,21 @@ void concurrentReservationsNeverExceedTheGlobalLimit() throws Exception {
8182
ExecutorService executor = Executors.newFixedThreadPool(8);
8283
CountDownLatch start = new CountDownLatch(1);
8384
List<AdmissionReservation> reservations = Collections.synchronizedList(new ArrayList<>());
85+
List<Future<?>> attempts = new ArrayList<>();
8486
try {
8587
for (int i = 0; i < 200; i++) {
86-
executor.submit(() -> {
88+
attempts.add(executor.submit(() -> {
8789
await(start);
8890
budget.tryReserve(1L).ifPresent(reservations::add);
89-
});
91+
}));
9092
}
9193
start.countDown();
9294
executor.shutdown();
9395
Assertions.assertTrue(executor.awaitTermination(10L, TimeUnit.SECONDS));
96+
// A refused attempt returns an empty Optional; an attempt that threw would surface here.
97+
for (Future<?> attempt : attempts) {
98+
attempt.get();
99+
}
94100
Assertions.assertEquals(100, reservations.size());
95101
Assertions.assertEquals(100L, manager.getGlobalUsedWeight());
96102
} finally {
@@ -404,6 +410,34 @@ void catalogAndGroupBucketsLiveExactlyAsLongAsTheirEntries() throws Exception {
404410
Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
405411
}
406412

413+
@Test
414+
void sharedEntryGroupKeepsItsLimitUntilItsLastPhysicalCacheCloses() throws Exception {
415+
// Iceberg's two physical partition-view caches share the partition_view entry limit.
416+
MetaCacheBudgetManager manager = new MetaCacheBudgetManager(NONE);
417+
EntryBudget mvccView = manager.createEntryBudget(
418+
1L, "iceberg", "mvcc-partition-view", "partition_view", NONE, OptionalLong.of(100L));
419+
EntryBudget listView = manager.createEntryBudget(
420+
1L, "iceberg", "list-partitions-view", "partition_view", NONE, OptionalLong.of(100L));
421+
AdmissionReservation held = mvccView.tryReserve(60L).get();
422+
423+
// Closing one sibling keeps the group bucket, so a re-created sibling still shares the 60 bytes held.
424+
listView.close();
425+
Assertions.assertEquals(1, privateMap(manager, "entryGroupBuckets").size());
426+
EntryBudget recreated = manager.createEntryBudget(
427+
1L, "iceberg", "list-partitions-view", "partition_view", NONE, OptionalLong.of(100L));
428+
Assertions.assertFalse(recreated.tryReserve(41L).isPresent(),
429+
"a fresh group bucket would admit 41 bytes on top of the live sibling's 60");
430+
AdmissionReservation fits = recreated.tryReserve(40L).get();
431+
432+
// The group bucket goes away only with its last physical cache.
433+
fits.release();
434+
held.release();
435+
mvccView.close();
436+
Assertions.assertEquals(1, privateMap(manager, "entryGroupBuckets").size());
437+
recreated.close();
438+
Assertions.assertTrue(privateMap(manager, "entryGroupBuckets").isEmpty());
439+
}
440+
407441
private static Map<?, ?> privateMap(MetaCacheBudgetManager manager, String fieldName)
408442
throws ReflectiveOperationException {
409443
Field field = MetaCacheBudgetManager.class.getDeclaredField(fieldName);

‎fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -250,6 +250,13 @@ public void weightLimitsReachEveryConnectorOwnedCache() throws Exception {
250250
Assertions.assertEquals(Long.valueOf(oneMb), viewLimited.get("iceberg.list-partitions-view"),
251251
"both physical partition views belong to the partition_view entry");
252252
Assertions.assertEquals(Long.valueOf(-1L), viewLimited.get("iceberg-partition"));
253+
254+
// Only a REST vended-credentials catalog builds the comment cache; the catalog limit bounds it as well.
255+
Map<String, String> vended = props("meta.cache.max-weight", "1MB");
256+
vended.put(IcebergCatalogProperties.ICEBERG_CATALOG_TYPE, IcebergCatalogProperties.TYPE_REST);
257+
vended.put("iceberg.rest.vended-credentials-enabled", "true");
258+
Assertions.assertEquals(Long.valueOf(oneMb), maxWeightsOf(vended).get("iceberg-comment"),
259+
"a catalog limit must bound iceberg-comment");
253260
}
254261

255262
/** MAX_WEIGHT of every cache one connector registers: its effective limit, or -1 when only count-bounded. */

0 commit comments

Comments
 (0)