Skip to content

Commit de828c5

Browse files
morningmanclaude
andcommitted
[fix](fe) Reserve 16 KB for a Paimon REST data token
### What problem does this PR solve? Issue Number: None Related PR: #66717, #67726 Problem Summary: With `paimon.data-token.enabled=true` each cached Paimon table owns a `RESTTokenFileIO`, and the vended token it holds is fetched on first data access, after the table was weighed and admitted. The estimator reserved 4 KB for it, about twice the STS token seen in practice (key id, secret, a 1,200-character security token and the endpoint). The token map is whatever the REST server returns, so a server that vends a larger credential, or a refresh that brings one, leaves the table cached below its retained size. Paimon keeps one token per `RESTTokenFileIO` and replaces it on refresh, so the retained size is bounded by the largest single token, but Paimon gives no hook to reweigh the table when the token changes. Raise the fixed reserve to 16 KB, the allowance branch-4.x charges for a table's FileIO, and test it with both a first fetch and a refresh to a security token of 8,192 characters. ### Release note None ### Check List (For Author) - Test: Unit Test - `PaimonCacheSizeEstimatorTest.restDataTokenFileIoIsChargedToItsTable` now checks a 1,200- and an 8,192-character security token; with the old 4 KB reserve the second case fails (4,128 reserved, 9,232 retained). All 678 fe-connector-paimon tests pass locally. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 5b940b3 commit de828c5

2 files changed

Lines changed: 25 additions & 12 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCacheSizeEstimator.java‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -77,9 +77,11 @@ final class PaimonCacheSizeEstimator {
7777
private static final long STORE_GROWTH_KEY_BYTES = 192L;
7878
// RESTCatalog with data tokens gives each table its own RESTTokenFileIO. The FileIO it delegates to lives in
7979
// Paimon's process-wide cache, but the vended token it fetches on first data access (an STS key id, secret and
80-
// security token, about 2 KB) belongs to the table and arrives after admission.
80+
// security token, about 2 KB) belongs to the table and arrives after admission. A refresh replaces that token
81+
// instead of adding to it, and Paimon gives no hook to reweigh the table when it does, so the reserve is a
82+
// fixed 16 KB, several times any STS token seen so far.
8183
private static final long REST_TOKEN_FILE_IO_BYTES =
82-
JvmSizeUtils.saturatedAdd(JvmSizeUtils.instanceSize(RESTTokenFileIO.class), 4096L);
84+
JvmSizeUtils.saturatedAdd(JvmSizeUtils.instanceSize(RESTTokenFileIO.class), 16L * 1024L);
8385

8486
private PaimonCacheSizeEstimator() {
8587
}

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCacheSizeEstimatorTest.java‎

Lines changed: 21 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -108,18 +108,29 @@ void restDataTokenFileIoIsChargedToItsTable() throws Exception {
108108
fileIO, local.location(), local.schema(), CatalogEnvironment.empty());
109109

110110
long reserved = weight(rest) - weight(local);
111-
Map<String, String> token = new HashMap<>();
112-
token.put("fs.oss.accessKeyId", "STS." + "a".repeat(28));
113-
token.put("fs.oss.accessKeySecret", "b".repeat(44));
114-
token.put("fs.oss.securityToken", "c".repeat(1200));
115-
token.put("fs.oss.endpoint", "oss-cn-hangzhou-internal.aliyuncs.com");
111+
// The token arrives after admission and a refresh replaces it, so the one reserve must cover the token
112+
// the first data access fetches as well as a later, larger one. The 1,200-character security token is
113+
// the usual STS size; the 8,192-character one stands for a credential several times larger.
114+
// MUTATION: a 4 KB reserve -> the refreshed token outgrows it -> red.
115+
for (int securityTokenChars : new int[] {1200, 8192}) {
116+
Map<String, String> token = new HashMap<>();
117+
token.put("fs.oss.accessKeyId", "STS." + "a".repeat(28));
118+
token.put("fs.oss.accessKeySecret", "b".repeat(44));
119+
token.put("fs.oss.securityToken", "c".repeat(securityTokenChars));
120+
token.put("fs.oss.endpoint", "oss-cn-hangzhou-internal.aliyuncs.com");
121+
setToken(fileIO, new RESTToken(token, System.currentTimeMillis() + 3_600_000L));
122+
long owned = EstimatorCalibrationAssertions.graphSize(fileIO)
123+
- EstimatorCalibrationAssertions.graphSize(context);
124+
Assertions.assertTrue(reserved >= owned, "reserved " + reserved
125+
+ " does not cover the table-owned graph with a " + securityTokenChars + "-character token "
126+
+ owned);
127+
}
128+
}
129+
130+
private static void setToken(RESTTokenFileIO fileIO, RESTToken token) throws ReflectiveOperationException {
116131
Field tokenField = RESTTokenFileIO.class.getDeclaredField("token");
117132
tokenField.setAccessible(true);
118-
tokenField.set(fileIO, new RESTToken(token, System.currentTimeMillis() + 3_600_000L));
119-
long owned = EstimatorCalibrationAssertions.graphSize(fileIO)
120-
- EstimatorCalibrationAssertions.graphSize(context);
121-
Assertions.assertTrue(reserved >= owned,
122-
"reserved " + reserved + " does not cover the table-owned token graph " + owned);
133+
tokenField.set(fileIO, token);
123134
}
124135

125136
private void assertGrownDelta(String fixture, FileStoreTable small, FileStoreTable large) throws Exception {

0 commit comments

Comments
 (0)