Skip to content

Commit ce25717

Browse files
morningmanclaude
andcommitted
[fix](fe) Charge every Paimon table for its FileIO and reserve partial-update growth
### What problem does this PR solve? Issue Number: None Related PR: #66717, #67726 Problem Summary: The Paimon table cache estimator charged a FileIO only when it was a RESTTokenFileIO, but other catalogs give each table its own FileIO as well. A REST catalog without data tokens creates a ResolvingFileIO per table (`RESTCatalog.fileIOFromOptions`), and a Hive catalog resolves one per table location (`HiveCatalog.fileIO(Path)`). Both build their backend maps after the table was weighed, so a weight-limited cache of such tables holds more heap than its limit allows. Paimon offers no IO-free way to tell an owned FileIO from one the catalog shares, so every table now carries the fixed 16 KB FileIO allowance, as on branch-4.x. Charging a shared FileIO to each table only makes the estimate conservative. A partial-update table also builds a merge function factory with its store, which keeps sequence-group and aggregator maps for every field. On Paimon 1.4.2 a 300-field partial-update table grows about 460 bytes per field once a scan has planned, about 150 more than the other merge engines, while the store reserve allows 320. Reserve another 256 bytes per field for partial-update tables. ### Release note Under an external metadata cache weight limit, every cached Paimon table also counts a fixed 16 KB for its FileIO, and partial-update tables count more per field, so fewer Paimon tables fit in the same limit. ### Check List (For Author) - Test: Unit Test - PaimonCacheSizeEstimatorTest: tables on LocalFileIO, ResolvingFileIO, HadoopFileIO and RESTTokenFileIO carry the same FileIO allowance, which covers a ResolvingFileIO with its backend and an 8,192-character REST token; the grown-graph check now leaves the FileIO allowance out and includes 300-field partial-update and aggregation tables. Charging only RESTTokenFileIO, dropping the partial-update reserve, or a 4 KB allowance each make it fail. All 679 fe-connector-paimon tests pass locally. - Behavior changed: Yes (see Release note) - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent de828c5 commit ce25717

2 files changed

Lines changed: 108 additions & 34 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCacheSizeEstimator.java‎

Lines changed: 30 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,11 @@
2121
import org.apache.doris.connector.cache.MetaCacheSizeEstimate;
2222
import org.apache.doris.connector.cache.ReflectiveObjectSizeEstimator;
2323

24+
import org.apache.paimon.CoreOptions;
2425
import org.apache.paimon.catalog.Identifier;
2526
import org.apache.paimon.fs.FileIO;
2627
import org.apache.paimon.fs.Path;
2728
import org.apache.paimon.privilege.PrivilegedFileStoreTable;
28-
import org.apache.paimon.rest.RESTTokenFileIO;
2929
import org.apache.paimon.schema.TableSchema;
3030
import org.apache.paimon.table.CatalogEnvironment;
3131
import org.apache.paimon.table.DelegatedFileStoreTable;
@@ -54,16 +54,18 @@
5454
* Retained-size formulas for Paimon table-cache entries.
5555
*
5656
* <p>The table's shallow size includes references to FileIO, catalog loaders, and lock factories,
57-
* but their graphs are catalog-scoped executable services rather than entry-owned metadata. Walking
58-
* those graphs both double-counts shared state and reaches strongly encapsulated JDK objects. The
59-
* estimator therefore expands only immutable metadata owned by the entry.
57+
* but their graphs are executable services rather than entry-owned metadata. Walking those graphs
58+
* both double-counts shared state and reaches strongly encapsulated JDK objects. The estimator
59+
* therefore expands only immutable metadata owned by the entry, plus a fixed allowance for a FileIO
60+
* the table may own.
6061
*
6162
* <p>A cached value is weighed once, at admission, but a {@link FileStoreTable} keeps growing afterwards: the
6263
* first {@code latestSnapshot()} or scan builds its transient store (a {@code FileStore} holding its own
6364
* {@code RowType} copies of the schema and a copy of the options) and field lookups build the four lazy index
6465
* maps of every {@code RowType}. That growth is reserved from the schema's shape. On Paimon 1.4.2 it measures
65-
* about 1-2 KB per table, 200-270 bytes per field, another 200 bytes per field nested in a row and 40 bytes per
66-
* option; the constants below keep a margin above that.
66+
* about 1-2 KB per table, 200-270 bytes per field, another 200 bytes per field nested in a row, 40 bytes per
67+
* option and, for a partial-update table's merge function, about 150 more bytes per field; the constants below
68+
* keep a margin above that.
6769
*/
6870
final class PaimonCacheSizeEstimator {
6971
private static final long STORE_GROWTH_BASE_BYTES = 2048L;
@@ -75,13 +77,17 @@ final class PaimonCacheSizeEstimator {
7577
// A partition, primary or bucket key: its slot in the derived key types; a primary key is also copied as a
7678
// renamed key field.
7779
private static final long STORE_GROWTH_KEY_BYTES = 192L;
78-
// RESTCatalog with data tokens gives each table its own RESTTokenFileIO. The FileIO it delegates to lives in
79-
// Paimon's process-wide cache, but the vended token it fetches on first data access (an STS key id, secret and
80-
// security token, about 2 KB) belongs to the table and arrives after admission. A refresh replaces that token
81-
// instead of adding to it, and Paimon gives no hook to reweigh the table when it does, so the reserve is a
82-
// fixed 16 KB, several times any STS token seen so far.
83-
private static final long REST_TOKEN_FILE_IO_BYTES =
84-
JvmSizeUtils.saturatedAdd(JvmSizeUtils.instanceSize(RESTTokenFileIO.class), 16L * 1024L);
80+
// A partial-update table's merge function factory, built with its store, keeps sequence-group and aggregator
81+
// maps for every field: about 150 bytes per field more than the other merge engines on Paimon 1.4.2.
82+
private static final long STORE_GROWTH_PARTIAL_UPDATE_FIELD_BYTES = 256L;
83+
// A table can own its FileIO: a REST catalog gives each table a RESTTokenFileIO (holding the vended token it
84+
// fetches on first data access, an STS key id, secret and security token of about 2 KB) or a ResolvingFileIO,
85+
// and a Hive catalog resolves a FileIO for each table location. Their tokens and backend maps appear only after
86+
// admission, and Paimon offers no IO-free way to read that state or to tell an owned FileIO from one the
87+
// catalog shares. Every table is therefore charged a fixed 16 KB for its FileIO, several times any STS token
88+
// seen so far; a token refresh replaces the token rather than adding to it. Charging a catalog-shared FileIO
89+
// to each table only makes the estimate conservative.
90+
private static final long FILE_IO_BYTES = 16L * 1024L;
8591

8692
private PaimonCacheSizeEstimator() {
8793
}
@@ -134,7 +140,7 @@ private static long estimateFileStoreTable(FileStoreTable table, Set<Object> vis
134140
}
135141
bytes = add(bytes, estimateCompleteOnce(table.schema(), visited));
136142
bytes = add(bytes, estimateStoreGrowth(table.schema()));
137-
bytes = add(bytes, estimateTableOwnedFileIO(table.fileIO(), visited));
143+
bytes = add(bytes, estimateTableFileIO(table.fileIO(), visited));
138144
bytes = add(bytes, estimatePath(table.location(), visited));
139145
return add(bytes, estimateCatalogEnvironment(table.catalogEnvironment(), visited));
140146
}
@@ -148,6 +154,9 @@ private static long estimateStoreGrowth(TableSchema schema) {
148154
for (String primaryKey : schema.primaryKeys()) {
149155
bytes = add(bytes, JvmSizeUtils.stringSize(primaryKey));
150156
}
157+
if (CoreOptions.fromMap(schema.options()).mergeEngine() == CoreOptions.MergeEngine.PARTIAL_UPDATE) {
158+
bytes = add(bytes, multiply(schema.fields().size(), STORE_GROWTH_PARTIAL_UPDATE_FIELD_BYTES));
159+
}
151160
return add(bytes, estimateFieldGrowth(schema.fields(), STORE_GROWTH_FIELD_BYTES));
152161
}
153162

@@ -185,8 +194,13 @@ private static long estimateChildTypeGrowth(DataType child) {
185194
return add(STORE_GROWTH_TYPE_NODE_BYTES, estimateTypeGrowth(child));
186195
}
187196

188-
private static long estimateTableOwnedFileIO(FileIO fileIO, Set<Object> visited) {
189-
return fileIO instanceof RESTTokenFileIO && visited.add(fileIO) ? REST_TOKEN_FILE_IO_BYTES : 0L;
197+
/** The allowance a table carries for its FileIO; the fallback branches of one table share theirs. */
198+
private static long estimateTableFileIO(FileIO fileIO, Set<Object> visited) {
199+
return visited.add(fileIO) ? estimateFileIO(fileIO) : 0L;
200+
}
201+
202+
static long estimateFileIO(FileIO fileIO) {
203+
return add(JvmSizeUtils.instanceSize(fileIO.getClass()), FILE_IO_BYTES);
190204
}
191205

192206
private static long estimateCompleteOnce(Object value, Set<Object> visited) {

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCacheSizeEstimatorTest.java‎

Lines changed: 78 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,10 @@
2121

2222
import org.apache.paimon.catalog.CatalogContext;
2323
import org.apache.paimon.catalog.Identifier;
24+
import org.apache.paimon.fs.FileIO;
2425
import org.apache.paimon.fs.Path;
26+
import org.apache.paimon.fs.ResolvingFileIO;
27+
import org.apache.paimon.fs.hadoop.HadoopFileIO;
2528
import org.apache.paimon.fs.local.LocalFileIO;
2629
import org.apache.paimon.options.Options;
2730
import org.apache.paimon.rest.RESTToken;
@@ -44,6 +47,7 @@
4447

4548
import java.lang.reflect.Field;
4649
import java.util.ArrayList;
50+
import java.util.Arrays;
4751
import java.util.HashMap;
4852
import java.util.List;
4953
import java.util.Map;
@@ -57,9 +61,11 @@ class PaimonCacheSizeEstimatorTest {
5761
@Test
5862
void admissionWeightCoversTheStoreAndIndexesBuiltAfterAdmission() throws Exception {
5963
// A cached table is weighed once, but the first snapshot lookup or scan builds its store (RowType copies
60-
// of the schema, a copy of the options) and field lookups build every RowType's lazy index maps. The
61-
// admission weight must already cover that grown graph. MUTATION: dropping the store-growth reservation
62-
// -> every fixture's grown graph exceeds its weight -> red.
64+
// of the schema, a copy of the options, a partial-update table's merge function) and field lookups build
65+
// every RowType's lazy index maps. The admission weight must already cover that grown graph without the
66+
// FileIO allowance, which pays for the FileIO. MUTATION: dropping the store-growth reservation -> every
67+
// fixture's grown graph exceeds its weight -> red; dropping the partial-update reservation -> the wide
68+
// partial-update fixture does -> red.
6369
List<FileStoreTable> fixtures = new ArrayList<>();
6470
fixtures.add(table("append_narrow", 10, 0, 0, 0, null));
6571
fixtures.add(table("append_wide", 300, 0, 0, 0, null));
@@ -72,8 +78,10 @@ void admissionWeightCoversTheStoreAndIndexesBuiltAfterAdmission() throws Excepti
7278
fixtures.add(table("array_of_rows", 0, 0, 0, 0, new ArrayType(rowOf(20))));
7379
fixtures.add(table("map_of_rows", 0, 0, 0, 0, new MapType(DataTypes.INT(), rowOf(20))));
7480
fixtures.add(table("multiset", 0, 0, 0, 0, new MultisetType(DataTypes.STRING())));
81+
fixtures.add(mergeEngineTable("partial_update_wide", 300, "partial-update"));
82+
fixtures.add(mergeEngineTable("aggregation_wide", 300, "aggregation"));
7583
for (FileStoreTable table : fixtures) {
76-
long weight = weight(table);
84+
long weight = weight(table) - PaimonCacheSizeEstimator.estimateFileIO(table.fileIO());
7785
Assertions.assertNull(readField(table, table.getClass(), "lazyStore"),
7886
table.name() + ": estimation must not build the store");
7987
grow(table);
@@ -96,22 +104,47 @@ void storeGrowthReservationScalesWithTheSchema() throws Exception {
96104
}
97105

98106
@Test
99-
void restDataTokenFileIoIsChargedToItsTable() throws Exception {
100-
// RESTCatalog with data tokens gives each table its own RESTTokenFileIO, which holds the vended token
101-
// fetched on first data access. Its underlying FileIO lives in Paimon's process-wide cache, so only the
102-
// RESTTokenFileIO and its token belong to the table. MUTATION: treating it as shared like other FileIOs
103-
// -> both tables weigh the same -> red.
107+
void everyTableCarriesAnAllowanceForItsFileIo() throws Exception {
108+
// A REST catalog gives each table its own RESTTokenFileIO or ResolvingFileIO and a Hive catalog a FileIO
109+
// per table location; all of them grow after admission, and none can be told apart from a FileIO the
110+
// catalog shares. So a table's weight carries the same allowance whatever its FileIO is. MUTATION:
111+
// charging only RESTTokenFileIO -> the other tables weigh a whole allowance less -> red.
112+
FileStoreTable local = table("file_io", 10, 0, 0, 0, null);
113+
CatalogContext context = CatalogContext.create(new Options());
114+
ResolvingFileIO resolving = new ResolvingFileIO();
115+
resolving.configure(context);
116+
HadoopFileIO hadoop = new HadoopFileIO(local.location());
117+
hadoop.configure(context);
118+
List<FileIO> fileIOs = Arrays.asList(local.fileIO(), resolving, hadoop,
119+
new RESTTokenFileIO(context, null, TABLE, local.location()));
120+
long withoutFileIO = weight(local) - PaimonCacheSizeEstimator.estimateFileIO(local.fileIO());
121+
for (FileIO fileIO : fileIOs) {
122+
FileStoreTable table = FileStoreTableFactory.create(
123+
fileIO, local.location(), local.schema(), CatalogEnvironment.empty());
124+
long allowance = PaimonCacheSizeEstimator.estimateFileIO(fileIO);
125+
Assertions.assertTrue(allowance >= 16L * 1024L);
126+
Assertions.assertEquals(withoutFileIO + allowance, weight(table),
127+
fileIO.getClass().getSimpleName() + " must be charged to its table");
128+
}
129+
130+
// A ResolvingFileIO builds its backend on the first data access, after admission.
131+
resolving.exists(local.location());
132+
long resolved = EstimatorCalibrationAssertions.graphSize(resolving)
133+
- EstimatorCalibrationAssertions.graphSize(context);
134+
Assertions.assertTrue(PaimonCacheSizeEstimator.estimateFileIO(resolving) >= resolved,
135+
"the allowance does not cover a ResolvingFileIO with its backend: " + resolved);
136+
}
137+
138+
@Test
139+
void fileIoAllowanceCoversARestDataTokenAndItsRefresh() throws Exception {
140+
// The token arrives after admission and a refresh replaces it, so the one allowance must cover the token
141+
// the first data access fetches as well as a later, larger one. The 1,200-character security token is the
142+
// usual STS size; the 8,192-character one stands for a credential several times larger.
143+
// MUTATION: a 4 KB allowance -> the refreshed token outgrows it -> red.
104144
FileStoreTable local = table("rest_token", 10, 0, 0, 0, null);
105145
CatalogContext context = CatalogContext.create(new Options());
106146
RESTTokenFileIO fileIO = new RESTTokenFileIO(context, null, TABLE, local.location());
107-
FileStoreTable rest = FileStoreTableFactory.create(
108-
fileIO, local.location(), local.schema(), CatalogEnvironment.empty());
109-
110-
long reserved = weight(rest) - weight(local);
111-
// The token arrives after admission and a refresh replaces it, so the one reserve must cover the token
112-
// the first data access fetches as well as a later, larger one. The 1,200-character security token is
113-
// the usual STS size; the 8,192-character one stands for a credential several times larger.
114-
// MUTATION: a 4 KB reserve -> the refreshed token outgrows it -> red.
147+
long allowance = PaimonCacheSizeEstimator.estimateFileIO(fileIO);
115148
for (int securityTokenChars : new int[] {1200, 8192}) {
116149
Map<String, String> token = new HashMap<>();
117150
token.put("fs.oss.accessKeyId", "STS." + "a".repeat(28));
@@ -121,12 +154,39 @@ void restDataTokenFileIoIsChargedToItsTable() throws Exception {
121154
setToken(fileIO, new RESTToken(token, System.currentTimeMillis() + 3_600_000L));
122155
long owned = EstimatorCalibrationAssertions.graphSize(fileIO)
123156
- EstimatorCalibrationAssertions.graphSize(context);
124-
Assertions.assertTrue(reserved >= owned, "reserved " + reserved
157+
Assertions.assertTrue(allowance >= owned, "allowance " + allowance
125158
+ " does not cover the table-owned graph with a " + securityTokenChars + "-character token "
126159
+ owned);
127160
}
128161
}
129162

163+
/**
164+
* A primary-key table with {@code fieldCount} INT value fields under {@code mergeEngine}: a partial-update
165+
* table puts them all in one sequence group with a default aggregate function, an aggregation table sums them.
166+
*/
167+
private FileStoreTable mergeEngineTable(String name, int fieldCount, String mergeEngine) throws Exception {
168+
Schema.Builder builder = Schema.newBuilder()
169+
.column("part", DataTypes.INT())
170+
.column("key_0", DataTypes.INT().notNull())
171+
.column("seq_0", DataTypes.INT());
172+
List<String> grouped = new ArrayList<>();
173+
for (int i = 0; i < fieldCount; i++) {
174+
builder.column("field_" + i, DataTypes.INT());
175+
grouped.add("field_" + i);
176+
}
177+
builder.primaryKey("key_0", "part").partitionKeys("part").option("merge-engine", mergeEngine);
178+
if ("partial-update".equals(mergeEngine)) {
179+
builder.option("fields.seq_0.sequence-group", String.join(",", grouped));
180+
builder.option("fields.default-aggregate-function", "last_non_null_value");
181+
} else if ("aggregation".equals(mergeEngine)) {
182+
builder.option("fields.default-aggregate-function", "sum");
183+
}
184+
LocalFileIO fileIO = LocalFileIO.create();
185+
Path path = new Path(warehouse.resolve(name).toUri());
186+
new SchemaManager(fileIO, path).createTable(builder.build());
187+
return FileStoreTableFactory.create(fileIO, path);
188+
}
189+
130190
private static void setToken(RESTTokenFileIO fileIO, RESTToken token) throws ReflectiveOperationException {
131191
Field tokenField = RESTTokenFileIO.class.getDeclaredField("token");
132192
tokenField.setAccessible(true);

0 commit comments

Comments
 (0)