You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ce25717
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](fe) Charge every Paimon table for its FileIO and reserve partial-update growth
### What problem does this PR solve?
Issue Number: None
Related PR: #66717, #67726
Problem Summary: The Paimon table cache estimator charged a FileIO only when it was a
RESTTokenFileIO, but other catalogs give each table its own FileIO as well. A REST catalog
without data tokens creates a ResolvingFileIO per table (`RESTCatalog.fileIOFromOptions`), and a
Hive catalog resolves one per table location (`HiveCatalog.fileIO(Path)`). Both build their
backend maps after the table was weighed, so a weight-limited cache of such tables holds more
heap than its limit allows. Paimon offers no IO-free way to tell an owned FileIO from one the
catalog shares, so every table now carries the fixed 16 KB FileIO allowance, as on branch-4.x.
Charging a shared FileIO to each table only makes the estimate conservative.
A partial-update table also builds a merge function factory with its store, which keeps
sequence-group and aggregator maps for every field. On Paimon 1.4.2 a 300-field partial-update
table grows about 460 bytes per field once a scan has planned, about 150 more than the other
merge engines, while the store reserve allows 320. Reserve another 256 bytes per field for
partial-update tables.
### Release note
Under an external metadata cache weight limit, every cached Paimon table also counts a fixed
16 KB for its FileIO, and partial-update tables count more per field, so fewer Paimon tables fit
in the same limit.
### Check List (For Author)
- Test: Unit Test
- PaimonCacheSizeEstimatorTest: tables on LocalFileIO, ResolvingFileIO, HadoopFileIO and
RESTTokenFileIO carry the same FileIO allowance, which covers a ResolvingFileIO with its
backend and an 8,192-character REST token; the grown-graph check now leaves the FileIO
allowance out and includes 300-field partial-update and aggregation tables. Charging only
RESTTokenFileIO, dropping the partial-update reserve, or a 4 KB allowance each make it fail.
All 679 fe-connector-paimon tests pass locally.
- Behavior changed: Yes (see Release note)
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCacheSizeEstimator.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCacheSizeEstimatorTest.java
+78-18Lines changed: 78 additions & 18 deletions
Original file line number
Diff line number
Diff line change
@@ -21,7 +21,10 @@
21
21
22
22
importorg.apache.paimon.catalog.CatalogContext;
23
23
importorg.apache.paimon.catalog.Identifier;
24
+
importorg.apache.paimon.fs.FileIO;
24
25
importorg.apache.paimon.fs.Path;
26
+
importorg.apache.paimon.fs.ResolvingFileIO;
27
+
importorg.apache.paimon.fs.hadoop.HadoopFileIO;
25
28
importorg.apache.paimon.fs.local.LocalFileIO;
26
29
importorg.apache.paimon.options.Options;
27
30
importorg.apache.paimon.rest.RESTToken;
@@ -44,6 +47,7 @@
44
47
45
48
importjava.lang.reflect.Field;
46
49
importjava.util.ArrayList;
50
+
importjava.util.Arrays;
47
51
importjava.util.HashMap;
48
52
importjava.util.List;
49
53
importjava.util.Map;
@@ -57,9 +61,11 @@ class PaimonCacheSizeEstimatorTest {
0 commit comments