Repository navigation
Commit b389299
[fix](paimon) Declare each file of a JNI split by the schema version it was written under
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
With enable_jni_heap_admission set, a paimon JNI DataSplit declares the heap its reader will hold: for
every file one row group (two for a file of more than one, never more than the file) plus a 1 MB
dictionary page per column. PaimonJniHeapEstimate took the row group size and the columns from the
table as it is today - file.block-size, else parquet.block.size or orc.stripe.size, else 128 MB or
64 MB - and applied them to every file. Paimon lets ALTER TABLE change those options, and the columns,
without rewriting the files written before: after file.block-size is lowered from 128 MB to 16 MB, a
900 MB file written before still has 128 MB row groups, yet was declared at min(900, 2 x 16) = 32 MB,
an eighth of the 256 MB it holds while moving from one row group to the next. Concurrent merge reads of
such files passed the gate on understated declarations and could run the JVM out of heap, as they do
with the variable off.
Every DataFileMeta records the schema id it was written under, and that schema version holds the
options and the columns its writer used. The estimate now lays out each file by its own version: the
table's own version is at hand, and every other version a split's files were written under is read
once per planning, the way PaimonScanPlanProvider already reads schema versions by file
(physicalVariantSchemaIds), and under the FE-injected authenticator as the manifests are read. $ro
resolves to its base table, whose files it reads; the other system tables keep being estimated by the
options they show. Only a statement that sets the variable plans this way, and a table whose files were
all written under its current version reads nothing more.
What the estimate still cannot see is a row group size a write job passed as a dynamic option, which no
schema version records; the class comment says so, beside the merge peaks it already named.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- PaimonJniHeapEstimateTest: a new case for a split of two files written before and after an ALTER
TABLE that lowered file.block-size and added a column - each declared by its own row groups and
columns, the older version read once, the table's own version never; the options case now checks
both entry points, and that the table's own version is not read
- PaimonScanPlanProviderTest: a new case that writes two overlapping files, lowers file.block-size to
one byte, adds a column and plans the table with enable_jni_heap_admission: the JNI split declares
each file by its 128 MB row groups and 5 columns. With the old PaimonJniHeapEstimate.of(table) at
the call site it fails: 12582916 bytes declared, today's 2 bytes and 6 columns a file, against
10487929
- fe-connector-paimon 685 tests (1 skipped, as on master) and fe-connector-fluss 295 tests pass;
checkstyle reports no violation
- Behavior changed: No (only statements that set enable_jni_heap_admission declare; a file written
before an ALTER TABLE changed its row group size or columns is now declared by what it holds)
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent 730a221 commit b389299
4 files changed
Lines changed: 224 additions & 40 deletions
File tree
- fe/fe-connector/fe-connector-paimon/src
- main/java/org/apache/doris/connector/paimon
- test/java/org/apache/doris/connector/paimon
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
| 24 | + | |
23 | 25 | | |
24 | 26 | | |
25 | 27 | | |
| 28 | + | |
26 | 29 | | |
27 | 30 | | |
| 31 | + | |
28 | 32 | | |
29 | 33 | | |
30 | 34 | | |
| |||
41 | 45 | | |
42 | 46 | | |
43 | 47 | | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
44 | 55 | | |
45 | 56 | | |
46 | 57 | | |
| |||
61 | 72 | | |
62 | 73 | | |
63 | 74 | | |
64 | | - | |
65 | | - | |
66 | | - | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
67 | 82 | | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
72 | 91 | | |
73 | 92 | | |
| 93 | + | |
74 | 94 | | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
88 | 98 | | |
89 | 99 | | |
90 | 100 | | |
91 | 101 | | |
92 | 102 | | |
93 | 103 | | |
94 | | - | |
| 104 | + | |
95 | 105 | | |
96 | 106 | | |
97 | 107 | | |
98 | 108 | | |
99 | 109 | | |
100 | 110 | | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
107 | 116 | | |
108 | | - | |
109 | | - | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
110 | 154 | | |
111 | 155 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
919 | 919 | | |
920 | 920 | | |
921 | 921 | | |
922 | | - | |
| 922 | + | |
923 | 923 | | |
924 | 924 | | |
925 | 925 | | |
| |||
1916 | 1916 | | |
1917 | 1917 | | |
1918 | 1918 | | |
| 1919 | + | |
| 1920 | + | |
| 1921 | + | |
| 1922 | + | |
| 1923 | + | |
| 1924 | + | |
| 1925 | + | |
| 1926 | + | |
| 1927 | + | |
| 1928 | + | |
| 1929 | + | |
| 1930 | + | |
| 1931 | + | |
| 1932 | + | |
| 1933 | + | |
| 1934 | + | |
| 1935 | + | |
| 1936 | + | |
| 1937 | + | |
| 1938 | + | |
| 1939 | + | |
| 1940 | + | |
| 1941 | + | |
| 1942 | + | |
| 1943 | + | |
| 1944 | + | |
| 1945 | + | |
| 1946 | + | |
| 1947 | + | |
| 1948 | + | |
| 1949 | + | |
1919 | 1950 | | |
1920 | 1951 | | |
1921 | 1952 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| 28 | + | |
27 | 29 | | |
| 30 | + | |
28 | 31 | | |
29 | 32 | | |
30 | 33 | | |
| |||
33 | 36 | | |
34 | 37 | | |
35 | 38 | | |
| 39 | + | |
36 | 40 | | |
37 | 41 | | |
38 | 42 | | |
| 43 | + | |
39 | 44 | | |
40 | 45 | | |
41 | 46 | | |
42 | 47 | | |
43 | 48 | | |
44 | 49 | | |
45 | 50 | | |
46 | | - | |
| 51 | + | |
| 52 | + | |
47 | 53 | | |
48 | 54 | | |
49 | 55 | | |
| |||
76 | 82 | | |
77 | 83 | | |
78 | 84 | | |
79 | | - | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
80 | 88 | | |
81 | 89 | | |
82 | 90 | | |
| |||
85 | 93 | | |
86 | 94 | | |
87 | 95 | | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
93 | 99 | | |
94 | 100 | | |
95 | 101 | | |
96 | 102 | | |
97 | | - | |
98 | | - | |
| 103 | + | |
99 | 104 | | |
100 | 105 | | |
101 | 106 | | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
102 | 147 | | |
103 | 148 | | |
104 | 149 | | |
| |||
113 | 158 | | |
114 | 159 | | |
115 | 160 | | |
116 | | - | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
117 | 166 | | |
118 | 167 | | |
119 | 168 | | |
| |||
0 commit comments