Skip to content

Commit b389299

Browse files
morningmanclaude
andcommitted
[fix](paimon) Declare each file of a JNI split by the schema version it was written under
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: With enable_jni_heap_admission set, a paimon JNI DataSplit declares the heap its reader will hold: for every file one row group (two for a file of more than one, never more than the file) plus a 1 MB dictionary page per column. PaimonJniHeapEstimate took the row group size and the columns from the table as it is today - file.block-size, else parquet.block.size or orc.stripe.size, else 128 MB or 64 MB - and applied them to every file. Paimon lets ALTER TABLE change those options, and the columns, without rewriting the files written before: after file.block-size is lowered from 128 MB to 16 MB, a 900 MB file written before still has 128 MB row groups, yet was declared at min(900, 2 x 16) = 32 MB, an eighth of the 256 MB it holds while moving from one row group to the next. Concurrent merge reads of such files passed the gate on understated declarations and could run the JVM out of heap, as they do with the variable off. Every DataFileMeta records the schema id it was written under, and that schema version holds the options and the columns its writer used. The estimate now lays out each file by its own version: the table's own version is at hand, and every other version a split's files were written under is read once per planning, the way PaimonScanPlanProvider already reads schema versions by file (physicalVariantSchemaIds), and under the FE-injected authenticator as the manifests are read. $ro resolves to its base table, whose files it reads; the other system tables keep being estimated by the options they show. Only a statement that sets the variable plans this way, and a table whose files were all written under its current version reads nothing more. What the estimate still cannot see is a row group size a write job passed as a dynamic option, which no schema version records; the class comment says so, beside the merge peaks it already named. ### Release note None ### Check List (For Author) - Test: Unit Test - PaimonJniHeapEstimateTest: a new case for a split of two files written before and after an ALTER TABLE that lowered file.block-size and added a column - each declared by its own row groups and columns, the older version read once, the table's own version never; the options case now checks both entry points, and that the table's own version is not read - PaimonScanPlanProviderTest: a new case that writes two overlapping files, lowers file.block-size to one byte, adds a column and plans the table with enable_jni_heap_admission: the JNI split declares each file by its 128 MB row groups and 5 columns. With the old PaimonJniHeapEstimate.of(table) at the call site it fails: 12582916 bytes declared, today's 2 bytes and 6 columns a file, against 10487929 - fe-connector-paimon 685 tests (1 skipped, as on master) and fe-connector-fluss 295 tests pass; checkstyle reports no violation - Behavior changed: No (only statements that set enable_jni_heap_admission declare; a file written before an ALTER TABLE changed its row group size or columns is now declared by what it holds) - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 730a221 commit b389299

4 files changed

Lines changed: 224 additions & 40 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonJniHeapEstimate.java‎

Lines changed: 73 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -20,11 +20,15 @@
2020
import org.apache.paimon.CoreOptions;
2121
import org.apache.paimon.io.DataFileMeta;
2222
import org.apache.paimon.options.MemorySize;
23+
import org.apache.paimon.schema.TableSchema;
24+
import org.apache.paimon.table.FileStoreTable;
2325
import org.apache.paimon.table.Table;
2426
import org.apache.paimon.table.source.DataSplit;
2527

28+
import java.util.HashMap;
2629
import java.util.Locale;
2730
import java.util.Map;
31+
import java.util.function.LongFunction;
2832

2933
/**
3034
* The JVM heap BE's JNI reader of a paimon {@link DataSplit} holds, declared to BE's JNI heap gate
@@ -41,6 +45,13 @@
4145
* overlap, which are read one section after another. A split that needs no merging reads its files one
4246
* after another and holds one file's share.
4347
*
48+
* <p>A file's row groups and columns are those of the schema version it was written under
49+
* ({@link DataFileMeta#schemaId()}): paimon's writers take the row group size from the table's options,
50+
* and an ALTER TABLE that changes them, or the columns, leaves the files written before it as they are.
51+
* A write job can also pass the row group size as a dynamic option, which no schema version records; a
52+
* file written so is taken by its schema version's options, and declared below what it holds when the
53+
* job's row groups were larger.
54+
*
4455
* <p>That steady hold is what is declared, not everything a merge ever holds: while it decodes, its column
4556
* batches add brief peaks on top, measured at about three times the declaration for a split of a 126 MB
4657
* file read on its own. The peaks are left to the heap outside the budget - the other half of it with
@@ -61,51 +72,84 @@ final class PaimonJniHeapEstimate {
6172
// The largest dictionary page paimon's parquet writer keeps for a column.
6273
static final long DICTIONARY_BYTES_PER_COLUMN = 1024L * 1024;
6374

64-
private final long parquetRowGroupBytes;
65-
private final long orcStripeBytes;
66-
private final long dictionaryBytesPerFile;
75+
// How the files written under a schema version are laid out, by its id, each worked out once.
76+
private final LongFunction<FileLayout> layoutOfSchema;
77+
private final Map<Long, FileLayout> layouts = new HashMap<>();
78+
79+
PaimonJniHeapEstimate(LongFunction<FileLayout> layoutOfSchema) {
80+
this.layoutOfSchema = layoutOfSchema;
81+
}
6782

68-
PaimonJniHeapEstimate(long parquetRowGroupBytes, long orcStripeBytes, int columnsPerFile) {
69-
this.parquetRowGroupBytes = parquetRowGroupBytes;
70-
this.orcStripeBytes = orcStripeBytes;
71-
this.dictionaryBytesPerFile = columnsPerFile * DICTIONARY_BYTES_PER_COLUMN;
83+
/**
84+
* For the splits of {@code table}, each file laid out by the schema version it was written under:
85+
* the table's own is at hand, and {@code schemaAt} reads the others by id.
86+
*/
87+
static PaimonJniHeapEstimate of(FileStoreTable table, LongFunction<TableSchema> schemaAt) {
88+
TableSchema own = table.schema();
89+
return new PaimonJniHeapEstimate(
90+
schemaId -> FileLayout.of(schemaId == own.id() ? own : schemaAt.apply(schemaId)));
7291
}
7392

93+
/** For a table whose schema versions are not at hand: every file laid out by the options it shows. */
7494
static PaimonJniHeapEstimate of(Table table) {
75-
Map<String, String> options = table.options();
76-
// The precedence paimon's writers apply: file.block-size for every format, else the format's
77-
// own option, else its default.
78-
String blockSize = options.get(CoreOptions.FILE_BLOCK_SIZE.key());
79-
long parquet = bytes(blockSize != null ? blockSize : options.get("parquet.block.size"),
80-
DEFAULT_PARQUET_ROW_GROUP_BYTES);
81-
long orc = bytes(blockSize != null ? blockSize : options.get("orc.stripe.size"),
82-
DEFAULT_ORC_STRIPE_BYTES);
83-
// A primary-key table's data file stores the keys a second time, as _KEY_ columns, beside the
84-
// sequence number and the row kind.
85-
int keys = table.primaryKeys().size();
86-
int columns = table.rowType().getFieldCount() + (keys > 0 ? keys + 2 : 0);
87-
return new PaimonJniHeapEstimate(parquet, orc, columns);
95+
FileLayout layout = FileLayout.of(table.options(), table.rowType().getFieldCount(),
96+
table.primaryKeys().size());
97+
return new PaimonJniHeapEstimate(schemaId -> layout);
8898
}
8999

90100
long bytesOf(DataSplit split) {
91101
long total = 0;
92102
long largest = 0;
93103
for (DataFileMeta file : split.dataFiles()) {
94-
long share = rowGroupsHeld(file) + dictionaryBytesPerFile;
104+
long share = layouts.computeIfAbsent(file.schemaId(), layoutOfSchema::apply).bytesHeld(file);
95105
total += share;
96106
largest = Math.max(largest, share);
97107
}
98108
return split.rawConvertible() ? largest : total;
99109
}
100110

101-
private long rowGroupsHeld(DataFileMeta file) {
102-
long rowGroup = file.fileName().toLowerCase(Locale.ROOT).endsWith(".orc")
103-
? orcStripeBytes
104-
: parquetRowGroupBytes;
105-
return file.fileSize() <= rowGroup ? file.fileSize() : Math.min(file.fileSize(), 2 * rowGroup);
106-
}
111+
/** The row group size and the columns of the files written under one schema version. */
112+
static final class FileLayout {
113+
private final long parquetRowGroupBytes;
114+
private final long orcStripeBytes;
115+
private final long dictionaryBytes;
107116

108-
private static long bytes(String value, long defaultBytes) {
109-
return value == null ? defaultBytes : MemorySize.parse(value).getBytes();
117+
FileLayout(long parquetRowGroupBytes, long orcStripeBytes, int columns) {
118+
this.parquetRowGroupBytes = parquetRowGroupBytes;
119+
this.orcStripeBytes = orcStripeBytes;
120+
this.dictionaryBytes = columns * DICTIONARY_BYTES_PER_COLUMN;
121+
}
122+
123+
static FileLayout of(TableSchema schema) {
124+
return of(schema.options(), schema.fields().size(), schema.primaryKeys().size());
125+
}
126+
127+
static FileLayout of(Map<String, String> options, int fields, int primaryKeys) {
128+
// The precedence paimon's writers apply: file.block-size for every format, else the format's
129+
// own option, else its default.
130+
String blockSize = options.get(CoreOptions.FILE_BLOCK_SIZE.key());
131+
long parquet = bytes(blockSize != null ? blockSize : options.get("parquet.block.size"),
132+
DEFAULT_PARQUET_ROW_GROUP_BYTES);
133+
long orc = bytes(blockSize != null ? blockSize : options.get("orc.stripe.size"),
134+
DEFAULT_ORC_STRIPE_BYTES);
135+
// A primary-key table's data file stores the keys a second time, as _KEY_ columns, beside the
136+
// sequence number and the row kind.
137+
return new FileLayout(parquet, orc, fields + (primaryKeys > 0 ? primaryKeys + 2 : 0));
138+
}
139+
140+
// A row group - two for a file that has more than one - and a dictionary page for every column.
141+
long bytesHeld(DataFileMeta file) {
142+
long rowGroup = file.fileName().toLowerCase(Locale.ROOT).endsWith(".orc")
143+
? orcStripeBytes
144+
: parquetRowGroupBytes;
145+
long rowGroups = file.fileSize() <= rowGroup
146+
? file.fileSize()
147+
: Math.min(file.fileSize(), 2 * rowGroup);
148+
return rowGroups + dictionaryBytes;
149+
}
150+
151+
private static long bytes(String value, long defaultBytes) {
152+
return value == null ? defaultBytes : MemorySize.parse(value).getBytes();
153+
}
110154
}
111155
}

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -919,7 +919,7 @@ && hasTrustworthyLimitAccounting(table)
919919
long weightDenominator = resolveSplitWeightDenominator(session);
920920
// Only a statement that asked for JNI heap admission has its JNI DataSplits declare their heap.
921921
PaimonJniHeapEstimate jniHeapEstimate =
922-
isJniHeapAdmissionEnabled(session) ? PaimonJniHeapEstimate.of(table) : null;
922+
isJniHeapAdmissionEnabled(session) ? jniHeapEstimate(table, paimonHandle) : null;
923923

924924
// Non-DataSplit → always JNI
925925
for (Split split : nonDataSplits) {
@@ -1916,6 +1916,37 @@ && containsVariant(field.type()))) {
19161916
return physicalVariantSchemaIds;
19171917
}
19181918

1919+
/**
1920+
* What the JNI DataSplits of {@code table} declare (PaimonJniHeapEstimate). A file holds row groups of
1921+
* the size set by the schema version it was written under, which an ALTER TABLE since leaves as it is,
1922+
* so the estimate reads the versions its files were written under. $ro reads its base table's files,
1923+
* under that table's schema ids, as {@link #physicalVariantSchemaIds} resolves it; the other system
1924+
* tables are estimated by the options they show.
1925+
*/
1926+
private PaimonJniHeapEstimate jniHeapEstimate(Table table, PaimonTableHandle handle) {
1927+
Table physicalSchemaTable = resolveSchemaDictTable(table, handle);
1928+
if (!(physicalSchemaTable instanceof FileStoreTable)) {
1929+
return PaimonJniHeapEstimate.of(table);
1930+
}
1931+
FileStoreTable fileStoreTable = (FileStoreTable) physicalSchemaTable;
1932+
SchemaManager schemaManager = fileStoreTable.schemaManager();
1933+
return PaimonJniHeapEstimate.of(fileStoreTable, schemaId -> readSchema(schemaManager, schemaId));
1934+
}
1935+
1936+
// A schema file, read under the FE-injected authenticator as planSplits reads the manifests.
1937+
private TableSchema readSchema(SchemaManager schemaManager, long schemaId) {
1938+
if (context == null) {
1939+
return schemaManager.schema(schemaId);
1940+
}
1941+
try {
1942+
return context.executeAuthenticated(() -> schemaManager.schema(schemaId));
1943+
} catch (RuntimeException e) {
1944+
throw e;
1945+
} catch (Exception e) {
1946+
throw new RuntimeException("Failed to read Paimon schema " + schemaId, e);
1947+
}
1948+
}
1949+
19191950
private static boolean containsVariant(DataType type) {
19201951
switch (type.getTypeRoot()) {
19211952
case VARIANT:

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonJniHeapEstimateTest.java‎

Lines changed: 59 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -17,14 +17,17 @@
1717

1818
package org.apache.doris.connector.paimon;
1919

20+
import org.apache.paimon.CoreOptions;
2021
import org.apache.paimon.catalog.Catalog;
2122
import org.apache.paimon.catalog.FileSystemCatalog;
2223
import org.apache.paimon.catalog.Identifier;
2324
import org.apache.paimon.data.BinaryRow;
2425
import org.apache.paimon.fs.local.LocalFileIO;
2526
import org.apache.paimon.io.DataFileMeta;
2627
import org.apache.paimon.schema.Schema;
28+
import org.apache.paimon.schema.SchemaChange;
2729
import org.apache.paimon.stats.SimpleStats;
30+
import org.apache.paimon.table.FileStoreTable;
2831
import org.apache.paimon.table.Table;
2932
import org.apache.paimon.table.source.DataSplit;
3033
import org.apache.paimon.types.DataTypes;
@@ -33,17 +36,20 @@
3336
import org.junit.jupiter.api.io.TempDir;
3437

3538
import java.nio.file.Path;
39+
import java.util.ArrayList;
3640
import java.util.Arrays;
3741
import java.util.Collections;
3842
import java.util.HashMap;
43+
import java.util.List;
3944
import java.util.Map;
4045

4146
public class PaimonJniHeapEstimateTest {
4247

4348
private static final long MB = 1024L * 1024;
4449

4550
// 128 MB row groups, 64 MB stripes, four columns a file (4 MB of dictionaries).
46-
private final PaimonJniHeapEstimate estimate = new PaimonJniHeapEstimate(128 * MB, 64 * MB, 4);
51+
private final PaimonJniHeapEstimate estimate =
52+
new PaimonJniHeapEstimate(schemaId -> new PaimonJniHeapEstimate.FileLayout(128 * MB, 64 * MB, 4));
4753

4854
/**
4955
* The split that ran a 2 GB heap out sixteen at a time: an uncompacted primary-key bucket whose 130 MB
@@ -76,7 +82,9 @@ public void anOrcFileHoldsItsStripe() {
7682

7783
/**
7884
* The row group comes from the table's options with the precedence paimon's writers apply, and every
79-
* file has a dictionary for each column, the keys a second time beside the sequence and row kind.
85+
* file has a dictionary for each column, the keys a second time beside the sequence and row kind. A
86+
* file written under the table's own schema version is laid out without reading any schema, and a
87+
* table whose versions are not at hand (a system table other than $ro) is laid out the same way.
8088
*/
8189
@Test
8290
public void theTableOptionsSetTheRowGroupAndTheColumns(@TempDir Path warehouse) throws Exception {
@@ -85,20 +93,57 @@ public void theTableOptionsSetTheRowGroupAndTheColumns(@TempDir Path warehouse)
8593
try (Catalog catalog = new FileSystemCatalog(LocalFileIO.create(),
8694
new org.apache.paimon.fs.Path(warehouse.toUri()))) {
8795
catalog.createDatabase("db", false);
88-
Assertions.assertEquals((100 + 6) * MB, PaimonJniHeapEstimate.of(
89-
table(catalog, "defaults", Collections.emptyMap())).bytesOf(oneBigFile));
90-
Assertions.assertEquals((64 + 6) * MB, PaimonJniHeapEstimate.of(
91-
table(catalog, "parquet_option", Collections.singletonMap("parquet.block.size",
92-
String.valueOf(32 * MB)))).bytesOf(oneBigFile));
96+
assertBothDeclare((100 + 6) * MB, table(catalog, "defaults", Collections.emptyMap()), oneBigFile);
97+
assertBothDeclare((64 + 6) * MB, table(catalog, "parquet_option",
98+
Collections.singletonMap("parquet.block.size", String.valueOf(32 * MB))), oneBigFile);
9399
// file.block-size wins over the format's own option.
94100
Map<String, String> both = new HashMap<>();
95101
both.put("file.block-size", "16 mb");
96102
both.put("parquet.block.size", String.valueOf(32 * MB));
97-
Assertions.assertEquals((32 + 6) * MB, PaimonJniHeapEstimate.of(
98-
table(catalog, "block_size", both)).bytesOf(oneBigFile));
103+
assertBothDeclare((32 + 6) * MB, table(catalog, "block_size", both), oneBigFile);
99104
}
100105
}
101106

107+
/**
108+
* An ALTER TABLE leaves the files written before it as they are, so a file is laid out by the schema
109+
* version it was written under: its row group size and its columns. The version is read once, however
110+
* many files were written under it, and the table's own version is never read.
111+
*/
112+
@Test
113+
public void eachFileIsLaidOutByTheSchemaVersionItWasWrittenUnder(@TempDir Path warehouse) throws Exception {
114+
try (Catalog catalog = new FileSystemCatalog(LocalFileIO.create(),
115+
new org.apache.paimon.fs.Path(warehouse.toUri()))) {
116+
catalog.createDatabase("db", false);
117+
// Version 0: 128 MB row groups by default, 6 columns a file.
118+
table(catalog, "altered", Collections.emptyMap());
119+
// Version 1: 16 MB row groups, and a column more: 7 columns a file.
120+
Identifier id = Identifier.create("db", "altered");
121+
catalog.alterTable(id, Arrays.asList(
122+
SchemaChange.setOption(CoreOptions.FILE_BLOCK_SIZE.key(), "16 mb"),
123+
SchemaChange.addColumn("x", DataTypes.INT())), false);
124+
FileStoreTable table = (FileStoreTable) catalog.getTable(id);
125+
Assertions.assertEquals(1L, table.schema().id());
126+
List<Long> reads = new ArrayList<>();
127+
PaimonJniHeapEstimate estimate = PaimonJniHeapEstimate.of(table, schemaId -> {
128+
reads.add(schemaId);
129+
return table.schemaManager().schema(schemaId);
130+
});
131+
132+
DataSplit merged = split(false, file("data-0.parquet", 900 * MB, 0), file("data-1.parquet", 900 * MB, 1));
133+
Assertions.assertEquals((256 + 6) * MB + (32 + 7) * MB, estimate.bytesOf(merged));
134+
Assertions.assertEquals((256 + 6) * MB,
135+
estimate.bytesOf(split(false, file("data-2.parquet", 900 * MB, 0))));
136+
Assertions.assertEquals(Collections.singletonList(0L), reads);
137+
}
138+
}
139+
140+
private static void assertBothDeclare(long expected, Table table, DataSplit split) {
141+
Assertions.assertEquals(expected, PaimonJniHeapEstimate.of((FileStoreTable) table, schemaId -> {
142+
throw new AssertionError("read schema " + schemaId + ", the table's own");
143+
}).bytesOf(split));
144+
Assertions.assertEquals(expected, PaimonJniHeapEstimate.of(table).bytesOf(split));
145+
}
146+
102147
private static Table table(Catalog catalog, String name, Map<String, String> options) throws Exception {
103148
Identifier id = Identifier.create("db", name);
104149
catalog.createTable(id, Schema.newBuilder()
@@ -113,7 +158,11 @@ private static Table table(Catalog catalog, String name, Map<String, String> opt
113158
}
114159

115160
private static DataFileMeta file(String name, long size) {
116-
return DataFileMeta.forAppend(name, size, 1000L, SimpleStats.EMPTY_STATS, 0L, 0L, 0L,
161+
return file(name, size, 0L);
162+
}
163+
164+
private static DataFileMeta file(String name, long size, long schemaId) {
165+
return DataFileMeta.forAppend(name, size, 1000L, SimpleStats.EMPTY_STATS, 0L, 0L, schemaId,
117166
Collections.emptyList(), null, null, null, null, null, null);
118167
}
119168

0 commit comments

Comments
 (0)