Skip to content

Commit f6e4416

Browse files
morningmanclaude
andcommitted
[fix](paimon) Declare JNI heap for query-auth splits and for the files system wrappers read
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Two gaps in what paimon's JNI splits declare with enable_jni_heap_admission set: 1. A table whose catalog applies query-auth rules (query-auth.enabled with a row filter or a column mask) has every split of its plan wrapped in a QueryAuthSplit (AbstractDataTableScan#plan -> TableQueryAuthResult#convertPlan), and a fallback-branch table puts each of those in its FallbackSplit in turn. A QueryAuthSplit is not a DataSplit, so the provider reads it through JNI, where the reader merges the files inside as it would unwrapped, but the estimate saw neither a DataSplit nor an IncrementalSplit and declared 0: an uncompacted primary-key table read under such rules opened every reader at once, as with the variable off. The estimate now looks at a split through paimon's own Splits#underlying, which takes off QueryAuthSplit and FallbackSplit (in either order) and stops at a DataSplit; what is serialized to BE stays the planned split. Picking the branch needs no change: a fallback-branch table always hands out its FallbackSplit outermost. 2. $audit_log and $binlog (and $row_tracking) read the files of the table they wrap and show its current options, but only a file-store table and $ro were resolved to their physical table, so these wrappers laid every file out by today's options: after file.block-size was lowered, a file written before it was declared by the new, smaller row groups. Their files are now declared by the schema versions of the pinned source the timestamp-history checks already resolve them to (resolveTimestampHistoryTable, as the binary-partition check does), split by split in the branch that planned it. ### Release note None ### Check List (For Author) - Test: Unit Test - PaimonJniHeapEstimateTest: a merge split wrapped in a QueryAuthSplit, also inside a FallbackSplit, and a diff split wrapped in one, declare what they declare unwrapped - PaimonScanPlanProviderTest: the table altered after its files were written, read through $audit_log and $binlog with the variable on, declares each file by the version it was written under - Both new checks fail with the old code: the wrapped merge split declared 0 instead of 150 MB, and $audit_log 8388612 bytes (today's 1-byte row groups) instead of 10487929 - On master 5caaeb3: fe-connector-paimon 901 tests (1 skipped, PaimonLiveConnectivityTest, which needs a live catalog) and fe-connector-fluss 296 tests pass; checkstyle reports no violation - Behavior changed: No (only statements that set enable_jni_heap_admission declare; such a statement now declares the splits of query-auth tables, and the files of $audit_log, $binlog and $row_tracking by their own schema versions) - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent dbe6a81 commit f6e4416

4 files changed

Lines changed: 66 additions & 31 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonJniHeapEstimate.java‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@
2121
import org.apache.paimon.io.DataFileMeta;
2222
import org.apache.paimon.options.MemorySize;
2323
import org.apache.paimon.schema.TableSchema;
24-
import org.apache.paimon.table.FallbackReadFileStoreTable;
2524
import org.apache.paimon.table.FileStoreTable;
2625
import org.apache.paimon.table.Table;
2726
import org.apache.paimon.table.source.DataSplit;
2827
import org.apache.paimon.table.source.IncrementalSplit;
2928
import org.apache.paimon.table.source.Split;
29+
import org.apache.paimon.table.source.Splits;
3030
import org.apache.paimon.types.ArrayType;
3131
import org.apache.paimon.types.DataType;
3232
import org.apache.paimon.types.MapType;
@@ -64,6 +64,10 @@
6464
* files one after another. The other splits that are not DataSplits, those of the metadata system tables
6565
* and of format tables, hold little and declare nothing.
6666
*
67+
* <p>A split is looked at through the wrappers paimon plans it in, which its reader reads through: the
68+
* {@code QueryAuthSplit} every split of a table with query-auth rules (a catalog's row filter or column
69+
* mask) comes in, and the {@code FallbackSplit} of a fallback-branch table, which wraps that in turn.
70+
*
6771
* <p>A file's row groups, dictionary pages and columns are those of the schema version it was written under
6872
* ({@link DataFileMeta#schemaId()}): paimon's writers take the row group size and the largest parquet
6973
* dictionary page ({@code parquet.dictionary.page.size}) from the table's options, and an ALTER TABLE that
@@ -124,14 +128,12 @@ static PaimonJniHeapEstimate of(Table table) {
124128
}
125129

126130
long bytesOf(Split split) {
127-
if (split instanceof DataSplit) {
128-
DataSplit dataSplit = (DataSplit) split;
131+
// A fallback-branch table's FallbackDataSplit is a DataSplit itself.
132+
Split read = Splits.underlying(split);
133+
if (read instanceof DataSplit) {
134+
DataSplit dataSplit = (DataSplit) read;
129135
return readsFileByFile(dataSplit) ? largest(dataSplit.dataFiles()) : total(dataSplit.dataFiles());
130136
}
131-
// A fallback-branch table wraps the splits that are not DataSplits; its FallbackDataSplit is one.
132-
Split read = split instanceof FallbackReadFileStoreTable.FallbackSplit
133-
? ((FallbackReadFileStoreTable.FallbackSplit) split).wrapped()
134-
: split;
135137
if (!(read instanceof IncrementalSplit)) {
136138
return 0;
137139
}

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2244,12 +2244,13 @@ && containsVariant(field.type()))) {
22442244
* size set by the schema version it was written under, which an ALTER TABLE since leaves as it is, so
22452245
* the estimate reads the versions its files were written under, of the branch that planned its split: a
22462246
* scan.fallback-branch or scan.primary-branch table plans the splits of two branches, each numbering its
2247-
* schema versions on its own. $ro reads its base table's files, under that table's schema ids, as
2248-
* {@link #physicalVariantSchemaIds} resolves it; the other system tables are estimated by the options
2249-
* they show.
2247+
* schema versions on its own. $ro, $audit_log, $binlog and $row_tracking read their base table's files,
2248+
* under that table's schema ids, while showing its current options; the metadata system tables are
2249+
* estimated by the options they show.
22502250
*/
22512251
private ToLongFunction<Split> jniHeapBytes(Table table, PaimonTableHandle handle) {
2252-
Table physicalSchemaTable = resolveSchemaDictTable(table, handle);
2252+
// The base table of a wrapper, resolved as the timestamp-history checks resolve it.
2253+
Table physicalSchemaTable = withSchemaHistoryAuthentication(() -> resolveTimestampHistoryTable(table, handle));
22532254
if (!(physicalSchemaTable instanceof FileStoreTable)) {
22542255
return PaimonJniHeapEstimate.of(table)::bytesOf;
22552256
}

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonJniHeapEstimateTest.java‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
import org.apache.paimon.catalog.Catalog;
2222
import org.apache.paimon.catalog.FileSystemCatalog;
2323
import org.apache.paimon.catalog.Identifier;
24+
import org.apache.paimon.catalog.TableQueryAuthResult;
2425
import org.apache.paimon.data.BinaryRow;
2526
import org.apache.paimon.fs.local.LocalFileIO;
2627
import org.apache.paimon.io.DataFileMeta;
@@ -32,6 +33,7 @@
3233
import org.apache.paimon.table.Table;
3334
import org.apache.paimon.table.source.DataSplit;
3435
import org.apache.paimon.table.source.IncrementalSplit;
36+
import org.apache.paimon.table.source.QueryAuthSplit;
3537
import org.apache.paimon.table.source.Split;
3638
import org.apache.paimon.types.DataTypes;
3739
import org.junit.jupiter.api.Assertions;
@@ -114,6 +116,25 @@ public void mergedSplitHoldsARowGroupOfEveryFile() {
114116
Assertions.assertEquals((130 + 4) * MB + (12 + 4) * MB, estimate.bytesOf(split));
115117
}
116118

119+
/**
120+
* A table with query-auth rules has every split planned in a QueryAuthSplit, which a fallback-branch table
121+
* wraps in its FallbackSplit; the reader reads the split inside, merging it as it would unwrapped.
122+
*/
123+
@Test
124+
public void splitPlannedForQueryAuthHoldsWhatTheSplitInsideHolds() {
125+
TableQueryAuthResult rules = new TableQueryAuthResult(Collections.emptyList(), Collections.emptyMap());
126+
DataSplit merged = split(false, file("data-0.parquet", 130 * MB), file("data-1.parquet", 12 * MB));
127+
long expected = (130 + 4) * MB + (12 + 4) * MB;
128+
Assertions.assertEquals(expected, estimate.bytesOf(new QueryAuthSplit(merged, rules)));
129+
Assertions.assertEquals(expected, estimate.bytesOf(
130+
FallbackReadFileStoreTable.toFallbackSplit(new QueryAuthSplit(merged, rules), true)));
131+
132+
IncrementalSplit diff = new IncrementalSplit(2L, BinaryRow.EMPTY_ROW, 0, 1,
133+
Collections.singletonList(file("data-2.parquet", 40 * MB)), null,
134+
Collections.singletonList(file("data-3.parquet", 300 * MB)), null, false);
135+
Assertions.assertEquals((40 + 4) * MB + (256 + 4) * MB, estimate.bytesOf(new QueryAuthSplit(diff, rules)));
136+
}
137+
117138
@Test
118139
public void splitThatNeedsNoMergingHoldsItsLargestFile() {
119140
DataSplit split = split(true, file("data-0.parquet", 130 * MB), file("data-1.parquet", 12 * MB));

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java‎

Lines changed: 31 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1354,7 +1354,8 @@ public void primaryKeyLimitKeepsAllRowsForUnsafeSplitAccounting(@TempDir Path wa
13541354
/**
13551355
* With enable_jni_heap_admission a JNI split declares each of its files by the schema version it was
13561356
* written under: files written before an ALTER TABLE lowered the block size and added a column are
1357-
* declared by the row groups and the columns they were written with, not by today's options.
1357+
* declared by the row groups and the columns they were written with, not by today's options. So are
1358+
* they when read through $audit_log or $binlog, which show today's options of the table they wrap.
13581359
*/
13591360
@Test
13601361
public void jniSplitDeclaresFilesWrittenBeforeAnAlterByTheirOwnSchema(@TempDir Path warehouse)
@@ -1385,29 +1386,39 @@ public void jniSplitDeclaresFilesWrittenBeforeAnAlterByTheirOwnSchema(@TempDir P
13851386
SchemaChange.setOption(CoreOptions.FILE_BLOCK_SIZE.key(), "1 b"),
13861387
SchemaChange.addColumn("x", DataTypes.INT())), false);
13871388

1389+
FileStoreTable altered = (FileStoreTable) catalog.getTable(id);
13881390
RecordingPaimonCatalogOps ops = new RecordingPaimonCatalogOps();
1389-
ops.table = catalog.getTable(id);
1391+
ops.table = altered;
13901392
PaimonScanPlanProvider provider = new PaimonScanPlanProvider(
13911393
PaimonCatalogProperties.of(Collections.emptyMap()), ops);
1392-
PaimonTableHandle handle = new PaimonTableHandle(
1393-
"db", "altered", Collections.emptyList(), Collections.emptyList());
1394-
List<ConnectorScanRange> ranges = provider.planScan(
1395-
sessionWithProps(Collections.singletonMap("enable_jni_heap_admission", "true")),
1396-
ConnectorScanRequest.builder(handle, Collections.emptyList()).build());
1397-
1398-
Assertions.assertEquals(1, ranges.size());
1399-
DataSplit split = (DataSplit) deserializeJniSplits(ranges).get(0);
1400-
Assertions.assertFalse(split.rawConvertible(), "the two files must be merged");
1401-
Assertions.assertEquals(2, split.dataFiles().size());
1402-
// Under version 0 a file has 128 MB row groups - all of a small file - and 5 columns, (id, val)
1403-
// keyed by id; version 1 would give it 2 bytes of 1-byte row groups and 6 columns.
1404-
long expected = 0;
1405-
for (DataFileMeta file : split.dataFiles()) {
1406-
Assertions.assertEquals(0L, file.schemaId());
1407-
expected += file.fileSize() + 5 * PaimonJniHeapEstimate.DEFAULT_DICTIONARY_PAGE_BYTES;
1394+
// $audit_log and $binlog read the same files, while showing the table's current options.
1395+
for (String kind : Arrays.asList("", "audit_log", "binlog")) {
1396+
PaimonTableHandle handle = kind.isEmpty()
1397+
? new PaimonTableHandle("db", "altered", Collections.emptyList(), Collections.emptyList())
1398+
: PaimonTableHandle.forSystemTable("db", "altered", kind, true);
1399+
if (!kind.isEmpty()) {
1400+
handle.setSysBaseTable(altered);
1401+
handle.setSystemTableSource(altered);
1402+
handle.setPaimonTable(org.apache.paimon.table.system.SystemTableLoader.load(kind, altered));
1403+
}
1404+
List<ConnectorScanRange> ranges = provider.planScan(
1405+
sessionWithProps(Collections.singletonMap("enable_jni_heap_admission", "true")),
1406+
ConnectorScanRequest.builder(handle, Collections.emptyList()).build());
1407+
1408+
Assertions.assertEquals(1, ranges.size(), kind);
1409+
DataSplit split = (DataSplit) deserializeJniSplits(ranges).get(0);
1410+
Assertions.assertFalse(split.rawConvertible(), "the two files must be merged");
1411+
Assertions.assertEquals(2, split.dataFiles().size());
1412+
// Under version 0 a file has 128 MB row groups - all of a small file - and 5 columns, (id, val)
1413+
// keyed by id; version 1 would give it 2 bytes of 1-byte row groups and 6 columns.
1414+
long expected = 0;
1415+
for (DataFileMeta file : split.dataFiles()) {
1416+
Assertions.assertEquals(0L, file.schemaId());
1417+
expected += file.fileSize() + 5 * PaimonJniHeapEstimate.DEFAULT_DICTIONARY_PAGE_BYTES;
1418+
}
1419+
Assertions.assertEquals(String.valueOf(expected),
1420+
ranges.get(0).getProperties().get("paimon.jni_heap_bytes"), kind);
14081421
}
1409-
Assertions.assertEquals(String.valueOf(expected),
1410-
ranges.get(0).getProperties().get("paimon.jni_heap_bytes"));
14111422
}
14121423
}
14131424

0 commit comments

Comments
 (0)