You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f6e4416
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](paimon) Declare JNI heap for query-auth splits and for the files system wrappers read
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Two gaps in what paimon's JNI splits declare with enable_jni_heap_admission set:
1. A table whose catalog applies query-auth rules (query-auth.enabled with a row filter or a column mask)
has every split of its plan wrapped in a QueryAuthSplit (AbstractDataTableScan#plan ->
TableQueryAuthResult#convertPlan), and a fallback-branch table puts each of those in its FallbackSplit
in turn. A QueryAuthSplit is not a DataSplit, so the provider reads it through JNI, where the reader
merges the files inside as it would unwrapped, but the estimate saw neither a DataSplit nor an
IncrementalSplit and declared 0: an uncompacted primary-key table read under such rules opened every
reader at once, as with the variable off. The estimate now looks at a split through paimon's own
Splits#underlying, which takes off QueryAuthSplit and FallbackSplit (in either order) and stops at a
DataSplit; what is serialized to BE stays the planned split. Picking the branch needs no change: a
fallback-branch table always hands out its FallbackSplit outermost.
2. $audit_log and $binlog (and $row_tracking) read the files of the table they wrap and show its current
options, but only a file-store table and $ro were resolved to their physical table, so these wrappers
laid every file out by today's options: after file.block-size was lowered, a file written before it
was declared by the new, smaller row groups. Their files are now declared by the schema versions of
the pinned source the timestamp-history checks already resolve them to (resolveTimestampHistoryTable,
as the binary-partition check does), split by split in the branch that planned it.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- PaimonJniHeapEstimateTest: a merge split wrapped in a QueryAuthSplit, also inside a FallbackSplit,
and a diff split wrapped in one, declare what they declare unwrapped
- PaimonScanPlanProviderTest: the table altered after its files were written, read through $audit_log
and $binlog with the variable on, declares each file by the version it was written under
- Both new checks fail with the old code: the wrapped merge split declared 0 instead of 150 MB, and
$audit_log 8388612 bytes (today's 1-byte row groups) instead of 10487929
- On master 5caaeb3: fe-connector-paimon 901 tests (1 skipped, PaimonLiveConnectivityTest, which
needs a live catalog) and fe-connector-fluss 296 tests pass; checkstyle reports no violation
- Behavior changed: No (only statements that set enable_jni_heap_admission declare; such a statement now
declares the splits of query-auth tables, and the files of $audit_log, $binlog and $row_tracking by
their own schema versions)
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonJniHeapEstimate.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonJniHeapEstimateTest.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java
+31-20Lines changed: 31 additions & 20 deletions
Original file line number
Diff line number
Diff line change
@@ -1354,7 +1354,8 @@ public void primaryKeyLimitKeepsAllRowsForUnsafeSplitAccounting(@TempDir Path wa
1354
1354
/**
1355
1355
* With enable_jni_heap_admission a JNI split declares each of its files by the schema version it was
1356
1356
* written under: files written before an ALTER TABLE lowered the block size and added a column are
1357
-
* declared by the row groups and the columns they were written with, not by today's options.
1357
+
* declared by the row groups and the columns they were written with, not by today's options. So are
1358
+
* they when read through $audit_log or $binlog, which show today's options of the table they wrap.
0 commit comments