You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2f5f467
Browse filesBrowse the repository at this point in the historyBrowse files
fix: FIX-NATIVE-SUBSPLIT — sub-split large native ORC/Parquet paimon files for read parallelism (M-3)
Root cause: after cutover, a large native (ORC/Parquet) paimon data file gets
ONE scanner — no intra-file parallelism. The connector's native arm emitted
exactly one PaimonScanRange per RawFile (start=0, length=file.length()). Legacy
PaimonScanNode:434-465 sub-splits each large file via determineTargetFileSplitSize
+ fileSplitter.splitFile. Result is correct (BE reads the whole file either way);
only read parallelism regresses.
Recon (wf_ad764bf6-1c9) confirmed: it is a real gap (ORC/Parquet are
PLAIN/splittable, legacy does sub-split); DV x sub-split is SAFE (paimon
deletion-vector rowids are GLOBAL file row positions, BE native readers report
global positions even within a partial byte range, _kv_cache shares the DV bitmap
across sub-splits keyed by path+offset, iceberg uses the identical machinery on
routinely-split files); and it is pure-connector (the splitter math + 5 session
vars re-stated with plain longs — the connector cannot import fe-core
FileSplitter/SessionVariable).
Solution (pure connector, zero SPI, zero fe-core; D-055):
- Two pure statics: computeFileSplitOffsets(fileLength, targetSplitSize) ports
FileSplitter.splitFile's specified-size branch byte-for-byte incl. the >1.1D
tail guard (the last range absorbs a remainder up to 1.1x instead of a tiny
tail split); determineTargetSplitSize(...) ports determineTargetFileSplitSize +
applyMaxFileSplitNumLimit (the isBatchMode->0 branch omitted — paimon is never
batch).
- sessionLong + lazy resolveTargetSplitSize read the 5 file-split session vars via
the VariableMgr.toMap channel (like isCppReaderEnabled) and sum native-eligible
file sizes once per scan.
- Native arm: emit one range per [start,length) sub-range via buildNativeRanges,
attaching the SAME unmodified per-RawFile DeletionFile to EVERY sub-range (DV is
global-row-position indexed; no offset re-basing). buildNativeRange gains
(start, length); fileSize stays the whole file length.
- Under COUNT(*) pushdown a native split that is not count-eligible (no precomputed
merged count, e.g. a DV with null cardinality) is kept WHOLE (target size 0 ->
one whole-file range), mirroring legacy splittable=!applyCountPushdown.
The split-weight/target-size scheduling nicety is not ported (pre-existing native
path already omitted it; perf/scheduling-only, not correctness) -> DV-033.
Tests: connector PaimonScanPlanProviderTest +6 — computeFileSplitOffsets math
(250MB/64MB->4 with 58MB tail, exact-multiple, small-file-whole, empty, target<=0);
determineTargetSplitSize heuristic (file_split_size override, 32MB<->64MB threshold,
max_file_split_num floor); end-to-end append-only fixture (tiny file_split_size ->
>=2 contiguous sub-ranges tiling [0,fileLength); default -> 1 range); DV on every
sub-range; whole-file under count pushdown. Updated the 3 existing buildNativeRange
call sites to the new signature. Connector 258/0/0 (1 CI-gated live skip),
checkstyle 0, import-gate clean. Fail-before verified: neuter computeFileSplitOffsets
-> the 3 splitting tests red; attach DV only to the first sub-range -> the DV test
red. Real BE multi-range + DV read = CI-gated live-e2e (legacy paimon regression
covers the BE contract; no BE change).
Adversarially reviewed (workflow wf_4ac7479d-39d): 2 confirmed and fixed (the
count-pushdown sub-split parity gap + false comment; the missing DV-on-every-sub-range
test), 2 refuted.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java
+159-6Lines changed: 159 additions & 6 deletions
Original file line number
Diff line number
Diff line change
@@ -140,6 +140,21 @@ public class PaimonScanPlanProvider implements ConnectorScanPlanProvider {
140
140
// bypassing the native ORC/Parquet readers to dodge native-reader bugs. Default false (legacy default).
0 commit comments