Skip to content

Commit 2f5f467

Browse files
morningmanclaude
andcommitted
fix: FIX-NATIVE-SUBSPLIT — sub-split large native ORC/Parquet paimon files for read parallelism (M-3)
Root cause: after cutover, a large native (ORC/Parquet) paimon data file gets ONE scanner — no intra-file parallelism. The connector's native arm emitted exactly one PaimonScanRange per RawFile (start=0, length=file.length()). Legacy PaimonScanNode:434-465 sub-splits each large file via determineTargetFileSplitSize + fileSplitter.splitFile. Result is correct (BE reads the whole file either way); only read parallelism regresses. Recon (wf_ad764bf6-1c9) confirmed: it is a real gap (ORC/Parquet are PLAIN/splittable, legacy does sub-split); DV x sub-split is SAFE (paimon deletion-vector rowids are GLOBAL file row positions, BE native readers report global positions even within a partial byte range, _kv_cache shares the DV bitmap across sub-splits keyed by path+offset, iceberg uses the identical machinery on routinely-split files); and it is pure-connector (the splitter math + 5 session vars re-stated with plain longs — the connector cannot import fe-core FileSplitter/SessionVariable). Solution (pure connector, zero SPI, zero fe-core; D-055): - Two pure statics: computeFileSplitOffsets(fileLength, targetSplitSize) ports FileSplitter.splitFile's specified-size branch byte-for-byte incl. the >1.1D tail guard (the last range absorbs a remainder up to 1.1x instead of a tiny tail split); determineTargetSplitSize(...) ports determineTargetFileSplitSize + applyMaxFileSplitNumLimit (the isBatchMode->0 branch omitted — paimon is never batch). - sessionLong + lazy resolveTargetSplitSize read the 5 file-split session vars via the VariableMgr.toMap channel (like isCppReaderEnabled) and sum native-eligible file sizes once per scan. - Native arm: emit one range per [start,length) sub-range via buildNativeRanges, attaching the SAME unmodified per-RawFile DeletionFile to EVERY sub-range (DV is global-row-position indexed; no offset re-basing). buildNativeRange gains (start, length); fileSize stays the whole file length. - Under COUNT(*) pushdown a native split that is not count-eligible (no precomputed merged count, e.g. a DV with null cardinality) is kept WHOLE (target size 0 -> one whole-file range), mirroring legacy splittable=!applyCountPushdown. The split-weight/target-size scheduling nicety is not ported (pre-existing native path already omitted it; perf/scheduling-only, not correctness) -> DV-033. Tests: connector PaimonScanPlanProviderTest +6 — computeFileSplitOffsets math (250MB/64MB->4 with 58MB tail, exact-multiple, small-file-whole, empty, target<=0); determineTargetSplitSize heuristic (file_split_size override, 32MB<->64MB threshold, max_file_split_num floor); end-to-end append-only fixture (tiny file_split_size -> >=2 contiguous sub-ranges tiling [0,fileLength); default -> 1 range); DV on every sub-range; whole-file under count pushdown. Updated the 3 existing buildNativeRange call sites to the new signature. Connector 258/0/0 (1 CI-gated live skip), checkstyle 0, import-gate clean. Fail-before verified: neuter computeFileSplitOffsets -> the 3 splitting tests red; attach DV only to the first sub-range -> the DV test red. Real BE multi-range + DV read = CI-gated live-e2e (legacy paimon regression covers the BE contract; no BE change). Adversarially reviewed (workflow wf_4ac7479d-39d): 2 confirmed and fixed (the count-pushdown sub-split parity gap + false comment; the missing DV-on-every-sub-range test), 2 refuted. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 525be03 commit 2f5f467

6 files changed

Lines changed: 528 additions & 15 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java‎

Lines changed: 159 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,21 @@ public class PaimonScanPlanProvider implements ConnectorScanPlanProvider {
140140
// bypassing the native ORC/Parquet readers to dodge native-reader bugs. Default false (legacy default).
141141
private static final String FORCE_JNI_SCANNER = "force_jni_scanner";
142142

143+
// FIX-NATIVE-SUBSPLIT (M-3): file-split session vars (byte-identical to SessionVariable.{FILE_SPLIT_SIZE,
144+
// MAX_INITIAL_FILE_SPLIT_SIZE, MAX_FILE_SPLIT_SIZE, MAX_INITIAL_FILE_SPLIT_NUM, MAX_FILE_SPLIT_NUM}),
145+
// read via the same VariableMgr.toMap channel as ENABLE_PAIMON_CPP_READER. They drive the native
146+
// sub-split target size, mirroring legacy PaimonScanNode.determineTargetFileSplitSize without
147+
// importing fe-core SessionVariable/FileSplitter. Defaults below are byte-identical to SessionVariable.
148+
private static final String FILE_SPLIT_SIZE = "file_split_size";
149+
private static final String MAX_INITIAL_FILE_SPLIT_SIZE = "max_initial_file_split_size";
150+
private static final String MAX_FILE_SPLIT_SIZE = "max_file_split_size";
151+
private static final String MAX_INITIAL_FILE_SPLIT_NUM = "max_initial_file_split_num";
152+
private static final String MAX_FILE_SPLIT_NUM = "max_file_split_num";
153+
private static final long DEFAULT_MAX_INITIAL_FILE_SPLIT_SIZE = 32L * 1024 * 1024;
154+
private static final long DEFAULT_MAX_FILE_SPLIT_SIZE = 64L * 1024 * 1024;
155+
private static final long DEFAULT_MAX_INITIAL_FILE_SPLIT_NUM = 200L;
156+
private static final long DEFAULT_MAX_FILE_SPLIT_NUM = 100000L;
157+
143158
// FIX-SCHEMA-EVOLUTION (B-1a): scan-level prop carrying the base64 TBinaryProtocol-serialized
144159
// schema dictionary (a throwaway TFileScanRangeParams holding current_schema_id +
145160
// history_schema_info). getScanNodeProperties builds it from the live table; populateScanLevelParams
@@ -347,6 +362,10 @@ private List<ConnectorScanRange> planScanInternal(
347362
long countSum = 0;
348363
DataSplit countRepresentative = null;
349364

365+
// FIX-NATIVE-SUBSPLIT: target file split size for native ORC/Parquet sub-splitting, computed
366+
// lazily ONCE on the first native split (legacy hasDeterminedTargetFileSplitSize parity).
367+
long targetSplitSize = -1;
368+
350369
// Process DataSplits
351370
for (DataSplit dataSplit : dataSplits) {
352371
if (isCountPushdownSplit(countPushdown, dataSplit)) {
@@ -365,14 +384,25 @@ private List<ConnectorScanRange> planScanInternal(
365384

366385
if (shouldUseNativeReader(paimonHandle.isForceJni(),
367386
isForceJniScannerEnabled(session), optRawFiles)) {
368-
// Native reader path
387+
// Native reader path: sub-split large ORC/Parquet files for read parallelism
388+
// (FIX-NATIVE-SUBSPLIT), mirroring legacy fileSplitter.splitFile. Under COUNT(*) pushdown
389+
// legacy passes splittable=!applyCountPushdown, so a native split that reaches this arm
390+
// (i.e. NOT siphoned to the count arm because its merged count is not precomputed — e.g. a
391+
// DV with null cardinality) is kept WHOLE. We mirror that by passing target size 0, which
392+
// makes buildNativeRanges emit a single whole-file range; the target heuristic is then not
393+
// needed (and not computed) under count pushdown.
394+
if (!countPushdown && targetSplitSize < 0) {
395+
targetSplitSize = resolveTargetSplitSize(session, dataSplits);
396+
}
397+
long effectiveSplitSize = countPushdown ? 0L : targetSplitSize;
369398
List<RawFile> rawFiles = optRawFiles.get();
370399
for (int i = 0; i < rawFiles.size(); i++) {
400+
RawFile file = rawFiles.get(i);
371401
DeletionFile deletionFile =
372402
(optDeletionFiles.isPresent() && i < optDeletionFiles.get().size())
373403
? optDeletionFiles.get().get(i) : null;
374-
ranges.add(buildNativeRange(
375-
rawFiles.get(i), deletionFile, defaultFileFormat, partitionValues));
404+
ranges.addAll(buildNativeRanges(file, deletionFile, defaultFileFormat,
405+
partitionValues, effectiveSplitSize));
376406
}
377407
} else {
378408
// JNI reader path
@@ -403,12 +433,12 @@ private List<ConnectorScanRange> planScanInternal(
403433
* unit-testable without a live deletion-vector-bearing split.
404434
*/
405435
PaimonScanRange buildNativeRange(RawFile file, DeletionFile deletionFile,
406-
String defaultFileFormat, Map<String, String> partitionValues) {
436+
String defaultFileFormat, Map<String, String> partitionValues, long start, long length) {
407437
String fileFormat = getFileFormatBySuffix(file.path()).orElse(defaultFileFormat);
408438
PaimonScanRange.Builder builder = new PaimonScanRange.Builder()
409439
.path(normalizeUri(file.path()))
410-
.start(0)
411-
.length(file.length())
440+
.start(start)
441+
.length(length)
412442
.fileSize(file.length())
413443
.fileFormat(fileFormat)
414444
.partitionValues(partitionValues)
@@ -420,6 +450,27 @@ PaimonScanRange buildNativeRange(RawFile file, DeletionFile deletionFile,
420450
return builder.build();
421451
}
422452

453+
/**
454+
* Builds the native sub-range(s) for one raw ORC/Parquet file (FIX-NATIVE-SUBSPLIT): slices it at
455+
* {@code targetSplitSize} via {@link #computeFileSplitOffsets} and emits one {@link PaimonScanRange}
456+
* per {@code [start, length)} sub-range. The SAME per-file deletion vector is attached to EVERY
457+
* sub-range — BE indexes the DV by GLOBAL file row position, so disjoint sub-ranges share the
458+
* unmodified deletion file (no offset re-basing); attaching it to only some sub-ranges would let
459+
* deleted rows reappear in the others (merge-on-read corruption). A non-positive
460+
* {@code targetSplitSize} yields a single whole-file range (used under COUNT(*) pushdown, where
461+
* legacy keeps the split whole via {@code splittable=!applyCountPushdown}). Package-private so the
462+
* DV-on-every-sub-range invariant is unit-testable without a live DV-bearing split.
463+
*/
464+
List<PaimonScanRange> buildNativeRanges(RawFile file, DeletionFile deletionFile,
465+
String defaultFileFormat, Map<String, String> partitionValues, long targetSplitSize) {
466+
List<PaimonScanRange> result = new ArrayList<>();
467+
for (long[] offset : computeFileSplitOffsets(file.length(), targetSplitSize)) {
468+
result.add(buildNativeRange(
469+
file, deletionFile, defaultFileFormat, partitionValues, offset[0], offset[1]));
470+
}
471+
return result;
472+
}
473+
423474
/**
424475
* Normalizes a raw paimon-SDK storage URI (native data-file or deletion-vector path) into BE's
425476
* canonical scheme via the engine ({@code oss://}/{@code cos://}/{@code obs://}/{@code s3a://}
@@ -595,6 +646,108 @@ private PaimonScanRange buildCountRange(DataSplit dataSplit, String tableLocatio
595646
.build();
596647
}
597648

649+
/**
650+
* Slices a native data file into {@code [start, length]} sub-ranges for read parallelism
651+
* (FIX-NATIVE-SUBSPLIT), porting the specified-size branch of legacy {@code FileSplitter.splitFile}
652+
* (the connector has no block locations, so the block-based branch is never reached). Byte-identical
653+
* to {@code FileSplitter.java:129-144}, including the
654+
* <b>{@code > 1.1D} tail guard</b> — the LAST range absorbs a remainder of up to 1.1&times; the
655+
* target instead of emitting a tiny tail split (a naive {@code ceilDiv} would differ). The ranges
656+
* tile {@code [0, fileLength)} contiguously with no gap/overlap. A zero/negative file length yields
657+
* no range (legacy skips empty files); a non-positive target yields a single whole-file range —
658+
* used under COUNT(*) pushdown (see {@link #buildNativeRanges}, where legacy keeps the split whole
659+
* via {@code splittable=!applyCountPushdown}); {@link #determineTargetSplitSize} otherwise never
660+
* returns &le; 0. Pure static so the offset math is unit-testable against the fe-core source it ports.
661+
*/
662+
static List<long[]> computeFileSplitOffsets(long fileLength, long targetSplitSize) {
663+
List<long[]> result = new ArrayList<>();
664+
if (fileLength <= 0) {
665+
return result;
666+
}
667+
if (targetSplitSize <= 0) {
668+
result.add(new long[] {0L, fileLength});
669+
return result;
670+
}
671+
long bytesRemaining;
672+
for (bytesRemaining = fileLength;
673+
(double) bytesRemaining / (double) targetSplitSize > 1.1D;
674+
bytesRemaining -= targetSplitSize) {
675+
result.add(new long[] {fileLength - bytesRemaining, targetSplitSize});
676+
}
677+
if (bytesRemaining != 0L) {
678+
result.add(new long[] {fileLength - bytesRemaining, bytesRemaining});
679+
}
680+
return result;
681+
}
682+
683+
/**
684+
* Computes the native target file split size, porting legacy
685+
* {@code PaimonScanNode.determineTargetFileSplitSize} + {@code FileQueryScanNode.applyMaxFileSplitNumLimit}
686+
* with plain longs (the connector cannot import {@code SessionVariable}). The legacy
687+
* {@code isBatchMode -> 0} branch is omitted: paimon is never batch-mode on the plugin path. Pure
688+
* static so the heuristic is unit-testable.
689+
*/
690+
static long determineTargetSplitSize(long fileSplitSize, long maxInitialSplitSize, long maxSplitSize,
691+
long maxInitialSplitNum, long maxFileSplitNum, long totalNativeFileSize) {
692+
if (fileSplitSize > 0) {
693+
return fileSplitSize;
694+
}
695+
long result = (totalNativeFileSize >= maxSplitSize * maxInitialSplitNum)
696+
? maxSplitSize : maxInitialSplitSize;
697+
if (maxFileSplitNum > 0 && totalNativeFileSize > 0) {
698+
long minSplitSizeForMaxNum = (totalNativeFileSize + maxFileSplitNum - 1L) / maxFileSplitNum;
699+
result = Math.max(result, minSplitSizeForMaxNum);
700+
}
701+
return result;
702+
}
703+
704+
/**
705+
* Reads the 5 file-split session vars (VariableMgr.toMap channel) and sums the native-eligible
706+
* file sizes across {@code dataSplits}, then delegates to the pure-static
707+
* {@link #determineTargetSplitSize}. Mirrors legacy {@code determineTargetFileSplitSize}'s
708+
* once-per-scan computation (summing every {@code supportNativeReader}-eligible RawFile, like
709+
* {@code PaimonScanNode.java:552-564}).
710+
*/
711+
private long resolveTargetSplitSize(ConnectorSession session, List<DataSplit> dataSplits) {
712+
long totalNativeFileSize = 0;
713+
for (DataSplit dataSplit : dataSplits) {
714+
Optional<List<RawFile>> rawFiles = dataSplit.convertToRawFiles();
715+
if (!supportNativeReader(rawFiles)) {
716+
continue;
717+
}
718+
for (RawFile file : rawFiles.get()) {
719+
totalNativeFileSize += file.fileSize();
720+
}
721+
}
722+
return determineTargetSplitSize(
723+
sessionLong(session, FILE_SPLIT_SIZE, 0L),
724+
sessionLong(session, MAX_INITIAL_FILE_SPLIT_SIZE, DEFAULT_MAX_INITIAL_FILE_SPLIT_SIZE),
725+
sessionLong(session, MAX_FILE_SPLIT_SIZE, DEFAULT_MAX_FILE_SPLIT_SIZE),
726+
sessionLong(session, MAX_INITIAL_FILE_SPLIT_NUM, DEFAULT_MAX_INITIAL_FILE_SPLIT_NUM),
727+
sessionLong(session, MAX_FILE_SPLIT_NUM, DEFAULT_MAX_FILE_SPLIT_NUM),
728+
totalNativeFileSize);
729+
}
730+
731+
/**
732+
* Reads a long session var from the SPI session properties (VariableMgr.toMap channel), falling
733+
* back to {@code defaultValue} when absent/blank/unparseable. Mirrors the null-tolerant
734+
* {@link #isCppReaderEnabled} pattern.
735+
*/
736+
private static long sessionLong(ConnectorSession session, String key, long defaultValue) {
737+
if (session == null) {
738+
return defaultValue;
739+
}
740+
String value = session.getSessionProperties().get(key);
741+
if (value == null || value.trim().isEmpty()) {
742+
return defaultValue;
743+
}
744+
try {
745+
return Long.parseLong(value.trim());
746+
} catch (NumberFormatException e) {
747+
return defaultValue;
748+
}
749+
}
750+
598751
private long computeSplitWeight(DataSplit dataSplit) {
599752
List<DataFileMeta> metas = dataSplit.dataFiles();
600753
if (metas != null && !metas.isEmpty()) {

0 commit comments

Comments
 (0)